Daniel S. Katz: Copies of Papers

Access to published material may be restricted.

E. Nyenah, P. Döll, D. S. Katz, and R. Reinecke, "Software sustainability of global impact models," Geoscientific Model Development, v.17(23), pp. 8593–8611, 2024. https://doi.org/10.5194/gmd-17-8593-2024
(This document is open access)
Abstract: Research software for simulating Earth processes enables the estimation of past, current, and future world states and guides policy. However, this modelling software is often developed by scientists with limited training, time, and funding, leading to software that is hard to understand, (re)use, modify, and maintain and that is, in this sense, non-sustainable. Here we evaluate the sustainability of global-scale impact models across 10 research fields. We use nine sustainability indicators for our assessment. Five of these indicators – documentation, version control, open-source license, provision of software in containers, and the number of active developers – are related to best practices in software engineering and characterize overall software sustainability. The remaining four – comment density, modularity, automated testing, and adherence to coding standards – contribute to code quality, an important factor in software sustainability. We found that 29% (32 out of 112) of the global impact models (GIMs) participating in the Inter-Sectoral Impact Model Intercomparison Project were accessible without contacting the developers. Regarding best practices in software engineering, 75% of the 32 GIMs have some kind of documentation, 81% use version control, and 69% have an open-source license. Only 16% provide the software in a containerized form, which can potentially limit result reproducibility. Four models had no active development after 2020. Regarding code quality, we found that models suffer from low code quality, which impedes model improvement, maintenance, reusability, and reliability. Key issues include a non-optimal comment density in 75% of the GIMs, insufficient modularity in 88% of the GIMs, and the absence of a testing suite in 72% of the GIMs. Furthermore, only 5 out of 10 models for which the source code, either in part or in its entirety, is written in Python show good compliance with PEP8 coding standards, with the rest showing low compliance. To improve the sustainability of GIMs and other research software, we recommend best practices for sustainable software development to the scientific community. As an example of implementing these best practices, we show how reprogramming a legacy model using best practices has improved software sustainability.
E. A. Jensen, D. S. Katz, "Strategic priorities and challenges in research software funding: Results from an international survey [version 1; peer review: awaiting peer review]," F1000Research 13:1447, 2024. https://doi.org/10.12688/f1000research.155879.1
(This document is open access)
Abstract:
Background
Research software is increasingly recognized as critical infrastructure in contemporary science. It spans a broad spectrum, including source code files, algorithms, scripts, computational workflows, and executables, all created for or during research. While research funders have developed programs, initiatives, and policies to bolster research software’s role, there has been no empirical study of how these funders prioritize support for research software. Understanding their priorities is essential to clarify where current support is concentrated and to identify strategic gaps.
Methods
We conducted an online mixed methods survey of international research funders (n=36) to explore their priorities in supporting research software. The survey gathered data on the specific outcomes funders emphasize in their programs and initiatives for research software.
Results
The survey revealed that funders place strong emphasis on developing skills, promoting software sustainability, embedding open science practices, building community and collaboration, advancing research software funding mechanisms, increasing software visibility and use, fostering innovation, and ensuring security.
Conclusions
The findings highlight opportunities to enhance research software’s role through increased funder attention on professional recognition for software contributions and the non-technical, social aspects of research software sustainability. Addressing these areas could lead to more effective support and development of research software, ultimately benefitting the entire research ecosystem.
N. Karle, B. Clifford, Y. Babuji, R. Chard, D. S. Katz, K. Chard, "Parsl+CWL: Experiences Combining the Python and CWL Ecosystems," 19th Workshop on Workflows in Support of Large-Scale Science (WORKS 2024), in conjuction with International Conference for High Performance Computing, Networking, Storage, and Analysis (SC24), 2024. https://doi.org/10.1109/SCW63240.2024.00255
(This document is not yet open access)
Abstract: The Common Workflow Language (CWL) is a widely adopted language for defining and sharing computational workflows. It is designed to be independent of the execution engine on which workflows are executed. In this paper, we describe our experiences integrating CWL with Parsl, a Python-based parallel programming library designed to manage execution of workflows across diverse computing environments. We propose a new method that converts CWL CommandLineTool definitions into Parsl apps, enabling Parsl scripts to easily import and use tools represented in CWL. We describe a Parsl runner that is capable of executing a CWL CommandLineTool directly. We also describe a proof-of-concept extension to support inline Python in a CWL workflow definition, enabling seamless use in Parsl’s Python ecosystem. We demonstrate the benefits of this integration by presenting example CWL CommandLineTool definitions that show how they can be used in Parsl, and comparing performance of executing an image processing workflow using the Parsl integration and other CWL runners.
R. Ananthakrishnan, Y. Babuji, J. Bryan, K. Chard, R. Chard, I. Foster, K. Hunter Kesling, C. Janidlo, D. S. Katz, R. Mello, G. Pauloski, L. Wang, "Establishing a High-Performance and Productive Ecosystem for Distributed Execution of Python Functions Using Globus Compute," HUST-24: 11th International Workshop on HPC User Support Tools, in conjuction with International Conference for High Performance Computing, Networking, Storage, and Analysis (SC24), 2024. https://doi.org/10.1109/SCW63240.2024.00083
(This document is not yet open access)
Abstract: The research computing ecosystem is increasingly heterogeneous and diverse. Democratizing access to these essential resources is critical for accelerating research progress. However, the gap between a high-level workload, such as Python in a Jupyter notebook, and the resources and interfaces exposed by HPC systems is significant. Users must securely authenticate, manage network connections, deploy and manage software, provision and configure nodes, and manage workload execution. Globus Compute reduces these barriers by providing a managed, fire-and-forget model that enables execution of Python functions across any resource to which a user has access. However, while Globus Compute has relieved users from many of the challenges of remote computing, we have observed some inefficiencies that remain in terms of use. For example, many users wrap external applications, such as C/C++, Fortran, and even MPI applications, in Python functions and users must deploy many endpoints on a single computer to exploit different configurations. In this paper we describe enhancements to Globus Compute to address these barriers: an asynchronous, future-based executor interface for submitting and monitoring tasks, shell and MPI-based function types, and a multi-user endpoint that can be deployed by administrators and used by authorized users.
R. Ananthakrishnan, Y. Babuji, M. Baughman, J. Bryan, K. Chard, R. Chard, B. Clifford, I. Foster, D. S. Katz, K. Hunter Kesling, C. Janidlo, R. Mello, and L. Wang, "Enabling Remote Management of FaaS Endpoints with Globus Compute Multi-User Endpoints," PEARC '24: Practice and Experience in Advanced Research Computing, 2024. https://doi.org/10.1145/3626203.3670612
(This document is open access)
Abstract: Globus Compute implements a hybrid Function as a Service (FaaS) model in which a single cloud-hosted service is used by users to manage execution of Python functions on user-owned and -managed Globus Compute endpoints deployed on arbitrary compute resources. Here we describe a new multi-user and multi-configuration Globus Compute endpoint. This system, which can be deployed by administrators in a privileged account, enables dynamic creation of user endpoints that are forked as new processes in user space. The multi-user endpoint is designed to provide the security interfaces necessary for deployment on large, shared HPC clusters by, for example, restricting user endpoint configurations, enforcing various authorization policies, and via customizable identity-username mapping.
D. S. Katz, J. Carver, "Thoughts on Learning Human and Programming Languages," Computing in Science & Engineering, v.26(1), pp. 77-80, Jan.-Mar. 2024. https://doi.org/10.1109/MCSE.2024.3398949 (preprint: arXiv:2407.15907)
(This document is green open access)
Abstract: This is a virtual dialog between Jeffrey C. Carver and Daniel S. Katz on how people learn programming languages. It's based on a talk Jeff gave at the first US-RSE Conference (US-RSE'23), which led Dan to think about human languages versus computer languages. Dan discussed this with Jeff at the conference, and this discussion continued asynchronous, with this column being a record of the discussion.
S. Grayson, F. Aguilar, R. Milewicz, D. S. Katz, D. Marinov, "A benchmark suite and performance analysis of user-space provenance collectors," 2024 ACM Conference on Reproducibility and Replicability (ACM REP '24), 2024. https://doi.org/10.1145/3641525.3663627
(This document is open access)
Abstract: Computational provenance has many important applications, especially to reproducibility. System-level provenance collectors can track provenance data without requiring the user to change anything about their application. However, system-level provenance collectors have performance overheads, and, worse still, different works use different and incomparable benchmarks to assess their performance overhead. This work identifies user-space system-level provenance collectors in prior work, collates the benchmarks, and evaluates each collector on each benchmark. We use benchmark minimization to select a minimal subset of benchmarks, which can be used as goalposts for future work on system-level provenance collectors.
D. S. Katz, V. Kindratenko, O. Kindratenko, P. Mazumdar, "Training Next Generation AI Users and Developers at NCSA," Computing in Science & Engineering, v.25(6), pp. 28-32, Nov.-Dec. 2023. https://doi.org/10.1109/MCSE.2024.3375572 (preprint: arXiv:2406.14744)
(This document is green open access)
Abstract: This article focuses on training work carried out in artificial intelligence (AI) at the National Center for Supercomputing Applications (NCSA) at the University of Illinois Urbana-Champaign via a research experience for undergraduates (REU) program named FoDOMMaT. It also describes why we are interested in AI, and concludes by discussing what we've learned from running this program and its predecessor over six years.
R. Sonabend, H. Gruson, L. Wolansky, A. Kiragga, D. S. Katz, "FAIR-USE4OS: Guidelines for creating impactful open-source software," PLOS Computational Biology, v.20(5), e1012045, 2024. https://doi.org/10.1371/journal.pcbi.1012045
(This document is open access)
Abstract: This paper extends the FAIR (Findable, Accessible, Interoperable, Reusable) guidelines to provide criteria for assessing if software conforms to best practices in open source. By adding "USE" (User-Centered, Sustainable, Equitable), software development can adhere to open source best practice by incorporating user-input early on, ensuring front-end designs are accessible to all possible stakeholders, and planning long-term sustainability alongside software design. The FAIR-USE4OS guidelines will allow funders and researchers to more effectively evaluate and plan open-source software projects. There is good evidence of funders increasingly mandating that all funded research software is open source; however, even under the FAIR guidelines, this could simply mean software released on public repositories with a Zenodo DOI. By creating FAIR-USE software, best practice can be demonstrated from the very beginning of the design process and the software has the greatest chance of success by being impactful.
M. Hosseini, A. O. Holcombe, M. Kovacs, H. Zwart, D. S. Katz, K. Holmes, "Group authorship, an excellent opportunity laced with ethical, legal and technical challenges," Accountability in Research: Ethics, Integrity and Policy, 2024. https://doi.org/10.1080/08989621.2024.2322557
(This document is open access)
Abstract: Group authorship (also known as corporate authorship, team authorship, consortium authorship) refers to attribution practices that use the name of a collective (be it team, group, project, corporation, or consortium) in the authorship byline. Data shows that group authorships are on the rise but thus far, in scholarly discussions about authorship, they have not gained much specific attention. Group authorship can minimize tensions within the group about authorship order and the criteria used for inclusion/exclusion of individual authors. However, current use of group authorships has drawbacks, such as ethical challenges associated with the attribution of credit and responsibilities, legal challenges regarding how copyrights are handled, and technical challenges related to the lack of persistent identifiers (PIDs), such as ORCID, for groups. We offer two recommendations: 1) Journals should develop and share context-specific and unambiguous guidelines for group authorship, for which they can use the four baseline requirements offered in this paper; 2) Using persistent identifiers for groups and consistent reporting of members’ contributions should be facilitated through devising PIDs for groups and linking these to the ORCIDs of their individual contributors and the Digital Object Identifier (DOI) of the published item.
J. Duarte, H. Li, A. Roy, R. Zhu, E. A. Huerta, D. Diaz, P. Harris, R. Kansal, D. S. Katz, I. H. Kavoori, V. V. Kindratenko, F. Mokhtar, M. S. Neubauer, S. E. Park, M. Quinnan, R. Rusack, and Z. Zhao, "FAIR AI Models in High Energy Physics," Machine Learning: Science and Technology, v.4(4), 045062, 2023. https://doi.org/10.1088/2632-2153/ad12e3
(This document is open access)
Abstract: The findable, accessible, interoperable, and reusable (FAIR) data principles provide a framework for examining, evaluating, and improving how data is shared to facilitate scientific discovery. Generalizing these principles to research software and other digital products is an active area of research. Machine learning (ML) models—algorithms that have been trained on data without being explicitly programmed—and more generally, artificial intelligence (AI) models, are an important target for this because of the ever-increasing pace with which AI is transforming scientific domains, such as experimental high energy physics (HEP). In this paper, we propose a practical definition of FAIR principles for AI models in HEP and describe a template for the application of these principles. We demonstrate the template's use with an example AI model applied to HEP, in which a graph neural network is used to identify Higgs bosons decaying to two bottom quarks. We report on the robustness of this FAIR AI model, its portability across hardware architectures and software frameworks, and its interpretability.
E. Karrels, L. Huang, Y. Kan, I. Arora, Y. Wang, D. S. Katz, W. Gropp, Z. Zhang, "Fine-grained Policy-driven I/O Sharing for Burst Buffers," International Conference for High Performance Computing, Networking, Storage, and Analysis (SC23), 2023. https://doi.org/10.1145/3581784.3607041
(This document is open access)
Abstract: A burst buffer is a common method to bridge the performance gap between the I/O needs of modern supercomputing applications and the performance of the shared file system on large-scale supercomputers. However, existing I/O sharing methods require resource isolation, offline profiling, or repeated execution that significantly limit the utilization and applicability of these systems. Here we present ThemisIO, a policy-driven I/O sharing framework for a remote-shared burst buffer: a dedicated group of I/O nodes, each with a local storage device. ThemisIO preserves high utilization by implementing opportunity fairness so that it can reallocate unused I/O resources to other applications. ThemisIO accurately and efficiently allocates I/O cycles among applications, purely based on real-time I/O behavior without requiring user-supplied information or offline-profiled application characteristics. ThemisIO supports a variety of fair sharing policies, such as user-fair, size-fair, as well as composite policies, e.g., group-then-user-fair. All these features are enabled by its statistical token design. ThemisIO can alter the execution order of incoming I/O requests based on assigned tokens to precisely balance I/O cycles between applications via time slicing, thereby enforcing processing isolation. Experiments using I/O benchmarks show that ThemisIO sustains 13.5–13.7% higher I/O throughput and 19.5–40.4% lower performance variation than existing algorithms. For real applications, ThemisIO significantly reduces the slowdown by 59.1–99.8% caused by I/O interference.
E. A. Huerta, B. Blaiszik, L. C. Brinson, K. E. Bouchard, D. Diaz, C. Doglioni, J. M. Duarte, M. Emani, I. Foster, G. Fox, P. Harris, L. Heinrich, S. Jha, D. S. Katz, V. Kindratenko, C. R. Kirkpatrick, K. Lassila-Perini, R. K. Madduri, M. S. Neubauer, F. E. Psomopoulos, A. Roy, O. Rübel, Z. Zhao, R. Zhu, "FAIR for AI: An interdisciplinary and international community building perspective," Scientific Data v.10, 487, 2023. https://doi.org/10.1038/s41597-023-02298-6
(This document is open access)
Abstract: A foundational set of findable, accessible, interoperable, and reusable (FAIR) principles were proposed in 2016 as prerequisites for proper data management and stewardship, with the goal of enabling the reusability of scholarly data. The principles were also meant to apply to other digital assets, at a high level, and over time, the FAIR guiding principles have been re-interpreted or extended to include the software, tools, algorithms, and workflows that produce data. FAIR principles are now being adapted in the context of AI models and datasets. Here, we present the perspectives, vision, and experiences of researchers from different countries, disciplines, and backgrounds who are leading the definition and adoption of FAIR principles in their communities of practice, and discuss outcomes that may result from pursuing and incentivizing FAIR AI research. The material for this report builds on the FAIR for AI Workshop held at Argonne National Laboratory on June 7, 2022.
E. C. McKiernan, L. Barba, P. E. Bourne, C. Carter, Z. Chandler, S. Choudhury, S. Jacobs, D. S. Katz, S. Lieggi, B. Plale, G. Tananbaum, "Policy recommendations to ensure that research software is openly accessible and reusable," PLoS Biology, 2023. https://doi.org/10.1371/journal.pbio.3002204
(This document is open access)
Abstract: Research data is optimized when it can be freely accessed and reused. To maximize research equity, transparency, and reproducibility, policymakers should take concrete steps to ensure that research software is openly accessible and reusable.
D. S. Katz, B. Clifford, Y. Babuji, K. Hunter Kesling, A. Woodard, K. Chard, "The Changing Role of RSEs over the Lifetime of Parsl," Software Engineering for Research Software (SE4RS'23), in conjunction with PEARC23, 2023. https://doi.org/10.48550/arXiv.2307.11060
(This document is open access)
Abstract: This position paper describes the Parsl open source research software project and its various phases over seven years. It defines four types of research software engineers (RSEs) who have been important to the project in those phases; we believe this is also applicable to other research software projects.
S. Grayson, R. Milewicz, J. Teves, D. S. Katz, D. Marinov, "Wanted: standards for automatic reproducibility of computational experiments," Software Engineering for Research Software (SE4RS'23), in conjunction with PEARC23, 2023. https://doi.org/10.48550/arXiv.2307.11383
(This document is open access)
S. Grayson, D. Marinov, D. S. Katz, R. Milewicz, "Automatic Reproduction of Workflows in the Snakemake Workflow Catalog and nf-core Registries," 2023 ACM Conference on Reproducibility and Replicability (ACM REP '23), pp. 74-84, 2023. https://doi.org/10.1145/3589806.3600037
(This document is not open access)
Abstract: Workflows make it easier for scientists to assemble computational experiments consisting of many disparate components. However, those disparate components also increase the probability that the computational experiment fails to be reproducible. Even if software is reproducible today, it may become irreproducible tomorrow without the software itself changing at all, because of the constantly changing software environment in which the software is run.
To alleviate irreproducibility, workflow engines integrate with container engines. Additionally, communities that sprung up around workflow engines started to host registries for workflows that follow standards. These standards reduce the effort needed to make workflows automatically reproducible.
In this paper, we study automatic reproduction of workflows from two registries, focusing on non-crashing executions. The experimental data lets us analyze the upper bound to which workflow engines could achieve reproducibility. We identify lessons learned in achieving reproducibility in practice.
T. Gamblin, D. S. Katz, "Overcoming Challenges to Continuous Integration in HPC," Computing in Science & Engineering (Software Engineering Department), v.24(6), pp. 54-59, Nov.-Dec. 2022. https://doi.org/10.1109/MCSE.2023.3263458 (preprint: arXiv:2303.17034)
(This document is green open access)
Abstract: Continuous integration (CI) has become a ubiquitous practice in modern software development, with major code hosting services offering free automation on popular platforms. CI offers major benefits, as it enables detecting bugs in code prior to committing changes. While high-performance computing (HPC) research relies heavily on software, HPC machines are not considered "common" platforms. This presents several challenges that hinder the adoption of CI in HPC environments, making it difficult to maintain bug-free HPC projects, and resulting in adverse effects on the research community. In this article, we explore the challenges that impede HPC CI, such as hardware diversity, security, isolation, administrative policies, and nonstandard authentication, environments, and job submission mechanisms. We propose several solutions that could enhance the quality of HPC software and the experience of developers. Implementing these solutions would require significant changes at HPC centers, but if these changes are made, it would ultimately enable faster and better science.
I. A. Cosden, K. McHenry, D. S. Katz, "Research Software Engineers: Career Entry Points and Training Gaps," Computing in Science & Engineering, v.24(6), pp. 14-21, Nov.-Dec. 2022. https://doi.org/10.1109/MCSE.2023.3258630 (preprint: arXiv:2210.04275)
(This document is green open access)
Abstract: As software has become more essential to research across disciplines, and as the recognition of this fact has grown, the importance of professionalizing the development and maintenance of this software has also increased. The community of software professionals who work on this software have come together under the title Research Software Engineer (RSE) over the last decade. This has led to the formalization of RSE roles and organized RSE groups in universities, national labs, and industry. This, in turn, has created the need to understand how RSEs come into this profession and into these groups, how to further promote this career path to potential members, as well as the need to understand what training gaps need to be filled for RSEs coming from different entry points. We have categorized three main classifications of entry paths into the RSE profession and identified key elements, both advantages and disadvantages, that should be acknowledged and addressed by the broader research community in order to attract and retain a talented and diverse pool of future RSEs.
W. F. Godoy, R. Arora, K. Beattie, D. E. Bernholdt, S. E. Bratt, D. S. Katz, I. Laguna, A. K. Maji, A. Malviya Thakur, R. M. Mudafort, N. Sukhija, D. Rouson, Cindy Rubio-González, Karan Vahi, "Giving RSEs a Larger Stage through the Better Scientific Software Fellowship," Computing in Science & Engineering, v.24(5), pp. 6-13, Sept.-Oct. 2022. https://doi.org/10.1109/MCSE.2023.3253847 (preprint: arXiv:2211.07436)
(This document is green open access)
Abstract: The Better Scientific Software Fellowship (BSSwF) was launched in 2018 to foster and promote practices, processes, and tools to improve developer productivity and software sustainability of scientific codes. BSSwF vision is to grow the community with practitioners, leaders, mentors, and consultants to increase the visibility of scientific software. Over the last five years, many fellowship recipients and honorable mentions have identified as research software engineers (RSEs). Case studies from several of the program’s participants illustrate the diverse ways BSSwF has benefited both the RSE and scientific communities. In an environment where the contributions of RSEs are too often undervalued, we believe that programs such as BSSwF can help recognize and encourage community members to step outside of their regular commitments and expand on their work, collaborations and ideas for a larger audience.
A.-L. Lamprecht, C. Martinez-Ortiz, M. Barker, S. L. Bartholomew, J. Barton, N. Chue Hong, J. Cohen, S. Druskat, J. Forest, J.-N. Grad, D. S. Katz, R. Richardson, R. Rosca D. Schulte, A. Struck, M. Weinzierl, "What Do We (Not) Know About Research Software Engineering?," Journal of Open Research Software, v.10(1): p. 11, 2022. https://doi.org/10.5334/jors.384
(This document is open access)
Abstract: As recognition of the vital importance of software for contemporary research is increasing, Research Software Engineering (RSE) is emerging as a discipline in its own right. We present an inventory of relevant research questions about RSE as a basis for future research and initiatives to advance the field, highlighting selected literature and initiatives. This work is the outcome of a RSE community workshop held as part of the 2020 International Series of Online Research Software Events (SORSE) which identified and prioritized key questions across three overlapping themes: people, policy and infrastructure. Almost half of the questions focus on the people theme, which addresses issues related to career paths, recognition and motivation; recruitment and retention; skills; and diversity, equity and inclusion. However, the people and policy themes have the same number of prioritized questions. We recommend that different types of stakeholders, such as RSE employers and policy makers, take responsibility for supporting or encouraging answering of these questions by organizations that have an interest. Initiatives such as the International Council of RSE Associations should also be engaged in this work.
M. Barker, N. Chue Hong, D. S. Katz, A.-L. Lamprecht, C. Martinez-Ortiz, F. Psomopoulos, J. Harrow, L. J. Castro, M. Gruenpeter, P. A. Martinez, and T. Honeyman, "Introducing the FAIR Principles for research software," Scientific Data, 9, 622, 2022. https://doi.org/10.1038/s41597-022-01710-x
(This document is open access)
Abstract: Research software is a fundamental and vital part of research, yet significant challenges to discoverability, productivity, quality, reproducibility, and sustainability exist. Improving the practice of scholarship is a common goal of the open science, open source, and FAIR (Findable, accessible, Interoperable and Reusable) communities and research software is now being understood as a type of digital object to which FAIR should be applied. This emergence reflects a maturation of the research community to better understand the crucial role of FAIR esearch software in maximising research value. The FAIR for Research Software (FAIR4RS) Working Group has adapted the FAIR Guiding Principles to create the FAIR Principles for Research Software (FAIR4RS Principles). The contents and context of the FAIR4RS Principles are summarised here to provide the basis for discussion of their adoption. Examples of implementation by organisations are provided to share information on how to maximise the value of research outputs, and to encourage others to amplify the importance and impact of this work.
Z. Li, R. Chard, Y. Babuji, B. Galewsky, T. Skluzacek, K. Nagaitsev, A. Woodard, B. Blaiszik, J. Bryan, D. S. Katz, I. Foster, K. Chard, "funcX: Distributed and High Performance Function as a Service for Science," IEEE Transactions on Parallel and Distributed Systems, 2022. https://doi.org/10.1109/TPDS.2022.3208767 (preprint: arXiv:2209.11631)
(This document is not open access)
Abstract: funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. funcX’s endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and supercomputers, in effect turning them into function serving systems. funcX’s cloud-hosted service provides a single location for registering, sharing, and managing both functions and endpoints. It allows for transparent, secure, and reliable function execution across the federated ecosystem of endpoints—enabling users to route functions to endpoints based on specific needs. funcX uses containers (e.g., Docker, Singularity, and Shifter) to provide common execution environments across endpoints. funcX implements various container management strategies to execute functions with high performance and efficiency on diverse funcX endpoints. funcX also integrates with an in-memory data store and Globus for managing data that may span endpoints. We motivate the need for funcX, present our prototype design and implementation, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than 130 000 concurrent workers. We show that funcX’s container warming-aware routing algorithm can reduce the completion time for 3000 functions by up to 61% compared to a randomized algorithm and the in-memory data store can speed up data transfers by up to 3x compared to a shared file system.
N. P. Chue Hong, D. S. Katz, M. Barker, A.-L. Lamprecht, C. Martinez, F. Psomopoulos, J. Harrow, L. J. Castro, M. Gruenpeter, P. A. Martinez, T. Honeyman, A. Struck, A. Lee, A. Loewe, B. van Werkhoven, C. Jones, D. Garijo, E. Plomp, F. Genova, H. Shanahan, J. Leng, M. Hellström, M. Sandström, M. Sinha, M. Kuzak, P. Herterich, Q. Zhang, S. Islam, S.-A. Sansone, T. Pollard, U. D. Atmojo, A. Williams, A. Czerniak, A. Niehues, A. C. Fouilloux, B. Desinghu, C. Goble, C. Richard, C. Gray, C. Erdmann, D. Nüst, D. Tartarini, E. Ranguelova, H. Anzt, I. Todorov, J. McNally, J. Moldon, J. Burnett, J. Garrido-Sánchez, K. Belhajjame, L. Sesink, L. Hwang, M. R. Tovani-Palone, M. D. Wilkinson, M. Servillat, M. Liffers, M. Fox, N. Miljković, N. Lynch, P. Martinez Lavanchy, S. Gesing, S. Stevens, S. Martinez Cuesta, S. Peroni, S. Soiland-Reyes, T. Bakker, T. Rabemanantsoa, V. Sochat, Y. Yehudi, RDA FAIR4RS WG, "FAIR Principles for Research Software (FAIR4RS Principles)," Research Data Alliance, May 2022. https://doi.org/10.15497/RDA00068
(This document is open access)
Abstract: Research software is a fundamental and vital part of research worldwide, yet there remain significant challenges to software productivity, quality, reproducibility, and sustainability. Improving the practice of scholarship is a common goal of the open science, open source software and FAIR (Findable, Accessible, Interoperable and Reusable) communities, but improving the sharing of research software has not yet been a strong focus of the latter. To improve the FAIRness of research software, the FAIR for Research Software (FAIR4RS) Working Group has sought to understand how to apply the FAIR Guiding Principles for scientific data management and stewardship to research software, bringing together existing and new community efforts. Many of the FAIR Guiding Principles can be directly applied to research software by treating software and data as similar digital research objects. However, specific characteristics of software — such as its executability, composite nature, and continuous evolution and versioning — make it necessary to revise and extend the principles.
This document presents the first version of the FAIR Principles for Research Software (FAIR4RS Principles). It is an outcome of the FAIR for Research Software Working Group (FAIR4RS WG).
The FAIR for Research Software Working Group is jointly convened as an RDA Working Group, FORCE11 Working Group, and Research Software Alliance (ReSA) Task Force.
Going forward, the RDA Software Source Code Interest Group is the maintenance home for the principles. Concerns or queries about the principles can be raised at RDA plenary events organized by the SSC IG, where there may be opportunities for adopters to report back on progress. The full maintenance and retirement plan for the principles can be found on the RDA website.
J. C. Carver, N. Weber, K. Ram, S. Gesing, D. S. Katz, "A survey of the state of the practice for research software in the United States" PeerJ Computer Science, v.8, p. e963, 2022. https://doi.org/10.7717/peerj-cs.963
(This paper is open access)
Abstract: Research software is a critical component of contemporary scholarship. Yet, most research software is developed and managed in ways that are at odds with its long-term sustainability. This paper presents findings from a survey of 1,149 researchers, primarily from the United States, about sustainability challenges they face in developing and using research software. Some of our key findings include a repeated need for more opportunities and time for developers of research software to receive training. These training needs cross the software lifecycle and various types of tools. We also identified the recurring need for better models of funding research software and for providing credit to those who develop the software so they can advance in their careers. The results of this survey will help inform future infrastructure and service support for software developers and users, as well as national research policy aimed at increasing the sustainability of research software.
Y. Chen, E. A. Huerta, J. Duarte, P. Harris, D. S. Katz, M. S. Neubauer, D. Diaz, F. Mokhtar, R. Kansal, S. E. Park, V. V. Kindratenko, Z. Zhao, R. Rusack, "A FAIR and AI-ready Higgs Boson Decay Dataset," Scientific Data, v.9(31), 2022. https://doi.org/10.1038/s41597-021-01109-0 (preprint: arXiv:2108.02214)
(This paper is open access)
Abstract: To enable the reusability of massive scientific datasets by humans and machines, researchers aim to adhere to the principles of findability, accessibility, interoperability, and reusability (FAIR) for data and artificial intelligence (AI) models. This article provides a domain-agnostic, step-by-step assessment guide to evaluate whether or not a given dataset meets these principles. We demonstrate how to use this guide to evaluate the FAIRness of an open simulated dataset produced by the CMS Collaboration at the CERN Large Hadron Collider. This dataset consists of Higgs boson decays and quark and gluon background, and is available through the CERN Open Data Portal. We use additional available tools to assess the FAIRness of this dataset, and incorporate feedback from members of the FAIR community to validate our results. This article is accompanied by a Jupyter notebook to visualize and explore this dataset. This study marks the first in a planned series of articles that will guide scientists in the creation of FAIR AI models and datasets in high energy particle physics.
D. S. Katz, S. Stall, "Software citation," Information Services & Use, v.41(1-2), pp. 39-42, 2021. https://doi.org/10.3233/ISU-210108
(This paper is open access)
Abstract: Nearly all research today has a digital component, and typically, scholarly results are strongly dependent on software. For the research results to be fully understood, the software that is used must be uniquely identified. Research software is frequently developed by researchers themselves, often initially to solve a single problem, and then later generalized to solve additional problems. Ideally, the software is shared so that other researchers can also benefit and avoid the duplicate work required for development and maintenance. The researchers must expect and receive value for their contribution and sharing. Because publishing is a key element of our existing scholarly structures, the research that was done must be clearly explained in papers. This can be used to create incentives for researchers not only to share their software, but also to contribute to community software, in both cases through software citation. Contributors to software that is used in papers and is cited by those papers can become authors of the software as it is tracked by indexes, which also track how often the software is cited.
C. Jay, R. Haines, D. S. Katz, "Software must be recognised as an important output of scholarly research," International Journal of Digital Curation, v.16(1), 2021. https://doi.org/10.2218/ijdc.v16i1.745 (preprint: arXiv:2011.07571)
(This paper is open access)
Abstract: Software now lies at the heart of scholarly research. Here we argue that as well as being important from a methodological perspective, software should, in many instances, be recognised as an output of research, equivalent to an academic paper. The article discusses the different roles that software may play in research and highlights the relationship between software and research sustainability and reproducibility. It describes the challenges associated with the processes of citing and reviewing software, which differ from those used for papers. We conclude that whilst software outputs do not necessarily fit comfortably within the current publication model, there is a great deal of positive work underway that is likely to make an impact in addressing this.
R. Ferreira da Silva, H. Casanova, K. Chard, I. Altintas, R. M. Badia, B. Balis, T. Coleman, F. Coppens, F. Di Natale, T. Fahringer, R. Filgueira, G. Fursin, D. Garijo, C. Goble, D. Howell, S. Jha, D. S. Katz, D. Laney, U. Leser, M. Malawski, K. Mehta, L. Pottier, J. Ozik, J. L. Peterson, L. Ramakrishnan, S. Soiland-Reyes, D. Thain, M. Wolf, "A Community Roadmap for Scientific Workflows Research and Development," 16th Workshop on Workflows in Support of Large-Scale Science (WORKS21), in conjuction with the International Conference for High Performance Computing, Networking, Storage and Analysis (SC21), 2021. https://doi.org/10.1109/WORKS54523.2021.00016 (preprint: arXiv 2110.02168)
(This paper is green open access)
Abstract: The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual “Workflows Community Summits” (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.
S. Malik, S. Meehan, K. Lieret, M. O. Evans, M. H. Villanueva, D. S. Katz, G. A. Stewart, P. Elmer, S. Aziz, M. Bellis, R. M. Bianchi, G. Bianco, J. S. Bonilla, A. Burger, J. Burzynski, D. Chamont, M. Feickert, P. Gadow, B. M. Gruber, D. Guest, S. Hageboeck, L. Heinrich, M. M. Horzela, M. Huwiler, C. Lange, K. Lehmann, K. Li, D. Majumder, J. Mamužić, K. Nelson, R. Newhouse, E. Nibigira, S. Norberg, A. S. Pineda, M. Proffitt, B. Regnery, A. Roepe, S. Roiser, H. Schreiner, O. Shadura, G. Stark, S. N. Swatman, S. Thais, A. Valassi, S. Wunsch, D. Yakobovitch, S. Yuan, "Software Training in HEP," Computing and Software for Big Science, v.5:22, 2021. https://doi.org/10.1007/s41781-021-00069-9
(This paper is open access)
Abstract: The long-term sustainability of the high-energy physics (HEP) research software ecosystem is essential to the field. With new facilities and upgrades coming online throughout the 2020s, this will only become increasingly important. Meeting the sustainability challenge requires a workforce with a combination of HEP domain knowledge and advanced software skills. The required software skills fall into three broad groups. The first is fundamental and generic software engineering (e.g., Unix, version control, C++, and continuous integration). The second is knowledge of domain-specific HEP packages and practices (e.g., the ROOT data format and analysis framework). The third is more advanced knowledge involving specialized techniques, including parallel programming, machine learning and data science tools, and techniques to maintain software projects at all scales. This paper discusses the collective software training program in HEP led by the HEP Software Foundation (HSF) and the Institute for Research and Innovation in Software in HEP (IRIS-HEP). The program equips participants with an array of software skills that serve as ingredients for the solution of HEP computing challenges. Beyond serving the community by ensuring that members are able to pursue research goals, the program serves individuals by providing intellectual capital and transferable skills important to careers in the realm of software and computing, inside or outside HEP.
D. S. Katz, F. E. Psomopoulos, L. J. Castro, "Working Towards Understanding the Role of FAIR for Machine Learning," DaMaLOS 2021 (2nd Workshop on Data and research objects Management for Linked Open Science), 2021. https://doi.org/10.4126/FRL01-006429415
(This paper is open access)
Abstract: The FAIR Guiding Principles aim to improve findability, accessibility, interoperability and reusability for both humans and machines, initially aimed at scientific data, but also intended to apply to all sorts of research digital objects, with recent developments about their modification and application to software and computational workflows. In this position paper we argue that the FAIR principles also can apply to machine learning tools and models, though a direct application is not always possible as machine learning combines aspects of data and software. Here we discuss some of the elements of machine learning that lead to the need for some adaptation of the original FAIR principles, along with stakeholders that would benefit from this adaptation. We introduce the initial steps towards this adaptation, i.e., creating a community around it, some possible benefits beyond FAIR, and some of the open questions that such a community could tackle.
A. Villarreal, Y. Babuji, T. Uram, D. S. Katz, K. Chard, K. Heitmann, "Extreme Scale Survey Simulation with Python Workflows," 17th IEEE International Conference on eScience (eScience2021), 2021. https://doi.org/10.1109/eScience51609.2021.00031 (preprint: arXiv 2109.12060)
(This paper is green open access)
Abstract: The Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) will soon carry out an unprecedented wide, fast, and deep survey of the sky in multiple optical bands. The data from LSST will open up a new discovery space in astronomy and cosmology, simultaneously providing clues toward addressing burning issues of the day, such as the origin of dark energy and and the nature of dark matter, while at the same time yielding data that will, in turn, pose fresh new questions. To prepare for the imminent arrival of this remarkable data set, it is crucial that the associated scientific communities be able to develop the software needed to analyze it. Computational power now available allows us to generate synthetic data sets that can be used as a realistic training ground for such an effort. This effort raises its own challenges—the need to generate very large simulations of the night sky, scaling up simulation campaigns to large numbers of compute nodes across multiple computing centers with different architectures, and optimizing the complex workload around memory requirements and widely varying wall clock times. We describe here a large-scale workflow that melds together Python code to steer the workflow, Parsl to manage the large-scale distributed execution of workflow components, and containers to carry out the image simulation campaign across multiple sites. Taking advantage of these tools, we developed an extreme-scale computational framework and used it to simulate five years of observations for 300 square degrees of sky area. We describe our experiences and lessons learned in developing this workflow capability, and highlight how the scalability and portability of our approach enabled us to efficiently execute it on up to 4000 compute nodes on two supercomputers.
Y. Babuji, J. Bryan, R. Chard, K. Chard, I. Foster, B. Galewsky, D. S. Katz, Z. Li, "Federated Function as a Service for eScience," 17th IEEE International Conference on eScience (eScience2021), 2021. https://doi.org/10.1109/eScience51609.2021.00046 (preprint: https://doi.org/10.5281/zenodo.5519584)
(This paper is green open access)
Abstract: The function as a service paradigm aims to abstract the complexities of managing computing infrastructure for users. While adoption in industry has been swift, we have yet to see widespread adoption in academia. This is in part due to barriers such as the need to access large research data, diverse hardware requirements, monolithic code bases, and existing systems available to researchers. We describe funcX, a federated function-as-a-service platform that addresses important requirements for use of FaaS in research computing. We outline how funcX has been used in early science deployments.
E. A. Huerta, A. Khan, X. Huang, M. Tian, M. Levental, R. Chard, W. Wei, M. Heflin, D. S. Katz, V. Kindratenko, D. Mu, B. Blaiszik, I. Foster, "Accelerated, scalable and reproducible AI-driven gravitational wave detection," Nature Astronomy, v.5, pp. 1062–1068, 2021. https://doi.org/10.1038/s41550-021-01405-0
(This paper not open access)
Abstract: The development of reusable artificial intelligence (AI) models for wider use and rigorous validation by the community promises to unlock new opportunities in multi-messenger astrophysics. Here we develop a workflow that connects the Data and Learning Hub for Science, a repository for publishing AI models, with the Hardware-Accelerated Learning (HAL) cluster, using funcX as a universal distributed computing service. Using this workflow, an ensemble of four openly available AI models can be run on HAL to process an entire month’s worth (August 2017) of advanced Laser Interferometer Gravitational-Wave Observatory data in just seven minutes, identifying all four binary black hole mergers previously identified in this dataset and reporting no misclassifications. This approach combines advances in AI, distributed computing and scientific data infrastructure to open new pathways to conduct reproducible, accelerated, data-driven discovery.
K. Chard, Y. Babuji, A. Woodard, B. Clifford, Z. Li, M. Hategan, I. Foster, M. Wilde, D. S. Katz, "Extended Abstract: Productive Parallel Programming with Parsl," ACM SIGAda Ada Letters, v. 40(2), pp. 73–75, 2021. https://doi.org/10.1145/3463478.3463486 (preprint: arXiv:2205.01527)
(This paper not open access)
Abstract: Parsl is a parallel programming library for Python that aims to make it easy to specify parallelism in programs and to realize that parallelism on arbitrary parallel and distributed computing systems. Parsl relies on developers annotating Python functions-wrapping either Python or external applications-to indicate that these functions may be executed concurrently. Developers can then link together functions via the exchange of data. Parsl establishes a dynamic dependency graph and sends tasks for execution on connected resources when dependencies are resolved. Parsl's runtime system enables different compute resources to be used, from laptops to supercomputers, without modification to the Parsl program.
D. S. Katz, J. Carver, N. Chue Hong, S. Gesing, S. Hettrick, T. Honeyman, K. Ram, N. Weber, "Addressing Research Software Sustainability via Institutes," 1st International Workshop on the Body of Knowledge for Software Sustainability (BoKSS'21), in conjunction with 43rd International Conference on Software Engineering (ICSE 2021), 2021. https://doi.org/10.1109/BoKSS52540.2021.00013 (preprint: arXiv 2103.03690)
(This paper is green open access)
Abstract: Research software is essential to modern research, but it requires ongoing human effort to sustain: to continually adapt to changes in dependencies, to fix bugs, and to add new features. Software sustainability institutes, amongst others, develop, maintain, and disseminate best practices for research software sustainability, and build community around them. These practices can both reduce the amount of effort that is needed and create an environment where the effort is appreciated and rewarded. The UK SSI is such an institute, and the US URSSI and the Australian AuSSI are planning to become institutes, and this extended abstract discusses them and the strengths and weaknesses of this approach.
S. Druskat, D. S. Katz, I. T. Todorov, "Research Software Sustainability and Citation," 1st International Workshop on the Body of Knowledge for Software Sustainability (BoKSS'21), in conjunction with 43rd International Conference on Software Engineering (ICSE 2021), 2021. https://doi.org/10.1109/BoKSS52540.2021.00008 (preprint: arXiv 2103.06681)
(This paper is green open access)
Abstract: Software citation contributes to achieving software sustainability in two ways: It provides an impact metric to incentivize stakeholders to make software sustainable. It also provides references to software used in research, which can be reused and adapted to become sustainable. While software citation faces a host of technical and social challenges, community initiatives have defined the principles of software citation and are working on implementing solutions.
J. C. Carver, I. A. Cosden, C. Hill, S. Gesing, D. S. Katz, "Sustaining Research Software via Research Software Engineers and Professional Associations," 1st International Workshop on the Body of Knowledge for Software Sustainability (BoKSS'21), in conjunction with 43rd International Conference on Software Engineering (ICSE 2021), 2021. https://doi.org/10.1109/BoKSS52540.2021.00016 (preprint: arXiv 2103.01880)
(This paper is green open access)
Abstract: Research software is a class of software developed to support research. Today a wealth of such software is created daily in universities, government, and commercial research enterprises worldwide. The sustainability of this software faces particular challenges due, at least in part, to the type of people who develop it. These Research Software Engineers (RSEs) face challenges in developing and sustaining software that differ from those faced by the developers of traditional software. As a result, professional associations have begun to provide support, advocacy, and resources for RSEs. These benefits are critical to sustaining RSEs, especially in environments where their contributions are often undervalued and not rewarded. This paper focuses on how professional associations, such as the United States Research Software Engineer Association (US-RSE), can provide this.
D. S. Katz, M. Gruenpeter, T. Honeyman, "Taking a fresh look at FAIR for research software," Patterns v.2(3), 100222, March 12, 2021. https://doi.org/10.1016/j.patter.2021.100222
(This paper is open access)
Abstract: Software is increasingly essential in most research, and much of this software is developed specifically for and during research. To make this research software findable, accessible, interoperable, and reusable (FAIR), we need to define exactly what FAIR means for research software and acknowledge that software is a living and complex object for which it is impossible to propose one solution that fits all software.
D. S. Katz, N. P. Chue Hong, T. Clark, A. Muench, S. Stall, D. Bouquin, M. Cannon, S. Edmunds, T. Faez, P. Feeney, M. Fenner, M. Friedman, G. Grenier, M. Harrison, J. Heber, A. Leary, C. MacCallum, H. Murray, E. Pastrana, K. Perry, D. Schuster, M. Stockhause, J. Yeston, "Recognizing the value of software: a software citation guide [version 2; peer review: 2 approved]," F1000Research 9:1257, 2021. https://doi.org/10.12688/f1000research.26932.2
(This paper is open access)
Abstract: Software is as integral as a research paper, monograph, or dataset in terms of facilitating the full understanding and dissemination of research. This article provides broadly applicable guidance on software citation for the communities and institutions publishing academic journals and conference proceedings. We expect those communities and institutions to produce versions of this document with software examples and citation styles that are appropriate for their intended audience. This article (and those community-specific versions) are aimed at authors citing software, including software developed by the authors or by others. We also include brief instructions on how software can be made citable, directing readers to more comprehensive guidance published elsewhere. The guidance presented in this article helps to support proper attribution and credit, reproducibility, collaboration and reuse, and encourages building on the work of others to further research.
D. S. Katz, M. Gruenpeter, T. Honeyman, L. Hwang, M. D. Wilkinson, V. Sochat, H. Anzt, C. Goble, for FAIR4RS Subgroup 1, "A Fresh Look at FAIR for Research Software," arXiv 2101.10883 [cs.SE], 2021.
(This paper is open access)
Abstract: This document captures the discussion and deliberation of the FAIR for Research Software (FAIR4RS) subgroup that took a fresh look at the applicability of the FAIR Guiding Principles for scientific data management and stewardship for research software. We discuss the vision of research software as ideally reproducible, open, usable, recognized, sustained and robust, and then review both the characteristic and practiced differences of research software and data. This vision and understanding of initial conditions serves as a backdrop for an attempt at translating and interpreting the guiding principles to more fully align with research software. We have found that many of the principles remained relatively intact as written, as long as considerable interpretation was provided. This was particularly the case for the "Findable" and "Accessible" foundational principles. We found that "Interoperability" and "Reusability" are particularly prone to a broad and sometimes opposing set of interpretations as written. We propose two new principles modeled on existing ones, and provide modified guiding text for these principles to help clarify our final interpretation. A series of gaps in translation were captured during this process, and these remain to be addressed. We finish with a consideration of where these translated principles fall short of the vision laid out in the opening.
D. S. Katz, K. McHenry, J. S. Lee, "Research Software Sustainability: Lessons Learned at NCSA," 54th Hawaii International Conference on System Sciences, pp. 7249-7256, 2021. http://hdl.handle.net/10125/71494
(This paper is open access)
Abstract: This paper discusses why research software is important, and what sustainability means in this context. It then talks about how research software sustainability can be achieved, and what our experiences at NCSA have been using specific examples, what we have learned from this, and how we think these lessons can help others.
Task Force on Best Practices for Software Registries: A. Monteil, A. Gonzalez-Beltran, A. Ioannidis, A. Allen, A. Lee, A. Bandrowski, B. E. Wilson, B. Mecum, C. F. Du, C. Robinson, D. Garijo, D. S. Katz, D. Long, G. Milliken, H. Ménager, J. Hausman, J. H. Spaaks, K. Fenlon, K. Vanderbilt, L. Hwang, L. Davis, M. Fenner, M. R. Crusoe, M. Hucka, M. Wu, N. Chue Hong, P. Teuben, S. Stall, S. Druskat, T. Carnevale, T. Morrell, "Nine Best Practices for Research Software Registries and Repositories: A Concise Guide," arXiv 2012.13117 [cs.DL], 2020.
(This paper is open access)
Abstract: Scientific software registries and repositories serve various roles in their respective disciplines. These resources improve software discoverability and research transparency, provide information for software citations, and foster preservation of computational methods that might otherwise be lost over time, thereby supporting research reproducibility and replicability. However, developing these resources takes effort, and few guidelines are available to help prospective creators of registries and repositories. To address this need, we present a set of nine best practices that can help managers define the scope, practices, and rules that govern individual registries and repositories. These best practices were distilled from the experiences of the creators of existing resources, convened by a Task Force of the FORCE11 Software Citation Implementation Working Group during the years 2019-2020. We believe that putting in place specific policies such as those presented here will help scientific software registries and repositories better serve their users and their disciplines.
C. Jay, R. Haines, D. S. Katz, "Software must be recognised as an important output of scholarly research", arXiv 2011.07571 [cs.CY], 2020.
(This paper is open access)
Abstract: Software now lies at the heart of scholarly research. Here we argue that as well as being important from a methodological perspective, software should, in many instances, be recognised as an output of research, equivalent to an academic paper. The article discusses the different roles that software may play in research and highlights the relationship between software and research sustainability and reproducibility. It describes the challenges associated with the processes of citing and reviewing software, which differ from those used for papers. We conclude that whilst software outputs do not necessarily fit comfortably within the current publication model, there is a great deal of positive work underway that is likely to make an impact in addressing this.
R. C. Vandewalle, W. C. Barley, A. Padmanabhan, D. S. Katz, and S. Wang, "Understanding the multifaceted geospatial software ecosystem: a survey approach," International Journal of Geographical Information Science, v.35(11), pp. 2168–2186, 2021. https://doi.org/10.1080/13658816.2020.1831514
(This paper is open access)
Abstract: Understanding the characteristics of the rapidly evolving geospatial software ecosystem in the United States is critical to enable convergence research and education that are dependent on geospatial data and software. This paper describes a survey approach to better understand geospatial use cases, software and tools, and limitations encountered while using and developing geospatial software. The survey was broadcast through a variety of geospatial-related academic mailing lists and listservs. We report both quantitative responses and qualitative insights. As 42% of respondents indicated that they viewed their work as limited by inadequacies in geospatial software, ample room for improvement exists. In general, respondents expressed concerns about steep learning curves and insufficient time for mastering geospatial software, and often limited access to high-performance computing resources. If adequate efforts were taken to resolve software limitations, respondents believed they would be able to better handle big data, cover broader study areas, integrate more types of data, and pursue new research. Insights gained from this survey play an important role in supporting the conceptualization of a national geospatial software institute in the United States with the aim to drastically advance the geospatial software ecosystem to enable broad and significant research and education advances.
E. A. Huerta, A. Khan, E. Davis, C. Bushell, W. D. Gropp, D. S. Katz, V. Kindratenko, S. Koric, W. T. C. Kramer, B. McGinty, K. McHenry, and A. Saxton, "Convergence of artificial intelligence and high performance computing on NSF-supported cyberinfrastructure," Journal of Big Data, 7:88, 2020. https://doi.org/10.1186/s40537-020-00361-2
(This paper is open access)
Abstract: Significant investments to upgrade and construct large-scale scientific facilities demand commensurate investments in R&D to design algorithms and computing approaches to enable scientific and engineering breakthroughs in the big data era. Innovative Artificial Intelligence (AI) applications have powered transformational solutions for big data challenges in industry and technology that now drive a multi-billion dollar industry, and which play an ever increasing role shaping human social patterns. As AI continues to evolve into a computing paradigm endowed with statistical and mathematical rigor, it has become apparent that single-GPU solutions for training, validation, and testing are no longer sufficient for computational grand challenges brought about by scientific facilities that produce data at a rate and volume that outstrip the computing capabilities of available cyberinfrastructure platforms. This realization has been driving the confluence of AI and high performance computing (HPC) to reduce time-to-insight, and to enable a systematic study of domain-inspired AI architectures and optimization schemes to enable data-driven discovery. In this article we present a summary of recent developments in this field, and describe specific advances that authors in this article are spearheading to accelerate and streamline the use of HPC platforms to design and apply accelerated AI algorithms in academia and industry.
L. J. Garcia Castro, M. Barker, N. Chue Hong, F. Psomopoulos, J. Harrow, D. S. Katz, M. Kuzak, P. A. Martinez, A. Via, "Software as a first-class citizen in research," First virtual workshop on Research data* management for Linked Open Science (DaMaLOS 2020), 2020. https://doi.org/10.4126/FRL01-006423290
(This paper is open access)
Abstract: In recent years the importance of software in research has become increasingly recognized by the research community. This journey still has a long way to go. Research data is currently backed by a variety of efforts to implement and make FAIR principles a reality, complemented by Data Management Plans. Both FAIR data principles and management plans offer elements that could be useful for research software but none of them can be directly applied; in both cases there is a need for adaptation and then adoption. In this position paper we discuss current efforts around FAIR for research software that will also support the advancement of Software Management Plans. In turn, use of SMPs encourages researchers to make their datasets FAIR.
D. S. Katz, S. Malik, M. S. Neubauer, G. A. Stewart, K. A. Assamagan, E. A. Becker, N. P. Chue Hong, I. A. Cosden, S. Meehan, E. J. W. Moyse, A. M. Price-Whelan, E. Sexton-Kennedy, M. O. Evans, M. Feickert, C. Lange, K. Lieret, R. Quick, A. Sánchez Pineda, and C. Tunnell, "Software Sustainability & High Energy Physics", 2020. arXiv:2010.05102 [hep-ex] & https://doi.org/10.5281/zenodo.4082137
(This paper is open access)
Abstract: New facilities of the 2020s, such as the High Luminosity Large Hadron Collider (HL-LHC), will be relevant through at least the 2030s. This means that their software efforts and those that are used to analyze their data need to consider sustainability to enable their adaptability to new challenges, longevity, and efficiency, over at least this period. This will help ensure that this software will be easier to develop and maintain, that it remains available in the future on new platforms, that it meets new needs, and that it is as reusable as possible. This report discusses a virtual half-day workshop on “Software Sustainability and High Energy Physics” that aimed 1) to bring together experts from HEP as well as those from outside to share their experiences and practices, and 2) to articulate a vision that helps the Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP) to create a work plan to implement elements of software sustainability. Software sustainability practices could lead to new collaborations, including elements of HEP software being directly used outside the field, and, as has happened more frequently in recent years, to HEP developers contributing to software developed outside the field rather than reinventing it. A focus on and skills related to sustainable software will give HEP software developers an important skill that is essential to careers in the realm of software, inside or outside HEP. The report closes with recommendations to improve software sustainability in HEP, aimed at the HEP community via IRIS-HEP and the HEP Software Foundation (HSF).
C. Jay, R. Haines, D. S. Katz, J. C. Carver, S. Gesing, S. R. Brandt, J. Howison, A. Dubey, J. C. Phillips, H. Wan, M. J. Turk, "The challenges of theory-software translation [version 1; peer review: 2 approved, 1 approved with reservations]", F1000Research, 9:1192, 2020. https://doi.org/10.12688/f1000research.25561.1
(This paper is open access)
Abstract: Background: Software is now ubiquitous within research. In addition to the general challenges common to all software development projects, research software must also represent, manipulate, and provide data for complex theoretical constructs. Ensuring this process of theory-software translation is robust is essential to maintaining the integrity of the science resulting from it, and yet there has been little formal recognition or exploration of the challenges associated with it.
Methods: We thematically analyse the outputs of the discussion sessions at the Theory-Software Translation Workshop 2019, where academic researchers and research software engineers from a variety of domains, and with particular expertise in high performance computing, explored the process of translating between scientific theory and software.
Results: We identify a wide range of challenges to implementing scientific theory in research software and using the resulting data and models for the advancement of knowledge. We categorise these within the emergent themes of design, infrastructure, and culture, and map them to associated research questions.
Conclusions: Systematically investigating how software is constructed and its outputs used within science has the potential to improve the robustness of research software and accelerate progress in its development. We propose that this issue be examined within a new research area of theory-software translation, which would aim to significantly advance both knowledge and scientific practice.
J. A. Schiller, R. Toro Santamaria, A. Shah, M. Surana, K. Zhang, M. Robertson, K. Miller, K. Cruse, K. Liu, B. Seong, C. Seol, I. Foster, B. Blaiszik, B. Galewsky, D. K. Adams, D. S. Katz, P. M. Ferreira, E. Ertekin, and S. Tawfick, "Crowd-Sourced Data and Analysis Tools for Advancing the Chemical Vapor Deposition of Graphene: Implications for Manufacturing", ACS Applied Nano Materials, 2020, https://doi.org/10.1021/acsanm.0c02018 (preprint: https://doi.org/10.31224/osf.io/7mwgd)
(This paper is not open access)
Abstract: Industrial production of graphene by chemical vapor deposition (CVD) requires more than the ability to synthesize large domain, high quality graphene in a lab reactor. The integration of graphene in the fabrication process of electronic devices requires the cost-effective and environmentally-friendly production of graphene on dielectric substrates, but current approaches can only produce graphene on metal catalysts. Sustainable manufacturing of graphene should also conserve the catalyst and reaction gases, but today the metal catalysts are typically dissolved after synthesis. Progress toward these objectives is hindered by the hundreds of coupled synthesis parameters that can strongly affect CVD of low-dimensional materials, and poor communication in the published literature of the rich experimental data that exists in individual laboratories. We report here on a platform "Graphene – Recipes for synthesis of high quality material" (Gr-ResQ: pronounced graphene rescue), that includes powerful new tools for data-driven graphene synthesis. At the core of Gr-ResQ is a crowd-sourced database of CVD synthesis recipes and associated experimental results. The database captures ~300 parameters ranging from synthesis conditions like catalyst material and preparation steps, to ambient lab temperature and reactor details, as well as resulting Raman spectra and microscopy images. These parameters are carefully selected to unlock the potential of machine-learning models to advance synthesis. A suite of associated tools enable fast, automated and standardized processing of Raman spectra and scanning electron microscopy images. To facilitate community-based efforts, Gr-ResQ provides tools for cyber-physical collaborations among research groups, allowing experiments to be designed, executed, and analyzed by different teams. Gr-ResQ also allows publication and discovery of recipes via the Materials Data Facility (MDF), which assigns each recipe a unique identifier when published and collects parameters in a search index. We envision that this holistic approach to data-driven synthesis can accelerate CVD recipe discovery and production control, and open opportunities for advancing not only graphene, but also many other 1D and 2D materials.
J. Stubbs, S. Marru, D. Mejia, D. S. Katz, K. Chard, M. Dahan, M. Pierce, M. Zentner, "Toward Interoperable Cyberinfrastructure: Common Descriptions for Computational Resources and Applications," Practice and Experience in Advanced Research Computing (PEARC '20), pp. 522-525, 2020. https://doi.org/10.1145/3311790.3400848 (Preprint: arXiv:2107.00417 [cs.DC])
(This paper is not open access)
Abstract: The user-facing components of the Cyberinfrastructure (CI) ecosystem, science gateways and scientific workflow systems, share a common need of interfacing with physical resources (storage systems and execution environments) to manage data and execute codes (applications). However, there is no uniform, platform-independent way to describe either the resources or the applications. To address this, we propose uniform semantics for describing resources and applications that will be relevant to a diverse set of stakeholders. We sketch a solution to the problem of a common description and catalog of resources: we describe an approach to implementing a resource registry for use by the community and discuss potential approaches to some long-term challenges. We conclude by looking ahead to the application description language.
J. Cohen, D. S. Katz, M. Barker, N. Chue Hong, R. Haines, C. Jay, "The Four Pillars of Research Software Engineering," IEEE Software, v.38(1), pp. 97-105, 2020. 10.1109/MS.2020.2973362 (Preprint: arXiv:2002.01035 [cs.SE])
(This paper is green open access)
Abstract: Building software that can support the huge growth in data and computation required by modern research needs individuals with increasingly specialist skill sets that take time to develop and maintain. The Research Software Engineering movement, which started in the UK and has been built up over recent years, aims to recognise and support these individuals. Why does research software matter to professional software development practitioners outside the research community? Research software can have great impact on the wider world and recent progress means the area can now be considered as a more realistic option for a professional software development career. In this article we present a structure, along with supporting evidence of real-world activities, that defines four elements that we believe are key to providing comprehensive and sustainable support for Research Software Engineering. We also highlight ways that the wider developer community can learn from, and engage with, these activities.
C. Jay, R. Haines, D. S. Katz, J. Carver, J. C. Phillips, A. Dubey, S. Gesing, M. Turk, H. Wan, H. van Dam, J. Howison, V. Morozov, S. R. Brandt, "Theory-Software Translation: Research Challenges and Future Directions," arXiv 1910.09902 [cs.SE], 2019.
(This paper is open access)
Abstract: The Theory-Software Translation Workshop, held in New Orleans in February 2019, explored in depth the process of both instantiating theory in software – for example, implementing a mathematical model in code as part of a simulation – and using the outputs of software – such as the behavior of a simulation – to advance knowledge. As computation within research is now ubiquitous, the workshop provided a timely opportunity to reflect on the particular challenges of research software engineering – the process of developing and maintaining software for scientific discovery. In addition to the general challenges common to all software development projects, research software additionally must represent, manipulate, and provide data for complex theoretical constructs. Ensuring this process is robust is essential to maintaining the integrity of the science resulting from it, and the workshop highlighted a number of areas where the current approach to research software engineering would benefit from an evidence base that could be used to inform best practice.
The workshop brought together expert research software engineers and academics to discuss the challenges of Theory-Software Translation over a two-day period. This report provides an overview of the workshop activities, and a synthesises of the discussion that was recorded. The body of the report presents a thematic analysis of the challenges of Theory-Software Translation as identified by workshop participants, summarises these into a set of research areas, and provides recommendations for the future direction of this work.
E. A. Huerta, G. Allen, I. Andreoni, J. M. Antelis, E. Bachelet, G. B. Berriman, F. B. Bianco, R. Biswas, M. Carrasco Kind, K. Chard, M. Cho, P. S. Cowperthwaite, Z. B. Etienne, M. Fishbach, F. Forster, D. George, T. Gibbs, M. Graham, W. Gropp, R. Gruendl, A. Gupta, R. Haas, S. Habib, E. Jennings, M. W. G. Johnson, E. Katsavounidis, D. S. Katz, A. Khan, V. Kindratenko, W. T. C. Kramer, X. Liu, A. Mahabal, Z. Marka, K. McHenry, J. M. Miller, C. Moreno, M. S. Neubauer, S. Oberlin, A. R. Olivas Jr, D. Petravick, A. Rebei, S. Rosofsky, M. Ruiz, A. Saxton, B. F. Schutz, A. Schwing, E. Seidel, S. L. Shapiro, H. Shen, Y. Shen, L. P. Singer, B. M. Sipocz, L. Sun, J. Towns, A. Tsokaros, W. Wei, J. Wells, T. J. Williams, J. Xiong, Z. Zhao, "Enabling real-time multi-messenger astrophysics discoveries with deep learning," Nature Reviews Physics, v.1, pp. 600–608, 2019. https://doi.org/10.1038/s42254-019-0097-4
(This paper is not open access)
Abstract: Multi-messenger astrophysics is a fast-growing, interdisciplinary field that combines data, which vary in volume and speed of data processing, from many different instruments that probe the Universe using different cosmic messengers: electromagnetic waves, cosmic rays, gravitational waves and neutrinos. In this Expert Recommendation, we review the key challenges of real-time observations of gravitational wave sources and their electromagnetic and astroparticle counterparts, and make a number of recommendations to maximize their potential for scientific discovery. These recommendations refer to the design of scalable and computationally efficient machine learning algorithms; the cyber-infrastructure to numerically simulate astrophysical sources, and to process and interpret multi-messenger astrophysics data; the management of gravitational wave detections to trigger real-time alerts for electromagnetic and astroparticle follow-ups; a vision to harness future developments of machine learning and cyber-infrastructure resources to cope with the big-data requirements; and the need to build a community of experts to realize the goals of multi-messenger astrophysics.
D. S. Katz, D. Bouquin, N. P. Chue Hong, J. Hausman, C. Jones, D. Chivvis, T. Clark, M. Crosas, S. Druskat, M. Fenner, T. Gillespie, A. Gonzalez-Beltran, M. Gruenpeter, T. Habermann, R. Haines, M. Harrison, E. Henneken, L. Hwang, M. B. Jones, A. A. Kelly, D. N. Kennedy, K. Leinweber, F. Rios, C. B. Robinson, I. Todorov, M. Wu, Q. Zhang, "Software Citation Implementation Challenges", arXiv 1905.08674 [cs.CY], 2019.
(This paper is open access)
Abstract: The main output of the FORCE11 Software Citation working group was a paper on software citation principles published in September 2016. This paper laid out a set of six high-level principles for software citation (importance, credit and attribution, unique identification, persistence, accessibility, and specificity) and discussed how they could be used to implement software citation in the scholarly community. In a series of talks and other activities, we have promoted software citation using these increasingly accepted principles. At the time the initial paper was published, we also provided the following (old) guidance and examples on how to make software citable, though we now realize there are unresolved problems with that guidance. The purpose of this document is to provide an explanation of current issues impacting scholarly attribution of research software, organize updated implementation guidance, and identify where best practices and solutions are still needed.
Z. Zhang, L. Huang, R. Huang, W. Xu and D. S. Katz, "Quantifying the Impact of Memory Errors in Deep Learning," IEEE International Conference on Cluster Computing (CLUSTER), pp. 1-12, 2019. https://doi.org/10.1109/CLUSTER.2019.8890989
(This paper is not open access)
Abstract: The use of deep learning (DL) on HPC resources has become common as scientists explore and exploit DL methods to solve domain problems. On the other hand, in the coming exascale computing era, a high error rate is expected to be problematic for most HPC applications. The impact of errors on DL applications, especially DL training, remains unclear given their stochastic nature. In this paper, we focus on understanding DL training applications on HPC in the presence of silent data corruption. Specifically, we design and perform a quantification study with three representative applications by manually injecting silent data corruption errors (SDCs) across the design space and compare training results with the error-free baseline. The results show only 0.61-1.76% of SDCs cause training failures, and taking the SDC rate in modern hardware into account, the actual chance of a failure is one in thousands to millions of executions. With this quantitatively measured impact, computing centers can make rational design decisions based on their application portfolio, the acceptable failure rate, and financial constraints; for example, they might determine their confidence in the correctness of training results performed on processors without error correction code (ECC) RAM. We also discover that over 75-90% of the SDCs that cause catastrophic errors can be easily detected by a training loss in the next iteration. Thus we propose this error-aware software solution to correct catastrophic errors, as it has significantly lower time and space overhead compared to algorithm-based fault-tolerance (ABFT) and ECC.
D. S. Katz, K. McHenry, C. Reinking, R. Haines, "Research Software Development & Management in Universities: Case Studies from Manchester's RSDS Group, Illinois' NCSA, and Notre Dame's CRC," Intl. Work. on Soft. Eng. for Science (SE4Science), 2019. https://doi.org/10.1109/SE4Science.2019.00009 (Preprint: arXiv 1903.00732)
(This paper is green open access)
Abstract: Modern research in the sciences, engineering, humanities, and other fields depends on software, and specifically, research software. Much of this research software is developed in universities, by faculty, postdocs, students, and staff. In this paper, we focus on the role of university staff. We examine three different, independently-developed models under which these staff are organized and perform their work, and comparatively analyze these models and their consequences on the staff and on the software, considering how the different models support software engineering practices and processes. This information can be used by software engineering researchers to understand the practices of such organizations and by universities who want to set up similar organizations and to better produce and maintain research software.
K. Chard, Y. Babuji, A. Woodard, B. Clifford, Z. Li, R. Kumar, M. Wilde, D. S. Katz, J. Wozniak, I. Foster, "Scalable Parallel Programming in Python with Parsl," Practice and Experience in Advanced Research Computing (PEARC’19), 2019. https://doi.org/10.1145/3332186.3332231
(This paper is not open access)
Abstract: Python is increasingly the lingua franca of scientific computing. It is used as a higher level language to wrap lower-level libraries and to compose scripts from various independent components. However, scaling and moving Python programs from laptops to supercomputers remains a challenge. Here we present Parsl, a parallel scripting library for Python. Parsl makes it straightforward for developers to implement parallelism in Python by annotating functions that can be executed asynchronously and in parallel, and to scale analyses from a laptop to thousands of nodes on a supercomputer or distributed system. We examine how Parsl is implemented, focusing on syntax and usage. We describe two scientific use cases in which Parsl’s intuitive and scalable parallelism is used.
A. E. Ahmed, J. Heldenbrand, Y. Asmann, F. M. Fadlelmola, D. S. Katz, K. Kendig, M. C. Kendzior, T. Li, Y. Ren, E. Rodriguez, M. R. Weber, J. M. Wozniak, J. Zermeno, L. S. Mainzer, "Managing genomic variant calling workflows with Swift/T," PLOS ONE v.14(7), pp. e0211608, 2019. 10.1371/journal.pone.0211608 (Preprint: https://doi.org/10.1101/524645)
(This paper is open access)
Abstract: Bioinformatics research is frequently performed using complex workflows with multiple steps, fans, merges, and conditionals. This complexity makes management of the workflow difficult on a computer cluster, especially when running in parallel on large batches of data: hundreds or thousands of samples at a time. Scientific workflow management systems could help with that. Many are now being proposed, but is there yet the "best" workflow management system for bioinformatics? Such a system would need to satisfy numerous, sometimes conflicting requirements: from ease of use, to seamless deployment at peta- and exa-scale, and portability to the cloud. We evaluated Swift/T as a candidate for such role by implementing a primary genomic variant calling workflow in the Swift/T language, focusing on workflow management, performance and scalability issues that arise from production-grade big data genomic analyses. In the process we introduced novel features into the language, which are now part of its open repository. Additionally, we formalized a set of design criteria for quality, robust, maintainable workflows that must function at-scale in a production setting, such as a large genomic sequencing facility or a major hospital system. The use of Swift/T conveys two key advantages. (1) It operates transparently in multiple cluster scheduling environments (PBS Torque, SLURM, Cray aprun environment, etc.), thus a single workflow is trivially portable across numerous clusters. (2) The leaf functions of Swift/T permit developers to easily swap executables in and out of the workflow, which makes it easy to maintain and to request resources optimal for each stage of the pipeline. While Swift/T’s data-level parallelism eliminates the need to code parallel analysis of multiple samples, it does make debugging more difficult, as is common for implicitly parallel code. Nonetheless, the language gives users a powerful and portable way to scale up analyses in many computing architectures. The code for our implementation of a variant calling workflow using Swift/T can be found on GitHub at https://github.com/ncsa/Swift-T-Variant-Calling, with full documentation provided at http://swift-t-variant-calling.readthedocs.io/en/latest/.
Y. Babuji, A. Woodard, Z. Li, D. S. Katz, B. Clifford, R. Kumar, L. Lacinski, R. Chard, J. M. Wozniak, I. Foster, M. Wilde, K. Chard, "Parsl: Pervasive Parallel Programming in Python," 28th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC). 2019. https://doi.org/10.1145/3307681.3325400 (preprint: arXiv 1905.02158)
(This paper is green open access)
Abstract: High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250 000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.
S. Druskat, D. S. Katz, "Mapping the research software sustainability space," International Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE 6.1), 2018. https://doi.org/10.1109/eScience.2018.00014
(This paper is open access)
Abstract: A growing number of largely uncoordinated initiatives focus on research software sustainability. A comprehensive mapping of the research software sustainability space can help identify gaps in their efforts, track results, and avoid duplication of work. To this end, this paper suggests enhancing an existing schematic of activities in research software sustainability, and formalizing it in a directed graph model. Such a model can be further used to define a classification schema which, applied to research results in the field, can drive the identification of past activities and the planning of future efforts.
J. Cohen, D. S. Katz, M. Barker, R. Haines, N. Chue Hong, "Building a Sustainable Structure for Research Software Engineering Activities," International Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE 6.1), 2018. https://doi.org/10.1109/eScience.2018.00015
(This paper is open access)
Abstract: The profile of research software engineering has been greatly enhanced by developments at institutions around the world to form groups and communities that can support effective, sustainable development of research software. We observe, however, that there is still a long way to go to build a clear understanding about what approaches provide the best support for research software developers in different contexts, and how such understanding can be used to suggest more formal structures, models or frameworks that can help to further support the growth of research software engineering. This short paper provides an overview of some preliminary thoughts and proposes an initial high-level framework based on discussions between the authors around the concept of a set of pillars representing key activities and processes that form the core structure of a successful research software engineering offering.
D. S. Katz, S. Druskat, R. Haines, C. Jay, A. Struck, "The State of Sustainable Research Software: Learning from the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1)," Journal of Open Research Software, v.7(1), p.11, 2019. https://doi.org/10.5334/jors.242
(This paper is open access)
Abstract: This paper uses the accepted submissions from the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1) held in Manchester, UK in September 2017 and the speed blogs written during the event to examine the state of research software. It presents a schematic of the space, then examines coverage in terms of topics, actors, actees, and themes by both the submissions and the blogs.
HEP Software Foundation: A. A. Alves Jr, G. Amadio, N. Anh-Ky, L. Aphecetche, J. Apostolakis, M. Asai, L. Atzori, M. Babik, G. Bagliesi, M. Bandieramonte, M. Barisits, L. A. T. Bauerdick, S. Belforte, D. Benjamin, C. Bernius, W. Bhimji, R. M. Bianchi, I. Bird, C. Biscarat, J. Blomer, K. Bloom, T. Boccali, B. Bockelman, T. Bold, D. Bonacorsi, A. Boveia, C. Bozzi, M. Bracko, D. Britton, A. Buckley, P. Buncic, P. Calafiura, S. Campana, P. Canal, L. Canali, G. Carlino, N. F. Castro, M. Cattaneo, G. Cerminara, P. Chang, J. Chapman, G. Chen, T. Childers, P. Clarke, M. Clemencic, E. Cogneras, I. Collier, G. Corti, G. Cosmo, D. Costanzo, B. Couturier, K. Cranmer, J. Cranshaw, L. Cristella, D. Crooks, S. Crépé-Renaudin, S. Dallmeier-Tiessen, K. De, M. De Cian, A. Di Girolamo, G. Dimitrov, C. Doglioni, A. Dotti, D. Duellmann, L. Duflot, D. Dykstra, K. Dziedziniewicz-Wojcik, A. Dziurda, U. Egede, P. Elmer, J. Elmsheuser, V. D. Elvira, G. Eulisse, T. Ferber, A. Filipcic, I. Fisk, C. Fitzpatrick, J. Flix, A. Formica, A. Forti, F. Gaede, G. Ganis, R. Gardner, V. Garonne, A. Gellrich, K. Genser, S. George, F. Geurts, A. Gheata, M. Gheata, F. Giacomini, S. Giagu, M. Giffels, D. Gingrich, M. Girone, V. V. Gligorov, I. Glushkov, W. Gohn, J. B. Gonzalez Lopez, I. González Caballero, J. R. González Fernández, G. Govi, C. Grandi, H. Grasland, H. Gray, L. Grillo, W. Guan, O. Gutsche, V. Gyurjyan, A. Hanushevsky, F. Hariri, T. Hartmann, J. Harvey, T. Hauth, B. Hegner, B. Heinemann, L. Heinrich, J. M. Hernández, M. Hildreth, S. Höche, M. Hodgkinson, P. Hristov, X. Huang, V. N. Ivanchenko, T. Ivanov, B. Jashal, B. Jayatilaka, R. Jones, M. Jouvin, S. Y. Jun, M. Kagan, C. W. Kalderon, E. Karavakis, D. S. Katz, D. Kcira, B. P. Kersevan, M. Kirby, A. Klimentov, M. Klute, I. Komarov, P. Koppenburg, J. Kowalkowski, L. Kreczko, T. Kuhr, R. Kutschke, V. Kuznetsov, W. Lampl, E. Lancon, D. Lange, M. Lassnig, P. Laycock, C. Leggett, J. Letts, B. Lewendel, T. Li, G. Lima, J. Linacre, T. Linden, G. Lo Presti, S. Lopienski, P. Love, Z. L. Marshall, E. Martelli, S. Martin-Haugh, P. Mato, K. Mazumdar, T. McCauley, J. McFayden, S. McKee, A. McNab, H. Meinhard, D. Menasce, P. Mendez Lorenzo, A. Serhan Mete, M. Michelotto, J. Mitrevski, L. Moneta, B. Morgan, R. Mount, E. Moyse, S. Murray, M. S. Neubauer, S. Novaes, M. Novak, A. Oyanguren, N. Ozturk, A. Pacheco Pages, M. Paganini, J. Pansanel, V. R. Pascuzzi, A. Pearce, B. Pearson, K. Pedro, G. Perdue, A. Perez-Calero Yzquierdo, L. Perrozzi, T. Petersen, M. Petric, J. Piedra, L. Piilonen, D. Piparo, W. Pokorski, F. Polci, K. Potamianos, F. Psihas, G. Raven, J. Reuter, A. Ribon, M. Ritter, J. Robinson, E. Rodrigues, S. Roiser, D. Rousseau, G. Roy, A. Sailer, T. Sakuma, R. Santana, A. Sartirana, H. Schellman, J. Schovancová, S. Schramm, M. Schulz, A. Sciabà, S. Seidel, S. Sekmen, C. Serfon, H. Severini, E. Sexton-Kennedy, M. Seymour, I. Shapoval, J. Shiers, J.-G. Shiu, H. Short, G. P. Siroli, S. Skipsey, T. Smith, S. Snyder, M. D. Sokoloff, H. Stadie, G. Stark, G. Stewart, A. Sánchez-Hernández, J. Templon, G. Tenaglia, V. Tsulaia, C. Tunnell, E. Vaandering, A. Valassi, S. Vallecorsa, L. Valsan, P. Van Gemmeren, R. Vernet, B. Viren, J.-R. Vlimant, C. Voss, C. Vuosalo, C. Vázquez Sierra, R. Wartel, T. Wenaus, S. Wenzel, F. Winklmeier, C. Wissing, F. Wuerthwein, B. Wynne, Z. Xiaomei, W. Yang and E. Yazgan, "A Roadmap for HEP Software and Computing R&D for the 2020s," Computing and Software for Big Science, v.3(7), December 2019. 10.1007/s41781-018-0018-8
(This paper is open access)
Abstract: Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.
D. Sholler, K. Ram, C. Boettiger, D. S. Katz, "Enforcing public data archiving policies in academic publishing: A study of ecology journals," Big Data & Society, March 25, 2019. https://doi.org/10.1177/2053951719836258
(This paper is open access)
Abstract: To improve the quality and efficiency of research, groups within the scientific community seek to exploit the value of data sharing. Funders, institutions, and specialist organizations are developing and implementing strategies to encourage or mandate data sharing within and across disciplines, with varying degrees of success. Academic journals in ecology and evolution have adopted several types of public data archiving policies requiring authors to make data underlying scholarly manuscripts freely available. The effort to increase data sharing in the sciences is one part of a broader “data revolution” that has prompted discussion about a paradigm shift in scientific research. Yet anecdotes from the community and studies evaluating data availability suggest that these policies have not obtained the desired effects, both in terms of quantity and quality of available datasets. We conducted a qualitative, interview-based study with journal editorial staff and other stakeholders in the academic publishing process to examine how journals enforce data archiving policies. We specifically sought to establish who editors and other stakeholders perceive as responsible for ensuring data completeness and quality in the peer review process. Our analysis revealed little consensus with regard to how data archiving policies should be enforced and who should hold authors accountable for dataset submissions. Themes in interviewee responses included hopefulness that reviewers would take the initiative to review datasets and trust in authors to ensure the completeness and quality of their datasets. We highlight problematic aspects of these thematic responses and offer potential starting points for improvement of the public data archiving process.
P. Chang, G. Allen, W. Anderson, F. B. Bianco, J. S. Bloom, P. R. Brady, A. Brazier, S. B. Cenko, S. M. Couch, T. DeYoung, E. Deelman, Z. B. Etienne, R. J. Foley, D. B Fox, V. Z. Golkhou, D. R. Grant, C. Hanna, K. Holley-Bockelmann, D. A. Howell, E. A. Huerta, M. W. G. Johnson, M. Juric, D. L. Kaplan, D. S. Katz, A. Keivani, W. Kerzendorf, C. Kopper, M. T. Lam, L. Lehner, Z. Marka, S. Marka, J. Nabrzyski, G. Narayan, B. W. O'Shea, D. Petravick, R. Quick, R. A. Street, I. Taboada, F. Timmes, M. J. Turk, A. Weltman, Z. Zhang, "Cyberinfrastructure Requirements to Enhance Multi-messenger Astrophysics," arXiv: 1903.04590 [astro-ph.IM], 2019.
(This paper is open access)
Abstract: The identification of the electromagnetic counterpart of the gravitational wave event, GW170817, and discovery of neutrinos and gamma-rays from TXS 0506+056 heralded the new era of multi-messenger astrophysics. As the number of multi-messenger events rapidly grow over the next decade, the cyberinfrastructure requirements to handle the increase in data rates, data volume, need for event follow up, and analysis across the different messengers will also explosively grow. The cyberinfrastructure requirements to enhance multi-messenger astrophysics will both be a major challenge and opportunity for astronomers, physicists, computer scientists and cyberinfrastructure specialists. Here we outline some of these requirements and argue for a distributed cyberinfrastructure institute for multi-messenger astrophysics to meet these challenges.
D. S. Katz, P. Aerts, N. P. Chue Hong, A. Dubey, S. Gesing, H. J. Neeman, D. E. Pearah, "Sustaining Research Software: an SC18 Panel," arXiv: 1902.08942 [cs.SE], 2019.
(This paper is open access)
Abstract: Many science advances have been possible thanks to the use of research software, which has become essential to advancing virtually every Science, Technology, Engineering and Mathematics (STEM) discipline and many non-STEM disciplines including social sciences and humanities. And while much of it is made available under open source licenses, work is needed to develop, support, and sustain it, as underlying systems and software as well as user needs evolve. In addition, the changing landscape of high-performance computing (HPC) platforms, where performance and scaling advances are ever more reliant on software and algorithm improvements as we hit hardware scaling barriers, is causing renewed tension between sustainability of software and its performance. We must do more to highlight the trade-off between performance and sustainability, and to emphasize the need for sustainability given the fact that complex software stacks don't survive without frequent maintenance; made more difficult as a generation of developers of established and heavily-used research software retire. Several HPC forums are doing this, and it has become an active area of funding as well. In response, the authors organized and ran a panel at the SC18 conference. The objectives of the panel were to highlight the importance of sustainability, to illuminate the tension between pure performance and sustainability, and to steer SC community discussion toward understanding and addressing this issue and this tension. The outcome of the discussions, as presented in this paper, can inform choices of advance compute and data infrastructures to positively impact future research software and future research.
E. A. Huerta, R. Haas, S. Jha, M. Neubauer, D. S. Katz, "Supporting High‐Performance and High‐Throughput Computing for Experimental Science," Computing and Software for Big Science, v.3(5), 2019. 10.1007/s41781-019-0022-7
(This paper is not open access)
Abstract: The advent of experimental science facilities—instruments and observatories, such as the Large Hadron Collider, the Laser Interferometer Gravitational Wave Observatory, and the upcoming Large Synoptic Survey Telescope—has brought about challenging, large-scale computational and data processing requirements. Traditionally, the computing infrastructure to support these facility’s requirements were organized into separate infrastructure that supported their high-throughput needs and those that supported their high-performance computing needs. We argue that to enable and accelerate scientific discovery at the scale and sophistication that is now needed, this separation between high-performance computing and high-throughput computing must be bridged and an integrated, unified infrastructure provided. In this paper, we discuss several case studies where such infrastructure has been implemented. These case studies span different science domains, software systems, and application requirements as well as levels of sustainability. A further aim of this paper is to provide a basis to determine the common characteristics and requirements of such infrastructure, as well as to begin a discussion of how best to support the computing requirements of existing and future experimental science facilities.
S. Olabarriaga, B. Corrie, A. Costa, S. Henwood, N. Chue Hong, M. Barker, A. Treloar, D. S. Katz, S. Gesing, S. Shahand, L. Wyborn, T. Glatard, H. Glaves, K. Jeffery, N. Wilkins-Diehr, "The Global Impact of Science Gateways, Virtual Research Environments and Virtual Laboratories," Future Generation Computing Systems (FGCS), 2019. https://doi.org/10.1016/j.future.2018.12.026
(This paper is not open access)
Abstract: Science gateways, virtual laboratories and virtual research environments are all terms used to refer to community-developed digital environments that are designed to meet a set of needs for a research community. Specifically, they refer to integrated access to research community resources including software, data, collaboration tools, workflows, instrumentation and high-performance computing, usually via Web and mobile applications. Science gateways, virtual laboratories and virtual research environments are enabling significant contributions to many research domains, facilitating more efficient, open, reproducible research in bold new ways. This paper explores the global impact achieved by the sum effects of these programs in increasing research impact, demonstrates their value in the broader digital landscape and discusses future opportunities. This is evidenced through examination of national and international programs in this field.
D. S. Katz, G. Allen, L. A. Barba, D. R. Berg, H. Bik, C. Boettiger, C. L. Borgman, C. T. Brown, S. Buck, R. Burd, A. de Waard, M. P. Eve, B. E. Granger, J. Greenberg, A. Howe, B. Howe, M. Khanna, T. L. Killeen, M. Mayernik, E. McKiernan, C. Mentzel, N. Merchant, K. E. Niemeyer, L. Noren, S. M. Nusser, D. A. Reed, E. Seidel, M. Smith, J. Spies, M. Turk, J. D. Van Horn, and J. Walsh, "The Principles of Tomorrow’s University [version 1; referees: 2 approved]," v.7.1926, F1000Research, 2018. https://doi.org/10.12688/f1000research.17425.1
(This paper is open access)
Abstract: In the 21st Century, research is increasingly data- and computation-driven. Researchers, funders, and the larger community today emphasize the traits of openness and reproducibility. In March 2017, 13 mostly early-career research leaders who are building their careers around these traits came together with ten university leaders (presidents, vice presidents, and vice provosts), representatives from four funding agencies, and eleven organizers and other stakeholders in an NIH- and NSF-funded one-day, invitation-only workshop titled "Imagining Tomorrow's University." Workshop attendees were charged with launching a new dialog around open research - the current status, opportunities for advancement, and challenges that limit sharing.
The workshop examined how the internet-enabled research world has changed, and how universities need to change to adapt commensurately, aiming to understand how universities can and should make themselves competitive and attract the best students, staff, and faculty in this new world. During the workshop, the participants re-imagined scholarship, education, and institutions for an open, networked era, to uncover new opportunities for universities to create value and serve society. They expressed the results of these deliberations as a set of 22 principles of tomorrow's university across six areas: credit and attribution, communities, outreach and engagement, education, preservation and reproducibility, and technologies.
Activities that follow on from workshop results take one of three forms. First, since the workshop, a number of workshop authors have further developed and published their white papers to make their reflections and recommendations more concrete. These authors are also conducting efforts to implement these ideas, and to make changes in the university system. Second, we plan to organise a follow-up workshop that focuses on how these principles could be implemented. Third, we believe that the outcomes of this workshop support and are connected with recent theoretical work on the position and future of open knowledge institutions.
D. S. Katz, L. C. McInnes, D. E. Bernholdt, A. C. Mayes, N. P. Chue Hong, J. Duckles, S. Gesing, M. A. Heroux, S. Hettrick, R. C. Jimenez, M. Pierce, B. Weaver, N. Wilkins-Diehr, "Community Organizations: Changing the Culture in Which Research Software Is Developed and Sustained," Computing in Science and Engineering (CiSE), 2018. https://doi.org/10.1109/MCSE.2018.2883051 (preprint: arXiv: 1811.08473)
(This paper is green open access)
Abstract: Software is the key crosscutting technology that enables advances in mathematics, computer science, and domain-specific science and engineering to achieve robust simulations and analysis for science, engineering, and other research fields. However, software itself has not traditionally received focused attention from research communities; rather, software has evolved organically and inconsistently, with its development largely as by-products of other initiatives. Moreover, challenges in scientific software are expanding due to disruptive changes in computer hardware, increasing scale and complexity of data, and demands for more complex simulations involving multiphysics, multiscale modeling and outer-loop analysis. In recent years, community members have established a range of grass-roots organizations and projects to address these growing technical and social challenges in software productivity, quality, reproducibility, and sustainability. This article provides an overview of such groups and discusses opportunities to leverage their synergistic activities while nurturing work toward emerging software ecosystems.
D. Sholler, K. Ram, C. Boettiger, D. S. Katz, "Enforcing public data archiving policies in academic publishing: A study of ecology journals," arXiv: 1810.13040 [cs.CY], 2018.
(This paper is open access)
Abstract: To improve the quality and efficiency of research, groups within the scientific community seek to exploit the value of data sharing. Funders, institutions, and specialist organizations are developing and implementing strategies to encourage or mandate data sharing within and across disciplines, with varying degrees of success. Academic journals in ecology and evolution have adopted several types of public data archiving policies requiring authors to make data underlying scholarly manuscripts freely available. Yet anecdotes from the community and studies evaluating data availability suggest that these policies have not obtained the desired effects, both in terms of quantity and quality of available datasets. We conducted a qualitative, interview-based study with journal editorial staff and other stakeholders in the academic publishing process to examine how journals enforce data archiving policies. We specifically sought to establish who editors and other stakeholders perceive as responsible for ensuring data completeness and quality in the peer review process. Our analysis revealed little consensus with regard to how data archiving policies should be enforced and who should hold authors accountable for dataset submissions. Themes in interviewee responses included hopefulness that reviewers would take the initiative to review datasets and trust in authors to ensure the completeness and quality of their datasets. We highlight problematic aspects of these thematic responses and offer potential starting points for improvement of the public data archiving process.
Y. Babuji, K. Chard, I. Foster, D. S. Katz, M. Wilde, A. Woodard, J. Wozniak, "Parsl: Scalable Parallel Scripting in Python," Proceedings of the 10th International Workshop on Science Gateways (IWSG), Editors: M. Atkinson, S. Gesing, 2018. CEUR-WS.org, http://ceur-ws.org/Vol-2357/paper11.pdf
(This paper is not open access)
Abstract: Computational and data-driven research practices have significantly changed over the past decade to encompass new analysis models such as interactive and online computing. Science gateways are simultaneously evolving to support this transforming landscape with the aim to enable transparent, scalable execution of a variety of analyses. Science gateways often rely on workflow management systems to represent and execute analyses efficiently and reliably. However, integrating workflow systems in science gateways can be challenging, especially as analyses become more interactive and dynamic, requiring sophisticated orchestration and management of applications and data, and customization for specific execution environments. Parsl (Parallel Scripting Library), a Python library for programming and executing data-oriented workflows in parallel, addresses these problems. Developers simply annotate a Python script with Parsl directives wrapping either Python functions or calls to external applications. Parsl manages the execution of the script on clusters, clouds, grids, and other resources; orchestrates required data movement; and manages the execution of Python functions and external applications in parallel. The Parsl library can be easily integrated into Python-based gateways, allowing for simple management and scaling of workflows.
D. S. Katz "Software Citations and the ACAT Community," ACAT2017 IOP Publishing IOP Conf. Series: Journal of Physics: Conf. Series 1085, 2018. https://doi.org/10.1088/1742-6596/1085/2/022010
(This paper is open access)
Abstract: Software is essential for the bulk of research today. It appears in the research cycle as infrastructure (both inputs and outputs, software obtained from others before the research is performed and software provided to others after the research is complete), as well as being part of the research itself (e.g., new software development). To measure and give credit for software contributions, the simplest path appears to be to overload the current paper citation system so that it also can support citations of software. A multidisciplinary working group built a set of principles for software citation in late 2016. Now, in ACAT 2017 and its proceedings, we want to experimentally encourage those principles to be followed, both to provide credit to the software developers and maintainers in the ACAT community and to try out the process, potentially finding flaws and places where it needs to be improved.
M. D. Hildreth, A. Boehnlein, K. Cranmer, S. Dallmeier, R. Gardner, T. Hacker, L. Heinrich, I. Jimenez, M. Kane, D. S. Katz, T. Malik, C. Maltzahn, M. Neubauer, S. Neubert, Jim Pivarski, E. Sexton, J. Shiers, T. Simko, S. Smith, D. South, A. Verbytskyi, G. Watts, J. Wozniak, "HEP Software Foundation Community White Paper Working Group - Data and Software Preservation to Enable Reuse," arXiv: 1810.01191 [physics.comp-ph], 2018.
(This paper is open access)
Abstract: In this chapter of the High Energy Physics Software Foundation Community Whitepaper, we discuss the current state of infrastructure, best practices, and ongoing developments in the area of data and software preservation in high energy physics. A re-framing of the motivation for preservation to enable re-use is presented. A series of research and development goals in software and other cyberinfrastructure that will aid in the enabling of reuse of particle physics analyses and production software are presented and discussed.
M. L. Mondelli, T. Magalhães, G. Loss, M. Wilde, I. Foster, M. Mattoso, D. S. Katz, H. Barbosa, A. T. R. de Vasconcelos, K. Ocaña, L. M. R. Gadelha Jr., BioWorkbench: a high-performance framework for managing and analyzing bioinformatics experiments," PeerJ, v.6, pp. e55551, 2018. https://doi.org/10.7717/peerj.5551
(This paper is open access)
Abstract: Advances in sequencing techniques have led to exponential growth in biological data, demanding the development of large-scale bioinformatics experiments. Because these experiments are computation- and data-intensive, they require high-performance computing techniques and can benefit from specialized technologies such as Scientific Workflow Management Systems and databases. In this work, we present BioWorkbench, a framework for managing and analyzing bioinformatics experiments. This framework automatically collects provenance data, including both performance data from workflow execution and data from the scientific domain of the workflow application. Provenance data can be analyzed through a web application that abstracts a set of queries to the provenance database, simplifying access to provenance information. We evaluate BioWorkbench using three case studies: SwiftPhylo, a phylogenetic tree assembly workflow; SwiftGECKO, a comparative genomics workflow; and RASflow, a RASopathy analysis workflow. We analyze each workflow from both computational and scientific domain perspectives, by using queries to a provenance and annotation database. Some of these queries are available as a pre-built feature of the BioWorkbench web application. Through the provenance data, we show that the framework is scalable and achieves high-performance, reducing up to 98% of the case studies execution time. We also show how the application of machine learning techniques can enrich the analysis process.
D. S. Katz, N. P. Chue Hong, "Software Citation in Theory and Practice," In: Davenport J., Kauers M., Labahn G., Urban J. (eds), Mathematical Software – ICMS 2018, Lecture Notes in Computer Science, v.10931, Springer, 2018. DOI: 10.1007/978-3-319-96418-8_34 (preprint: arXiv:1807.08149 [cs.CY], 2018.)
(This paper is green open access)
Abstract: In most fields, computational models and data analysis have become a significant part of how research is performed, in addition to the more traditional theory and experiment. Mathematics is no exception to this trend. While the system of publication and credit for theory and experiment (journals and books, often monographs) has developed and has become an expected part of the culture, how research is shared and how candidates for hiring, promotion are evaluated, software (and data) do not have the same history. A group working as part of the FORCE11 community developed a set of principles for software citation that fit software into the journal citation system, allow software to be published and then cited, and there are now over 50,000 DOIs that have been issued for software. However, some challenges remain, including: promoting the idea of software citation to developers and users; collaborating with publishers to ensure that systems collect and retain required metadata; ensuring that the rest of the scholarly infrastructure, particularly indexing sites, include software; working with communities so that software efforts “count”; and understanding how best to cite software that has not been published.
D. S. Katz, S. Druskat, R. Haines, C. Jay, A. Struck, "The State of Sustainable Research Software: Results from the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1)," arXiv:1807.07387 [cs.SE], 2018.
(This paper is open access)
Abstract: This article summarizes motivations, organization, and activities of the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1) held in Manchester, UK in September 2017. The WSSSPE series promotes sustainable research software by positively impacting principles and best practices, careers, learning, and credit. This article discusses the Code of Conduct, idea papers, position papers, experience papers, demos, and lightning talks presented during the workshop. The main part of the article discusses the speed-blogging groups that formed during the meeting, along with the outputs of those sessions.
HEP Software Foundation: D. Berzano, R. M. Bianchi, P. Elmer, J. Harvey, R. Jones, M. Jouvin, D. S. Katz, S. Malik, D. Menasce, M. Neubauer, A. Puig, G. A. Stewart, C. Tunnell, "HEP Software Foundation Community White Paper Working Group - Training, Staffing and Careers," arXiv: 1807.02875 [physics.ed-ph], 2018.
(This paper is open access)
Abstract: The rapid evolution of technology and the parallel increasing complexity of algorithmic analysis in HEP requires developers to acquire a much larger portfolio of programming skills. Young researchers graduating from universities worldwide currently do not receive adequate preparation in the very diverse fields of modern computing to respond to growing needs of the most advanced experimental challenges. There is a growing consensus in the HEP community on the need for training programmes to bring researchers up to date with new software technologies, in particular in the domains of concurrent programming and artificial intelligence. We review some of the initiatives under way for introducing new training programmes and highlight some of the issues that need to be taken into account for these to be successful.
HEP Software Foundation: L. Bauerdick, R. M. Bianchi, B. Bockelman, N. Castro, K. Cranmer, P. Elmer, R. Gardner, M. Girone, O. Gutsche, B. Hegner, J. M. Hernández, B. Jayatilaka, D. Lange, M. S. Neubauer, D. S. Katz, L. Kreczko, J. Letts, S. McKee, C. Paus, K. Pedro, J. Pivarski, M. Ritter, E. Rodrigues, T. Sakuma, E. Sexton-Kennedy, M. D. Sokoloff, C. Vuosalo, Frank Würthwein, G. Watts, "HEP Software Foundation Community White Paper Working Group – Data Analysis and Interpretation," arXiv: 1804.03983 [physics.comp-ph], 2018.
(This paper is open access)
Abstract: At the heart of experimental high energy physics (HEP) is the development of facilities and instrumentation that provide sensitivity to new phenomena. Our understanding of nature at its most fundamental level is advanced through the analysis and interpretation of data from sophisticated detectors in HEP experiments. The goal of data analysis systems is to realize the maximum possible scientific potential of the data within the constraints of computing and human resources in the least time. To achieve this goal, future analysis systems should empower physicists to access the data with a high level of interactivity, reproducibility and through-put capability. As part of the HEP Software Foundation’s Community White Paper process, a working group on Data Analysis and Interpretation was formed to assess the challenges and opportunities in HEP data analysis and develop a roadmap for activities in this area over the next decade. In this report, the key findings and recommendations of the Data Analysis and Interpretation Working Group are presented.
D. S. Katz, K. E. Niemeyer, S. Gesing, L. Hwang, W. Bangerth, S. Hettrick, R. Idaszak, J. Salac, N. Chue Hong, S. Núñez-Corrales, A. Allen, R. S. Geiger, J. Miller, E. Chen, A. Dubey, P. Lago, "Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4)," Journal of Open Research Software, v6.1, pp.10, 2018. DOI: 10.5334/jors.184
(This paper is open access)
Abstract: This article summarizes motivations, organization, and activities of the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4). The WSSSPE series promotes sustainable research software by positively impacting principles and best practices, careers, learning, and credit. This article discusses the code of conduct; the mission and vision statements that were drafted at the workshop and finalized shortly after it; the keynote and idea papers, position papers, experience papers, demos, and lightning talks presented during the workshop; and a panel discussion on best practices. The main part of the article discusses the set of working groups that formed during the meeting, along with contact information for readers who may want to join a group. Finally, it discusses a survey of the workshop attendees.
A. M. Smith, K. E. Niemeyer, D. S. Katz, L. A. Barba, G. Githinji, M. Gymrek, K. D. Huff, C. R. Madan, A. Cabunoc Mayes, K. M. Moerman, P. Prins, K. Ram, A. Rokem, T. K. Teal, R. Valls Guimera, J. T. Vanderplas, "Journal of Open Source Software (JOSS): design and first-year review," PeerJ Computer Science, v.4, pp. e147, 2018. DOI: 10.7717/peerj-cs.147
(This paper is open access)
Abstract: This article describes the motivation, design, and progress of the Journal of Open Source Software (JOSS). JOSS is a free and open-access journal that publishes articles describing research software. It has the dual goals of improving the quality of the software submitted and providing a mechanism for research software developers to receive credit. While designed to work within the current merit system of science, JOSS addresses the dearth of rewards for key contributions to science made in the form of software. JOSS publishes articles that encapsulate scholarship contained in the software itself, and its rigorous peer review targets the software components: functionality, documentation, tests, continuous integration, and the license. A JOSS article contains an abstract describing the purpose and functionality of the software, references, and a link to the software archive. The article is the entry point of a JOSS submission, which encompasses the full set of software artifacts. Submission and review proceed in the open, on GitHub. Editors, reviewers, and authors work collaboratively and openly. Unlike other journals, JOSS does not reject articles requiring major revision; while not yet accepted, articles remain visible and under review until the authors make adequate changes (or withdraw, if unable to meet requirements). Once an article is accepted, JOSS gives it a digital object identifier (DOI), deposits its metadata in Crossref, and the article can begin collecting citations on indexers like Google Scholar and other services. Authors retain copyright of their JOSS article, releasing it under a Creative Commons Attribution 4.0 International License. In its first year, starting in May 2016, JOSS published 111 articles, with more than 40 additional articles under review. JOSS is a sponsored project of the nonprofit organization NumFOCUS and is an affiliate of the Open Source Initiative (OSI).
M. L. Mondelli, T. Magalhães, G. Loss, M. Wilde, I. Foster, M. Mattoso, D. S. Katz, H. J. C. Barbosa, A. T. R. Vasconcelos, K. Ocaña, L. M. R. Gadelha Jr, "BioWorkbench: A High-Performance Framework for Managing and Analyzing Bioinformatics Experiments," arXiv: 1801.03915 [cs.DC], 2018.
(This paper is open access)
Abstract: Advances in sequencing techniques have led to exponential growth in biological data, demanding the development of large-scale bioinformatics experiments. Because these experiments are computation- and data-intensive, they require high-performance computing (HPC) techniques and can benefit from specialized technologies such as Scientific Workflow Management Systems (SWfMS) and databases. In this work, we present BioWorkbench, a framework for managing and analyzing bioinformatics experiments. This framework automatically collects provenance data, including both performance data from workflow execution and data from the scientific domain of the workflow application. Provenance data can be analyzed through a web application that abstracts a set of queries to the provenance database, simplifying access to provenance information. We evaluate BioWorkbench using three case studies: SwiftPhylo, a phylogenetic tree assembly workflow; SwiftGECKO, a comparative genomics workflow; and RASflow, a RASopathy analysis workflow. We analyze each workflow from both computational and scientific domain perspectives, by using queries to a provenance and annotation database. Some of these queries are available as a pre-built feature of the BioWorkbench web application. Through the provenance data, we show that the framework is scalable and achieves high-performance, reducing up to 98% of the case studies execution time. We also show how the application of machine learning techniques can enrich the analysis process.
HEP Software Foundation: A. A. Alves Jr, G. Amadio, N. Anh-Ky, L. Aphecetche, J. Apostolakis, M. Asai, L. Atzori, M. Babik, G. Bagliesi, M. Bandieramonte, M. Barisits, L. A. T. Bauerdick, S. Belforte, D. Benjamin, C. Bernius, W. Bhimji, R. M. Bianchi, I. Bird, C. Biscarat, J. Blomer, K. Bloom, T. Boccali, B. Bockelman, T. Bold, D. Bonacorsi, A. Boveia, C. Bozzi, M. Bracko, D. Britton, A. Buckley, P. Buncic, P. Calafiura, S. Campana, P. Canal, L. Canali, G. Carlino, N. F. Castro, M. Cattaneo, G. Cerminara, P. Chang, J. Chapman, G. Chen, T. Childers, P. Clarke, M. Clemencic, E. Cogneras, I. Collier, G. Corti, G. Cosmo, D. Costanzo, B. Couturier, K. Cranmer, J. Cranshaw, L. Cristella, D. Crooks, S. Crépé-Renaudin, S. Dallmeier-Tiessen, K. De, M. De Cian, A. Di Girolamo, G. Dimitrov, C. Doglioni, A. Dotti, D. Duellmann, L. Duflot, D. Dykstra, K. Dziedziniewicz-Wojcik, A. Dziurda, U. Egede, P. Elmer, J. Elmsheuser, V. D. Elvira, G. Eulisse, T. Ferber, A. Filipcic, I. Fisk, C. Fitzpatrick, J. Flix, A. Formica, A. Forti, F. Gaede, G. Ganis, R. Gardner, V. Garonne, A. Gellrich, K. Genser, S. George, F. Geurts, A. Gheata, M. Gheata, F. Giacomini, S. Giagu, M. Giffels, D. Gingrich, M. Girone, V. V. Gligorov, I. Glushkov, W. Gohn, J. B. Gonzalez Lopez, I. González Caballero, J. R. González Fernández, G. Govi, C. Grandi, H. Grasland, H. Gray, L. Grillo, W. Guan, O. Gutsche, V. Gyurjyan, A. Hanushevsky, F. Hariri, T. Hartmann, J. Harvey, T. Hauth, B. Hegner, B. Heinemann, L. Heinrich, J. M. Hernández, M. Hildreth, S. Höche, M. Hodgkinson, P. Hristov, X. Huang, V. N. Ivanchenko, T. Ivanov, B. Jashal, B. Jayatilaka, R. Jones, M. Jouvin, S. Y. Jun, M. Kagan, C. W. Kalderon, E. Karavakis, D. S. Katz, D. Kcira, B. P. Kersevan, M. Kirby, A. Klimentov, M. Klute, I. Komarov, P. Koppenburg, J. Kowalkowski, L. Kreczko, T. Kuhr, R. Kutschke, V. Kuznetsov, W. Lampl, E. Lancon, D. Lange, M. Lassnig, P. Laycock, C. Leggett, J. Letts, B. Lewendel, T. Li, G. Lima, J. Linacre, T. Linden, G. Lo Presti, S. Lopienski, P. Love, Z. L. Marshall, E. Martelli, S. Martin-Haugh, P. Mato, K. Mazumdar, T. McCauley, J. McFayden, S. McKee, A. McNab, H. Meinhard, D. Menasce, P. Mendez Lorenzo, A. Serhan Mete, M. Michelotto, J. Mitrevski, L. Moneta, B. Morgan, R. Mount, E. Moyse, S. Murray, M. S. Neubauer, S. Novaes, M. Novak, A. Oyanguren, N. Ozturk, A. Pacheco Pages, M. Paganini, J. Pansanel, V. R. Pascuzzi, A. Pearce, B. Pearson, K. Pedro, G. Perdue, A. Perez-Calero Yzquierdo, L. Perrozzi, T. Petersen, M. Petric, J. Piedra, L. Piilonen, D. Piparo, W. Pokorski, F. Polci, K. Potamianos, F. Psihas, G. Raven, J. Reuter, A. Ribon, M. Ritter, J. Robinson, E. Rodrigues, S. Roiser, D. Rousseau, G. Roy, A. Sailer, T. Sakuma, R. Santana, A. Sartirana, H. Schellman, J. Schovancová, S. Schramm, M. Schulz, A. Sciabà, S. Seidel, S. Sekmen, C. Serfon, H. Severini, E. Sexton-Kennedy, M. Seymour, I. Shapoval, J. Shiers, J.-G. Shiu, H. Short, G. P. Siroli, S. Skipsey, T. Smith, S. Snyder, M. D. Sokoloff, H. Stadie, G. Stark, G. Stewart, A. Sánchez-Hernández, J. Templon, G. Tenaglia, V. Tsulaia, C. Tunnell, E. Vaandering, A. Valassi, S. Vallecorsa, L. Valsan, P. Van Gemmeren, R. Vernet, B. Viren, J.-R. Vlimant, C. Voss, C. Vuosalo, C. Vázquez Sierra, R. Wartel, T. Wenaus, S. Wenzel, F. Winklmeier, C. Wissing, F. Wuerthwein, B. Wynne, Z. Xiaomei, W. Yang and E. Yazgan, "A Roadmap for HEP Software and Computing R&D for the 2020s," arXiv: 1712.06982 [physics.comp-ph], 2017.
(This paper is open access)
Abstract: Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.
HEP Software Foundation: B. Couturier, G. Eulisse, H. Grasland, B. Hegner, M. Jouvin, M. Kane, D. S. Katz, T. Kuhr, D. Lange, P. Mendez Lorenzo, M. Ritter, G. A. Stewart, A. Valassi, "HEP Software Foundation Community White Paper Working Group - Software Development, Deployment and Validation," arXiv: 1712.07959 [physics.comp-ph], 2017.
(This paper is open access)
Abstract: The High Energy Phyiscs community has developed and needs to maintain many tens of millions of lines of code and to integrate effectively the work of thousands of developers across large collaborations. Software needs to be built, validated, and deployed across hundreds of sites. Software also has a lifetime of many years, frequently beyond that of the original developer, it must be developed with sustainability in mind. Adequate recognition of software development as a critical task in the HEP community needs to be fostered and an appropriate publication and citation strategy needs to be developed. As part of the HEP Softare Foundation's Community White Paper process a working group on Software Development, Deployment and Validation was formed to examine all of these issues, identify best practice and to formulare recommendations for the next decade. Its report is presented here.
E. A. Huerta, R. Haas, E. Fajardo, D. S. Katz, S. Anderson, P. Couvares, J. Willis, T. Bouvet, J. Enos, W. T. C. Kramer, H. W. Leong, D. Wheeler, "BOSS-LDG: A Novel Computational Framework that Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery," Proceedings of the 13th IEEE International Conference on eScience (eScience 2017), 2017. DOI: 10.1109/eScience.2017.47
(This paper is not open access)
Abstract: We present a novel computational framework that connects Blue Waters, the NSF-supported, leadership-class supercomputer operated by NCSA, to the Laser Interferometer Gravitational-Wave Observatory (LIGO) Data Grid via Open Science Grid technology. To enable this computational infrastructure, we configured, for the first time, a LIGO Data Grid Tier-1 Center that can submit heterogeneous LIGO workflows using Open Science Grid facilities. In order to enable a seamless connection between the LIGO Data Grid and Blue Waters via Open Science Grid, we utilize Shifter to containerize LIGO’s workflow software. This work represents the first time Open Science Grid, Shifter, and Blue Waters are unified to tackle a scientific problem and, in particular, it is the first time a framework of this nature is used in the context of large scale gravitational wave data analysis. This new framework has been used in the last several weeks of LIGO’s second discovery campaign to run the most computationally demanding gravitational wave search workflows on Blue Waters, and accelerate discovery in the emergent field of gravitational wave astrophysics. We discuss the implications of this novel framework for a wider ecosystem of Higher Performance Computing users.
M. Turilli, Y. N. Babuji, A. Merzky, M. T. Ha, M. Wilde, D. S. Katz, S. Jha, "Evaluating Distributed Execution of Workloads," Proceedings of the 13th IEEE International Conference on eScience (eScience 2017), 2017. DOI: 10.1109/eScience.2017.41 (preprint: arXiv 1605.09513)
(This paper is green open access)
Abstract: Resource selection and task placement for distributed execution poses conceptual and implementation difficulties. Although resource selection and task placement are at the core of many tools and workflow systems, the methods are ad hoc rather than being based on models. Consequently, partial and non-interoperable implementations proliferate. We address both the conceptual and implementation difficulties by experimentally characterizing diverse modalities of resource selection and task placement. We compare the architectures and capabilities of two systems: the AIMES middleware and Swift workflow scripting language and runtime. We integrate these systems to enable the distributed execution of Swift workflows on Pilot-Jobs managed by the AIMES middleware. Our experiments characterize and compare alternative execution strategies by measuring the time to completion of heterogeneous uncoupled workloads executed at diverse scale and on multiple resources. We measure the adverse effects of pilot fragmentation and early binding of tasks to resources and the benefits of backfill scheduling across pilots on multiple resources. We then use this insight to execute a multi-stage workflow across five production-grade resources. We discuss the importance and implications for other tools and workflow systems.
U. Nangia and D. S. Katz, "Understanding Software in Research: Initial Results from Examining Nature and a Call for Collaboration", Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.2), 2017. DOI: 10.1109/eScience.2017.78
(This paper is not open access)
Abstract: This lightning talk paper discusses an initial data set that has been gathered to understand the use of software in research, and is intended to spark wider interest in gathering more data. The initial data analyzes three months of articles in the journal Nature for software mentions. The wider activity that we seek is a community effort to analyze a wider set of articles, including both a longer timespan of Nature articles as well as articles in other journals. Such a collection of data could be used to understand how the role of software has changed over time and how it varies across fields.
U. Nangia and D. S. Katz, "Track 1 Paper: Surveying the US National Postdoctoral Association Regarding Software Use and Training in Research", Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1), 2017. DOI: 10.6084/m9.figshare.5328442.v3
(This paper is open access)
Abstract: This paper reports on the results of a 2017 survey conducted by email and web of members of the US National Postdoctoral Association regarding their use of software in research and their training regarding software development. The responses show that that 95% of respondents use research software. Of all the respondents, 63% state they could not do their research without research software, 31% could do it but with more effort, and 6% would not find a significant difference in their research without research software. In addition, 54% of respondents have not received any training in software development, though all respondents who develop software for researchers have received either self-taught or formal software development training.
J. P. Tennant, J. M. Dugan, D. Graziotin, D. C. Jacques, F. Waldner, D. Mietchen, Y. Elkhatib, L. B. Collister, C. K. Pikas, T. Crick, P. Masuzzo, A. Caravaggi, D. R. Berg, K. E. Niemeyer, T. Ross-Hellauer, S. Mannheimer, L. Rigling, D. S. Katz, B. Greshake, J. Pacheco-Mendoza, N. Fatima, M. Poblet, M. Isaakidis, D. E. Irawan, S. Renaut, C. R. Madan, L. Matthias, J. N. Kjær, D. P. O’Donnell, C. Neylon, S. Kearns, M. Selvaraju, and J. Colomb, "A multi-disciplinary perspective on emergent and future innovations in peer review" [version 3; referees: 2 approved with reservations], F1000Research, v.6:1151, 2017. doi: 10.12688/f1000research.12037.3 (This paper is open access)
Abstract:
Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.
R. C. Jiménez, M. Kuzak, Mo. Alhamdoosh, M. Barker, B. Batut, M. Borg, S. Capella-Gutierrez, N. Chue Hong, M. Cook, M. Corpas, M. Flannery, L. Garcia, J. L. Gelpí, S. Gladman, C. Goble, M. González Ferreiro, A. Gonzalez-Beltran, P. C. Griffin, B. Grüning, J. Hagberg, P. Holub, R. Hooft, J. Ison, D. S. Katz, B. Leskošek, F. López Gómez, L. J. Oliveira, D. Mellor, R. Mosbergen, N. Mulder, Y. Perez-Riverol, R. Pergl, H. Pichler, B. Pope, F. Sanz, M. V. Schneider, V. Stodden, R. Suchecki, R. Svobodová Vařeková, H.-A. Talvik, I. Todorov, A. Treloar, S. Tyagi, M. van Gompel, D. Vaughan, A. Via, X. Wang, N. S. Watson-Haigh, S. Crouch, "Four simple recommendations to encourage best practices in research software" [version 1; referees: 3 approved], F1000Research v.6:876, 2017. doi: 10.12688/f1000research.11407.1
(This paper is open access)
Abstract: Scientific research relies on computer software, yet software is not always developed following practices that ensure its quality and sustainability. This manuscript does not aim to propose new software development best practices, but rather to provide simple recommendations that encourage the adoption of existing best practices. Software development best practices promote better quality software, and better quality software improves the reproducibility and reusability of research. These recommendations are designed around Open Source values, and provide practical suggestions that contribute to making research software and its source code more discoverable, reusable and transparent. This manuscript is aimed at developers, but also at organisations, projects, journals and funders that can increase the quality and sustainability of research software by encouraging the adoption of these recommendations.
A. Allen, C. Aragon, C. Becker, J. Carver, A. Chiş, B. Combemale, M. Croucher, K. Crowston, D. Garijo, A. Gehani, C. Goble, R. Haines, R. Hirschfeld, J. Howison, K. Huff, C. Jay, D. S. Katz, C. Kirchner, K. Kuksenok, R. Lämmel, O. Nierstrasz, M. Turk, R. van Nieuwpoort, M. Vaughn, J. J. Vinju, "Engineering Academic Software (Dagstuhl Perspectives Workshop 16252)," Dagstuhl Manifestos, v.6(1), 2017. doi: 10.4230/DagMan.6.1.1
(This paper is open access)
Abstract: Software is often a critical component of scientific research. It can be a component of the academic research methods used to produce research results, or it may itself be an academic research result. Software, however, has rarely been considered to be a citable artifact in its own right. With the advent of open-source software, artifact evaluation committees of conferences, and journals that include source code and running systems as part of the published artifacts, we foresee that software will increasingly be recognized as part of the academic process. The quality and sustainability of this software must be accounted for, both a priori and a posteriori.
The Dagstuhl Perspectives Workshop on "Engineering Academic Software" has examined the strengths, weaknesses, risks, and opportunities of academic software engineering. A key outcome of the workshop is this Dagstuhl Manifesto, serving as a roadmap towards future professional software engineering for software-based research instruments and other software produced and used in an academic context. The manifesto is expressed in terms of a series of actionable "pledges" that users and developers of academic research software can take as concrete steps towards improving the environment in which that software is produced.
A. Marshall-Colon, S. P. Long, D. K. Allen, G. Allen, D. A. Beard, B. Benes, S. von Caemmerer, A. J. Christensen, D. J. Cox, J. C. Hart, P. M. Hirst, K. Kannan, D. S. Katz, J. P. Lynch, A. J. Millar, B. Panneerselvam, N. D. Price, P. Prusinkiewicz, D. Raila, R. G. Shekar, S. Shrivastava, D. Shukla, V. Srinivasan, M. Stitt, M. J. Turk, E. O. Voit, Y. Wang, X. Yin, X.-G. Zhu, "Crops In Silico: Generating Virtual Crops Using an Integrative and Multi-scale Modeling Platform," Frontiers in Plant Science, v.8, page 786, 2017. doi: 10.3389/fpls.2017.00786
(This paper is open access)
Abstract: Multi-scale models can facilitate whole plant simulations by linking gene networks, protein synthesis, metabolic pathways, physiology, and growth. Whole plant models can be further integrated with ecosystem, weather, and climate models to predict how various interactions respond to environmental perturbations. These models have the potential to fill in missing mechanistic details and generate new hypotheses to prioritize directed engineering efforts. Outcomes will potentially accelerate improvement of crop yield, sustainability, and increase future food security. It is time for a paradigm shift in plant modeling, from largely isolated efforts to a connected community that takes advantage of advances in high performance computing and mechanistic understanding of plant processes. Tools for guiding future crop breeding and engineering, understanding the implications of discoveries at the molecular level for whole plant behavior, and improved prediction of plant and ecosystem responses to the environment are urgently needed. The purpose of this perspective is to introduce Crops in silico (cropsinsilico.org), an integrative and multi-scale modeling platform, as one solution that combines isolated modeling efforts toward the generation of virtual crops, which is open and accessible to the entire plant biology community. The major challenges involved both in the development and deployment of a shared, multi-scale modeling platform, which are summarized in this prospectus, were recently identified during the first Crops in silico Symposium and Workshop.
D. S. Katz, K. E. Niemeyer, S. Gesing, L. Hwang, W. Bangerth, S. Hettrick, R. Idaszak, J. Salac, N. Chue Hong, S. Núñez Corrales, A. Allen, R. S. Geiger, J. Miller, E. Chen, A. Dubey, P. Lago, "Report on the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4)", arXiv: 1705.02607 [cs.SE], 2017.
(This paper is open access)
Abstract: This report records and discusses the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4). The report includes a description of the keynote presentation of the workshop, the mission and vision statements that were drafted at the workshop and finalized shortly after it, a set of idea papers, position papers, experience papers, demos, and lightning talks, and a panel discussion. The main part of the report covers the set of working groups that formed during the meeting, and for each, discusses the participants, the objective and goal, and how the objective can be reached, along with contact information for readers who may want to join the group. Finally, we present results from a survey of the workshop attendees.
S. Jha, D. S. Katz, A. Luckow, N. Chue Hong, O. Rana, Y. Simmhan, "Introducing Distributed Dynamic Data-intensive (D3) Science: Understanding Applications and Infrastructure," Journal of Concurrency and Computation: Practice and Experience (CCPE), page e4032, 2017. DOI: 10.1002/cpe.4032
(This paper is not open access)
Abstract: A common feature across many science and engineering applications is the amount and diversity of data and computation that must be integrated to yield insights. Datasets are growing larger and becoming distributed; their location, availability, and properties are often time-dependent. Collectively, these characteristics give rise to dynamic distributed data-intensive applications. While "static" data applications have received significant attention, the characteristics, requirements, and software systems for the analysis of large volumes of dynamic, distributed data, and data-intensive applications have received relatively less attention. This paper surveys several representative dynamic distributed data-intensive application scenarios, provides a common conceptual framework to understand them, and examines the infrastructure used in support of applications.
D. S. Katz, K. E. Niemeyer, A. M. Smith, "Strategies for biomedical software management, sharing, and citation," PeerJ Preprints 4:e2640v1, 2016. DOI: 10.7287/peerj.preprints.2640v1
(This paper is open access)
Abstract: This document is an open response to the NIH Request for Information (RFI): Strategies for NIH Data Management, Sharing, and Citation, Notice Number: NOT-OD-17-015, written by the leaders of the FORCE11 Software Citation Working Group from its inception in mid-2015 through today. This group produced a set of Software Citation Principles and related discussion, which are the basis for this document. Here, we describe research software, summarize the software citation principles, discuss open issues related to software citation, and make recommendations to the NIH.
D. S. Katz, K. E. Niemeyer, A. M. Smith, W. L. Anderson, C. Boettiger, K. Hinsen, R. Hooft, A. Lee, F. Löffler, T. Pollard, F. Rios, "Software vs. data in the context of citation," PeerJ Preprints 4:e2630v1, 2016. DOI: 10.7287/peerj.preprints.2630v1
(This paper is open access)
Abstract: Software is data, but it is not just data. While "data" in computing and information science can refer to anything that can be processed by a computer, software is a special kind of data that can be a creative, executable tool that operates on data. However, software and data are similar in that they both traditionally have not been cited in publications. This paper discusses the differences between software and data in the context of citation, by providing examples and referring to evidence in the form of citations.
K. E. Niemeyer, A. M. Smith, D. S. Katz, "The Challenge and Promise of Software Citation for Credit, Identification, Discovery, and Reuse," ACM Journal of Data and Information Quality (JDIQ), v.4(7):16, 2016. doi: 10.1145/2968452 (preprint arXiv: 1601.04734 [cs.CY])
(This paper is green open access)
M. Gamell, D. S. Katz, K. Teranishi, M. A. Heroux, R. F. Van der Wijngaart, T. G. Mattson, M. Parashar, "Evaluating Online Global Recovery with Fenix Using Application-Aware In-Memory Checkpointing Techniques," International Workshop on Sustainable HPC Cloud (SHPCloud 2016), in conjunction with ICPP, 2016. doi: 10.1109/ICPPW.2016.56
(This paper is not open access)
Abstract: Exascale systems promise the potential for computation at unprecedented scales and resolutions, but achieving exascale by the end of this decade presents significant challenges. A key challenge is due to the very large number of cores and components and the resulting mean time between failures (MTBF) in the order of hours or minutes. Since the typical run times of target scientific applications are longer than this MTBF, fault tolerance techniques will be essential. An important class of failures that must be addressed is process or node failures. While checkpoint/restart (C/R) is currently the most widely accepted technique for addressing processor failures, coordinated, stable-storage-based global C/R might be unfeasible at exascale when the time to checkpoint exceeds the expected MTBF.
This paper explores transparent recovery via implicitly coordinated, diskless, application-driven checkpointing as a way to tolerate process failures in MPI applications at exascale. The discussed approach leverages User Level Failure Mitigation (ULFM), which is being proposed as an MPI extension to allow applications to create policies for tolerating process failures. Specifically, this paper demonstrates how different implementations of application-driven in-memory checkpoint storage and recovery compare in terms of performance and scalability. We also experimentally evaluate the effectiveness and scalability of the Fenix online global recovery framework on a production system&emdash;the Titan Cray XK7 at ORNL&emdash;and demonstrate the ability of Fenix to tolerate dynamically injected failures using the execution of four benchmarks and mini-applications with different behaviors.
A. M. Smith, D. S. Katz, K. E. Niemeyer, FORCE11 Software Citation Working Group, "Software Citation Principles," PeerJ Computer Science, v.2, page e86, 2016. doi: 10.7717/peerj-cs.86
(This paper is open access)
Abstract: Software is a critical part of modern research and yet there is little support across the scholarly ecosystem for its acknowledgement and citation. Inspired by the activities of the FORCE11 working group focused on data citation, this document summarizes the recommendations of the FORCE11 Software Citation Working Group and its activities between June 2015 and April 2016. Based on a review of existing community practices, the goal of the working group was to produce a consolidated set of citation principles that may encourage broad adoption of a consistent policy for software citation across disciplines and venues. Our work is presented here as a set of software citation principles, a discussion of the motivations for developing the principles, reviews of existing community practice, and a discussion of the requirements these principles would place upon different stakeholders. Working examples and possible technical solutions for how these principles can be implemented will be discussed in a separate paper.
D. S. Katz, S.-C. T. Choi, K. E. Niemeyer, J. Hetherington, F. Löffler, D. Gunter, R. Idaszak, S. R. Brandt, M. A. Miller, S. Gesing, N. D. Jones, N. Weber, S. Marru, G. Allen, B. Penzenstadler, C. C. Venters, E. Davis, L. Hwang, I. Todorov, A. Patra, M. de Val-Borro, "Report on the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3)," Journal of Open Research Software, v.4(1), p. e37, 2016. doi: 10.5334/jors.118 (Appendices) (preprint: arXiv: 1602.02296 [cs.SE])
(This paper is open access)
Abstract: This report records and discusses the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3). The report includes a description of the keynote presentation of the workshop, which served as an overview of sustainable scientific software. It also summarizes a set of lightning talks in which speakers highlighted to-the-point lessons and challenges pertaining to sustaining scientific software. The final and main contribution of the report is a summary of the discussions, future steps, and future organization for a set of self-organized working groups on topics including developing pathways to funding scientific software; constructing useful common metrics for crediting software stakeholders; identifying principles for sustainable software engineering design; reaching out to research software organizations around the world; and building communities for software sustainability. For each group, we include a point of contact and a landing page that can be used by those who want to join that group’s future activities. The main challenge left by the workshop is to see if the groups will execute these activities that they have scheduled, and how the WSSSPE community can encourage this to happen.
K. Chard, S. Caton, K. Kugler, O. Rana, and D. S. Katz, "A social content delivery network for e-Science," Concurrency and Computation: Practice and Experience, v.29(4), p. e3854, 2017. DOI: 10.1002/cpe.3854
(This paper is not open access)
Abstract: We are in the midst of a scientific data explosion in which the rate of data growth is rapidly increasing. While large-scale research projects have developed sophisticated data distribution networks to share their data with researchers globally, there is no such support for the many millions of research projects generating data of interest to much smaller audiences (as exemplified by the long tail scientist). In data-oriented research, every aspect of the research process is influenced by data access. However, sharing and accessing data efficiently as well as lowering access barriers are difficult. In the absence of dedicated large-scale storage, many have noted that there is an enormous storage capacity available via connected peers, none more so than the storage resources of many research groups. With widespread usage of the content delivery network model for disseminating web content, we believe a similar model can be applied to distributing, sharing, and accessing long tail research data in an e-Science context. We describe the vision and architecture of a social content delivery network - a model that leverages the social networks of researchers to automatically share and replicate data on peers’ resources based upon shared interests and trust. Using this model, we describe a simulator and investigate how aspects such as user activity, geographic distribution, trust, and replica selection algorithms affect data access and storage performance. From these results, we show that socially informed replication strategies are comparable with more general strategies in terms of availability and outperform them in terms of spatial efficiency.
Z. Zhang, D. S. Katz, A. Merzky, M. Turilli, S. Jha, Y. Nand, "Application Skeleton: Generating Synthetic Applications for Infrastructure Research," Journal of Open Source Software, 2016. DOI: 10.21105/joss.00017
(This paper is open access)
Abstract: Application Skeleton is a simple and powerful tool to build simplified synthetic science and engineering applications (for example, modeling and simulation, data analysis) with runtime and I/O close to that of the real applications. It is intended for applied computer scientists who need to use science and engineering applications to verify the effectiveness of new systems designed to efficiently run such applications, so that they can bypass obstacles that they often encounter when accessing and building real science and engineering applications. Using the applications generated by Application Skeleton guarantees that the CS systems' effectiveness on synthetic applications will apply to the real applications.
Application Skeleton can generate bag-of-task, (iterative) map-reduce, and (iterative) multistage workflow applications. These applications are represented as a set of tasks, a set of input files, and a set of dependencies. These applications can be generally considered many-task applications, and once created, can be run on single-core, single-node, multi-core, or multi-node (distributed or parallel) computers, depending on what workflow system is used to run them. The generated applications are compatible with workflow system such as Swift and Pegasus, as well as the ubiquitous UNIX shell. The application can also be created as a generic JSON object that can be used by other systems such as the AIMES middleware.
Y. Perez-Riverol, L. Gatto, R. Wang, T. Sachsenberg, J. Uszkoreit, F. da Veiga Leprevost, C. Fufezan, T. Ternent, S. J. Eglen, D. S. Katz, T. J. Pollard, A. Konovalov, R. M. Flight, K. Blin, and J. A. Vizcaino, "Ten Simple Rules for Taking Advantage of git and GitHub,” PLoX Computational Biology, v.12(7): e1004947, 2016. DOI: 10.1371/journal.pcbi.1004947.
(This paper is open access)
Abstract: A "Ten Simple Rules" guide to git and GitHub. We describe and provide examples on how to use these software to track projects, as users, teams and organizations. We document collaborative development using branching and forking, interaction between collaborators using issues and continuous integration and automation using, for example, Travis CI and codevoc. We also describe dissemination and social aspects of GitHub such as GitHub pages, following and watching repositories, and give advice on how to make code citable.
S. Jha, Daniel S. Katz, M. Turilli, and J. Weissman, From Abstractions to MODELS: MOdels for Distributed and Extremely Large-scale Science, Zenodo, 2016. DOI: 10.5281/zenodo.44236
(This paper is open access)
D. S. Katz, A. Merzky, Z. Zhang, S. Jha, "Application skeletons: Construction and use in eScience," Future Generation Computer Systems, v.59. pp 114-124, 2016. DOI: 10.1016/j.future.2015.10.001
(This paper is not open access)
Abstract: Computer scientists who work on tools and systems to support eScience (a variety of parallel and distributed) applications usually use actual applications to prove that their systems will benefit science and engineering (e.g., improve application performance). Accessing and building the applications and necessary data sets can be difficult because of policy or technical issues, and it can be difficult to modify the characteristics of the applications to understand corner cases in the system design. In this paper, we present the Application Skeleton, a simple yet powerful tool to build synthetic applications that represent real applications, with runtime and I/O close to those of the real applications. This allows computer scientists to focus on the system they are building; they can work with the simpler skeleton applications and be sure that their work will also be applicable to the real applications. In addition, skeleton applications support simple reproducible system experiments since they are represented by a compact set of parameters.
Our Application Skeleton tool (available as open source at https:// github.com/applicationskeleton/Skeleton) currently can create easy-to-access, easy-to-build, and easy-to-run bag-of-task, (iterative) map-reduce, and (iterative) multistage workflow applications. The tasks can be serial, parallel, or a mix of both. The parameters to represent the tasks can either be discovered through a manual profiling of the applications or through an automated method. We select three representative applications (Montage, BLAST, CyberShake Postprocessing), then describe and generate skeleton applications for each. We show that the skeleton applications have identical (or close) performance to that of the real applications. We then show examples of using skeleton applications to verify system optimizations such as data caching, I/O tuning, and task scheduling, as well as the system resilience mechanism, in some cases modifying the skeleton applications to emphasize some characteristic, and thus show that using skeleton applications simplifies the process of designing, implementing, and testing these optimizations.
J. Wozniak, T. Armstrong, K. Maheshwari, D. S, Katz, M. Wilde, I. Foster, "Interlanguage Parallel Scripting for Distributed Memory Scientific Computing," Proceedings of Workflows in Support of Large-Scale Science (WORKS 2015), 2015. DOI: 10.1145/2822332.2822338
(This paper is not open access)
Abstract: Scripting languages such as Python and R have been widely adopted as tools for the development of scientific software because of the expressiveness of the languages and their available libraries. However, deploying scripted applications on large-scale parallel computer systems such as the IBM Blue Gene/Q or Cray XE6 is a challenge because of issues including operating system limitations, interoperability challenges, and parallel filesystem overheads due to the small file system accesses common in scripted approaches. We present a new approach to these problems in which the Swift scripting system is used to integrate high-level scripts written in Python, R, and Tcl with native code developed in C, C++, and Fortran, by linking Swift to the library interfaces to the script interpreters. We present a technique to efficiently launch scripted applications on supercomputers, and we demonstrate high performance, such as invoking 14M Python interpreters per second on Blue Waters.
D. S. Katz and A. M. Smith, "Transitive Credit and JSON-LD," Journal of Open Research Software, v.3: e7, 2015. DOI: 10.5334/jors.by
(This paper is open access)
Abstract: Science and engineering research increasingly relies on activities that facilitate research but are not currently rewarded or recognized, such as: data sharing; developing common data resources, software and methodologies; and annotating data and publications. To promote and advance these activities, we must develop mechanisms for assigning credit, facilitate the appropriate attribution of research outcomes, devise incentives for activities that facilitate research, and allocate funds to maximize return on investment. In this article, we focus on addressing the issue of assigning credit for both direct and indirect contributions, specifically by using JSON-LD to implement a prototype transitive credit system.
K. Maheshwari, J. M. Wozniak, T. G. Armstrong, D. S. Katz, T. A. Binkowski, X. Zhong, O. Heinonen, D. Karpeyev, M. Wilde, "Porting Ordinary Applications to Blue Gene/Q Supercomputers," Proceedings of 11th IEEE International Conference on eScience (eScience 2015), 2015. DOI: 10.1109/eScience.2015.8
(This paper is not open access)
Abstract: Efficiently porting ordinary applications to Blue Gene/Q supercomputers is a significant challenge. Codes are often originally developed without considering advanced architectures and related tool chains. Science needs frequently lead users to want to run large numbers of relatively small jobs (often called many-task computing, an ensemble, or a workflow), which can conflict with supercomputer configurations. In this paper, we discuss techniques developed to execute ordinary applications over leadership class supercomputers. We use the high-performance Swift parallel scripting framework and build two workflow execution techniques – sub-jobs and main-wrap. The sub-jobs technique, built on top of the IBM Blue Gene/Q resource manager Cobalt’s sub-block jobs, lets users submit multiple, independent, repeated smaller jobs within a single larger resource block. The main-wrap technique is a scheme that enables C/C++ programs to be defined as functions that are wrapped by a high-performance Swift wrapper and that are invoked as a Swift script. We discuss the needs, benefits, technicalities, and current limitations of these techniques. We further discuss the real-world science enabled by these techniques and the results obtained.
J. M. Wozniak, T. G. Armstrong, K. C. Maheshwari, D. S. Katz, M. Wilde, I. Foster, "Toward Interlanguage Parallel Scripting for Distributed Memory Scientific Computing," Proceedings of 2015 IEEE International Conference on Cluster Computing, 2015. DOI: 10.1109/CLUSTER.2015.74 (preprint: arXiv 2107.02841)
(This paper is not open access)
Abstract: Scripting languages such as Python and R have been widely adopted as tools for the productive development of scientific software because of the power and expressiveness of the languages and available libraries. However, deploying scripted applications on large-scale parallel computer systems such as the IBM Blue Gene/Q or Cray XE6 is a challenge because of issues including operating system limitations, interoperability challenges, parallel filesystem overheads due to the small file system accesses common in scripted approaches, and other issues. We present here a new approach to these problems in which the Swift scripting system is used to integrate high-level scripts written in Python, R, and Tcl, with native code developed in C, C++, and Fortran, by linking Swift to the library interfaces to the script interpreters. In this approach, Swift handles data management, movement, and marshaling among distributed-memory processes without direct user manipulation of low-level communication libraries such as MPI. We present a technique to efficiently launch scripted applications on large-scale supercomputers using a hierarchical programming model.
D. S. Katz, S.-C. T. Choi, N. Wilkins-Diehr, N. Chue Hong, C. C. Venters, J. Howison, F. Seinstra, M. Jones, K. Cranston, T. L. Clune, M. de Val-Borro, R. Littauer, " Report on the Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2)," Journal of Open Research Software, v.4: e7, 2016, arXiv:1507.01715 [cs.SE], 2015. DOI: 10.5334/jors.85
(This paper is open access)
Abstract: This technical report records and discusses the Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2). The report includes a description of the alternative, experimental submission and review process, two workshop keynote presentations, a series of lightning talks, a discussion on sustainability, and five discussions from the topic areas of exploring sustainability; software development experiences; credit & incentives; reproducibility & reuse & sharing; and code testing & code review. For each topic, the report includes a list of tangible actions that were proposed and that would lead to potential change. The workshop recognized that reliance on scientific software is pervasive in all areas of world-leading research today. The workshop participants then proceeded to explore different perspectives on the concept of sustainability. Key enablers and barriers of sustainable scientific software were identified from their experiences. In addition, recommendations with new requirements such as software credit files and software prize frameworks were outlined for improving practices in sustainable software engineering. There was also broad consensus that formal training in software development or engineering was rare among the practitioners. Significant strides need to be made in building a sense of community via training in software and technical practices, on increasing their size and scope, and on better integrating them directly into graduate education programs. Finally, journals can define and publish policies to improve reproducibility, whereas reviewers can insist that authors provide sufficient information and access to data and software to allow them reproduce the results in the paper. Hence a list of criteria is compiled for journals to provide to reviewers so as to make it easier to review software submitted for publication as a "Software Paper."
L. B. Costa, H. Yang, E. Vairavanathan, A. Barros, K. Maheshwari, G. Fedak, D. S. Katz, M. Wilde, M. Ripeanu, S. Al-Kiswany, "The Case for Workflow-Aware Storage: An Opportunity Study," Journal of Grid Computing, v.13(1), pp 95-113, 2015. DOI: 10.1007/s10723-014-9307-6
(This paper is not open access)
Abstract: This article evaluates the potential gains a workflow-aware storage system can bring. Two observations make us believe such storage system is crucial to efficiently support workflow-based applications: First, workflows generate irregular and application-dependent data access patterns. These patterns render existing generic storage systems unable to harness all optimization opportunities as this often requires enabling conflicting optimizations or even conflicting design decisions at the storage system level. Second, most workflow runtime engines make suboptimal scheduling decisions as they lack the detailed data location information that is generally hidden by the storage system. This paper presents a limit study that evaluates the potential gains from building a workflow-aware storage system that supports per-file access optimizations and exposes data location. Our evaluation using synthetic benchmarks and real applications shows that a workflow-aware storage system can bring significant performance gains: up to 3x performance gains compared to a vanilla distributed storage system deployed on the same resources yet unaware of the possible file-level optimizations.
M. Gamell, D. S. Katz, H. Kolla, J. Chen, S. Klasky, M. Parashar, "Exploring Automatic, Online Failure Recovery for Scientific Applications at Extreme Scales," Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC14), 2014. DOI: 10.1109/SC.2014.78
(This paper is not open access)
Abstract: Application resilience is a key challenge that must be addressed in order to realize the exascale vision. Process/node failures, an important class of failures, are typically handled today by terminating the job and restarting it from the last stored checkpoint. This approach is not expected to scale to exascale. In this paper we present Fenix, a framework for enabling recovery from process/node/blade/cabinet failures for MPI-based parallel applications in an online (i.e., without disrupting the job) and transparent manner. Fenix provides mechanisms for transparently capturing failures, re-spawning new processes, fixing failed communicators, restoring application state, and returning execution control back to the application. To enable automatic data recovery, Fenix relies on application-driven, diskless, implicitly coordinated checkpointing. Using the S3D combustion simulation running on the Titan Cray-XK7 production system at ORNL, we experimentally demonstrate Fenix’s ability to tolerate high failure rates (e.g., more than one per minute) with low overhead and while sustaining performance.
K. Kugler, S. Caton, K. Chard, and D. S. Katz, "On Replica Placement in a Social CDN for e-Science," Proceedings of 10th IEEE International Conference on eScience, pp. 13-20, 2014. DOI: 10.1109/eScience.2014.26
(This paper is not open access)
Abstract: Research data is experiencing a seemingly endless increase in both volume and production rate. At the same time, efficiently transferring, storing, and analyzing large scale research data have become major research foci. In this paper, we expand on our approach to sharing data for e-Science: a Social Content Delivery Network (S-CDN). A S-CDN leverages the social networks of researchers to automatically share data and place replicas on peers' resources based upon the premises of trust and interest in shared data. We denote a consumer of shared data as a data follower, similar to the notion of Twitter followers, except we add the element of bilateral authorization to capture a notion of trust. We describe a prototypical implementation for a S-CDN that captures an efficient asynchronous transfer mechanism for data management and replication. In addition, we study via simulation the interplay of user behavior with different replication strategies that capture social as well as more general premises for data sharing. Our results illustrate the opportunities and pitfalls of various replication and data access management strategies. Specifically, we show that socially-informed replication strategies are competitive with more general strategies in terms of availability, and outperform them in terms of spatial efficiency.
Z. Zhang and D. S. Katz, "Using Application Skeletons to Improve eScience Infrastructure," Proceedings of 10th IEEE International Conference on eScience, pp. 111-118, 2014. DOI: 10.1109/eScience.2014.9
(This paper is not open access)
Abstract: Computer scientists who work on tools and systems to support eScience (a variety of parallel and distributed) applications usually use actual applications to prove that their systems will benefit science and engineering (e.g., improve application performance). Accessing and building the applications and necessary data sets can be difficult because of policy or technical issues, and it can be difficult to modify the characteristics of the applications to understand corner cases in the system design. In this paper, we present the Application Skeleton, a simple yet powerful tool to build synthetic applications that represent real applications, with runtime and I/O close to those of the real applications. This allows computer scientists to focus on the system they are building; they can work with the simpler skeleton applications and be sure that their work will also be applicable to the real applications. In addition, skeleton applications support simple reproducible system experiments since they are represented by a compact set of parameters.
Our Application Skeleton tool currently can create easy-to-access, easy-to-build, and easy-to-run bag-of-task, (iterative) map-reduce, (iterative) multistage workflow applications. The tasks can be serial or parallel or a mix of both. We select three representative applications (Montage, BLAST, CyberShake Postprocessing), then describe and generate skeleton applications for each. We show that the skeleton applications have identical (or close) performance to that of the real applications. We then show examples of using skeleton applications to verify system optimizations such as data caching, I/O tuning, and task scheduling, as well as the system resilience mechanism, in some cases modifying the skeleton applications to emphasize some characteristic, and thus show that using skeleton applications simplifies the process of designing, implementing, and testing these optimizations.
D. S. Katz, A. M. Smith, "Implementing Transitive Credit with JSON-LD," 2nd Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2), in conjunction with SC14, arXiv:1407.5117 [cs.CY], 2014. URL: http://arxiv.org/abs/1407.5117
(This paper is open access)
Abstract: Science and engineering research increasingly relies on activities that facilitate research but are not currently rewarded or recognized, such as: data sharing; developing common data resources, software and methodologies; and annotating data and publications. To promote and advance these activities, we must develop mechanisms for assigning credit, facilitate the appropriate attribution of research outcomes, devise incentives for activities that facilitate research, and allocate funds to maximize return on investment. In this article, we focus on addressing the issue of assigning credit for both direct and indirect contributions, specifically by using JSON-LD to implement a prototype transitive credit system.
D. S. Katz, S.-C. T. Choi, H. Lapp, K. Maheshwari, F. Löffler, M. Turk, M. Hanwell, N. Wilkins-Diehr, J. Hetherington, J. Howison, S. Swenson, G. D. Allen, A. C. Elster, B. Berriman, C. Venters, "Summary of the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1)," Journal of Open Research Software, v.2(1): e6, pp. 1-21 arXiv:1404.7414 [cs.SE], 2014. DOI: 10.5334/jors.an
(This paper is open access)
Abstract: Challenges related to development, deployment, and maintenance of reusable software for science are becoming a growing concern. Many scientists’ research increasingly depends on the quality and availability of software upon which their works are built. To highlight some of these issues and share experiences, the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1) was held in November 2013 in conjunction with the SC13 Conference. The workshop featured keynote presentations and a large number (54) of solicited extended abstracts that were grouped into three themes and presented via panels. A set of collaborative notes of the presentations and discussion was taken during the workshop.
Unique perspectives were captured about issues such as comprehensive documentation, development and deployment practices, software licenses and career paths for developers. Attribution systems that account for evidence of software contribution and impact were also discussed. These include mechanisms such as Digital Object Identifiers, publication of “software papers”, and the use of online systems, for example source code repositories like GitHub. This paper summarizes the issues and shared experiences that were discussed, including cross-cutting issues and use cases. It joins a nascent literature seeking to understand what drives software work in science, and how it is impacted by the reward systems of science. These incentives can determine the extent to which developers are motivated to build software for the long-term, for the use of others, and whether to work collaboratively or separately. It also explores community building, leadership, and dynamics in relation to successful scientific software.
D. S. Katz, "Transitive Credit as a Means to Address Social and Technological Concerns Stemming from Citation and Attribution of Digital Products," Journal of Open Research Software, v.2(1): e20, pp. 1-4, 2014. DOI: 10.5334/jors.be
(This paper is open access)
Abstract: The pursuit of science and engineering research increasingly relies on activities that facilitate research but are not currently rewarded or recognized, such as development of products and infrastructure. In research publications, citations are used to credit previous works. This paper suggests that a modified citation system that includes the technological idea of transient credit could be used to recognize the developers of products other than research publications and that if this were done in a systematic manner, it would lead to social and cultural changes that would provide incentives for the further development of such products, accelerating overall scientific and engineering advances.
D. S. Katz and D. Proctor, "A Framework for Discussing e-Research Infrastructure Sustainability," Journal of Open Research Software, v.2(1): e13, pp. 1-4, 2014. DOI: 10.5334/jors.av
(This paper is open access)
Abstract: e-Research infrastructure is increasingly important in the conduct of science and engineering research, and in many disciplines has become an essential part of the research infrastructure. However, this e-Research infrastructure does not appear from a vacuum; it needs both intent and effort first to be created and then to be sustained over time. Research cultures and practices in many disciplines have not adapted to this new paradigm, due in part to the absence of a deep understanding of the elements of e-Research infrastructure and the characteristics that influence their sustainability. This paper outlines a set of contexts in which e-Research infrastructure can be discussed, proposes characteristics that must be considered to sustain infrastructure elements, and highlights models that may be used to create and sustain e-Research infrastructure. We invite feedback on the proposed characteristics and models presented herein.
S. J. Krieder, J. M. Wozniak, T. Armstrong, M. Wilde, D. S. Katz, B. Grimmer, I. T. Foster, I. Raicu, "Design and Evaluation of the GeMTC Framework for GPU-enabled Many-Task Computing," Proceedings of 23rd ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC'14), pp. 153-164, 2014. DOI: 10.1145/2600212.2600228
(This paper is not open access)
Abstract: We present the design and first performance and usability evaluation of GeMTC, a novel execution model and runtime system that enables accelerators to be programmed with many concurrent and independent tasks of potentially short or variable duration. With GeMTC, a broad class of such "many-task" applications can leverage the increasing number of accelerated and hybrid high-end computing systems. GeMTC overcomes the obstacles to using GPUs in a many-task manner by scheduling and launching independent tasks on hardware designed for SIMD-style vector processing. We demonstrate the use of a high-level MTC programming model (the Swift parallel dataflow language) to run tasks on many accelerators and thus provide a high-productivity programming model for the growing number of supercomputers that are accelerator-enabled. While still in an experimental stage, GeMTC can already support tasks of fine (subsecond) granularity and execute concurrent heterogeneous tasks on 86,000 independent GPU warps spanning 2.7M GPU threads on the Blue Waters supercomputer.
K. Maheshwari, J. M. Wozniak, H. Yang, D. S. Katz, M. Ripeanu, V. Zavala, M. Wilde, "Evaluating Storage Systems for Scientific Data in the Cloud," Proceedings of 5th Workshop on Scientific Cloud Computing (ScienceCloud) 2014, (in conjunction with HPDC'14), pp. 33-40, 2014. DOI: 10.1145/2608029.2608034
(This paper is not open access)
Abstract: Infrastructure-as-a-Service (IaaS) clouds are an appealing resource for scientific computing. However, the bare-bones presentation of raw Linux virtual machines leaves much to the application developer. For many cloud applications, effective data handling is critical to ecient application execution. This paper investigates the capabilities of a variety of POSIX-accessible distributed storage systems to manage data access patterns resulting from workflow application executions in the cloud. We leverage the expressivity of the Swift parallel scripting framework to benchmark the performance of a number of storage systems using synthetic workloads and three real-world applications. We characterize two representative commercial storage systems (Amazon S3 and HDFS, respectively) and two emerging research-based storage systems (Chirp/Parrot and MosaStore). We find the use of aggregated node-local resources effective and economical compared with remotely located S3 storage. Our experiments show that applications run at scale with MosaStore show up to 30% improvement in makespan time compared with those run with S3. We also find that storage-system driven application deployments in the cloud results in better runtime performance compared with an on-demand data-staging driven approach.
Z. Zhang, D. S. Katz, T. G. Armstrong, J. M. Wozniak, I. Foster, "Parallelizing the Execution of Sequential Scripts," Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC13), 2013. DOI: 10.1145/2503210.2503222
(This paper is not open access)
Abstract: Scripting is often used in science to create applications via the composition of existing programs. Parallel scripting systems allow the creation of such applications, but each system introduces the need to adopt a somewhat specialized programming model. We present an alternative scripting approach, AMFS Shell, that lets programmers express parallel scripting applications via minor extensions to existing sequential scripting languages, such as Bash, and then execute them in-memory on large-scale computers. We define a small set of commands between the scripts and a parallel scripting runtime system, so that programmers can compose their scripts in a familiar scripting language. The underlying AMFS implements both collective (fast file movement) and functional (transformation based on content) file management. Tasks are handled by AMFS's built-in execution engine. AMFS Shell is expressive enough for a wide range of applications, and the framework can run such applications efficiently on large-scale computers.
Z. Zhang and D. S. Katz, "Application Skeletons: Encapsulating MTC Application Task Computation and I/O," (best paper), Proceedings of 6th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS), (in conjunction with SC13), 2013. DOI: 10.5281/zenodo.540367
(This paper is open access)
Abstract: Scripting is often used in science to create applications via the composition of existing programs. Parallel scripting systems allow the creation of such applications, but each system introduces the need to adopt a somewhat specialized programming model. We present an alternative scripting approach, AMFS Shell, that lets programmers express parallel scripting applications via minor extensions to existing sequential scripting languages, such as Bash, and then execute them in-memory on large-scale computers. We define a small set of commands between the scripts and a parallel scripting runtime system, so that programmers can compose their scripts in a familiar scripting language. The underlying AMFS implements both collective (fast file movement) and functional (transformation based on content) file management. Tasks are handled by AMFS's built-in execution engine. AMFS Shell is expressive enough for a wide range of applications, and the framework can run such applications efficiently on large-scale computers.
D. S. Katz, G. Allen, N. Chue Hong, M. Parashar, D. Proctor, "First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE): Submission and Peer-Review Process, and Results," arXiv:1311.3523 [cs.SE], 2013. URL: http://arxiv.org/abs/1311.3523
(This paper is open access)
Abstract: This technical report discusses the submission and peer-review process used by the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE) and the results of that process. It is intended to record both this alternative model as well as the papers associated with the workshop that resulted from that process.
D. S. Katz, "Citation and Attribution of Digital Products: Social and Technological Concerns," (an accepted contribution to Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE), in conjuction with SC13), figshare, 2013. DOI: 10.6084/m9.figshare.791606
(This paper is open access)
Abstract: The pursuit of science increasingly relies on activities that facilitate science but are not currently rewarded or recognized. Of particular concern are the sharing of data; development of common data resources, software, and methodologies; and annotation of data and publications. This situation has been documented in a number of recent reports that focus on changing needs and mechanisms for attribution and citation of digital products, from the use of alternative metrics that track popularity, to work on data.
To promote such activities, we must develop mechanisms for assigning credit, facilitate the appropriate attribution of research outcomes, devise incentives for activities that facilitate research, and allocate funds to maximize return on investment. In this article, I introduce the idea of transitive credit, which addresses the issue of crediting indirect contributions, and discuss potential solutions to these other problems.
D. S. Katz, D. Proctor, "A Framework for Discussing e-Research Infrastructure Sustainability," (an accepted contribution to Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE), in conjuction with SC13), figshare, 2013. DOI: 10.6084/m9.figshare.790767
(This paper is open access)
K. Maheshwari, D. Kelly, S. J. Krieder, J. M. Wozniak, D. S. Katz, M. Zhi-Gang, M. Mookherjee, "Reusability in Science: From Initial User Engagement to Dissemination of Results," (an accepted contribution to Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE), in conjuction with SC13), arXiv:1309.1813 [cs.SE], 2013. URL: http://arxiv.org/abs/1309.1813
(This paper is open access)
Abstract: Effective use of parallel and distributed computing in science depends upon multiple interdependent entities and activities that form an ecosystem. Active engagement between application users and technology catalysts is a crucial activity that forms an integral part of this ecosystem. Technology catalysts play a crucial role benefiting communities beyond a single user group. An effective user-engagement, use and reuse of tools and techniques has a broad impact on software sustainability. From our experience, we sketch a life-cycle for user-engagement activity in scientific computational environment and posit that application level reusability promotes software sustainability. We describe our experience in engaging two user groups from different scientific domains reusing a common software and configuration on different computational infrastructures.
J. M. Wozniak, T. G. Armstrong, K. Maheshwari, E. L. Lusk, D. S. Katz, M. Wilde and I. T. Foster, "Turbine: A Distributed-Memory Dataflow Engine for High Performance Many-task Applications," Fundamenta Informaticae, v.128(3), pp. 337-366, 2013. DOI: 10.3233/FI-2013-949
(This paper is not open access)
Abstract: Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge. One approach is to structure applications with an upper layer of many loosely coupled coarse-grained tasks, each comprising a tightly-coupled parallel function or program. “Many-task” programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tightly coupled parallelism at the lower level through multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and intertask data movement is a significant performance challenge. In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with automated self-distribution and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems.
K. Kugler, K. Chard, S. Caton, O. Rana, D. S. Katz, "Constructing a Social Content Delivery Network for eScience," Proceedings of Analyzing and Improving Collaborative eScience with Social Networks (eSoN 13), 2013. DOI: 10.1109/eScience.2013.52
(This paper is not open access)
Abstract: Increases in the size of research data and the move towards citizen science, in which everyday users contribute data and analyses, have resulted in a research data deluge. Researchers must now carefully determine how to store, transfer and analyze “Big Data” in collaborative environments. This task is even more complicated when considering budget and locality constraints on data storage and access. In this paper we investigate the potential to construct a Social Content Delivery Network (S-CDN) based upon the social networks that exist between researchers. The S-CDN model builds upon the incentives of collaborative researchers within a given scientific community to address their data challenges collaboratively and in proven trusted settings. In this paper we present a prototype implementation of a S-CDN and investigate the performance of the data transfer mechanisms (using Globus Online) and the potential cost advantages of this approach.
Z. Zhang, D. S. Katz, M. Wilde, J. Wozniak, I Foster, "MTC Envelope: Defining the Capability of Large Scale Computers in the Context of Parallel Scripting Applications," Proceedings of 22nd International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'13), 2013. DOI: 10.1145/2493123.2462913
(This paper is not open access)
Abstract: Many scientific applications can be efficiently expressed with the parallel scripting (many-task computing, MTC) paradigm. These applications are typically composed of several stages of computation, with tasks in different stages coupled by a shared file system abstraction. However, we often see poor performance when running these applications on large scale computers due to the applications' frequency and volume of filesystem I/O and the absence of appropriate optimizations in the context of parallel scripting applications. In this paper, we show the capability of existing large scale computers to run parallel scripting applications by first defining the MTC envelope and then evaluating the envelope by benchmarking a suite of shared filesystem performance metrics. We also seek to determine the origin of the performance bottleneck by profiling the parallel scripting applications' I/O behavior and mapping the I/O operations to the MTC envelope. We show an example shared filesystem envelope and present a method to predict the I/O performance given the applications' level of I/O concurrency and I/O amount. This work is instrumental in guiding the development of parallel scripting applications to make efficient use of existing large scale computers, and to evaluate performance improvements in the hardware/software stack that will better facilitate parallel scripting applications.
J. M. Wozniak, M. Wilde, D. S. Katz, "JETS: Language and System Support for Many-Parallel-Task Workflows," Journal of Grid Computing (Special Issue on Scientific Workflows), v.11(3), pp. 341-360, 2013. DOI: 10.1007/s10723-013-9259-2
(This paper is not open access)
Abstract: Many-task computing is a well-established paradigm for implementing loosely coupled applications (tasks) on large-scale computing systems. However, few of the model’s existing implementations provide efficient, low-latency support for executing tasks that are tightly coupled multiprocessing applications. Thus, a vast array of parallel applications cannot readily be used effectively within many-task workloads. In this work, we present JETS, a middleware component that provides high performance support for many-parallel-task computing (MPTC). JETS is based on a highly concurrent approach to parallel task dispatch and on new capabilities now available in the MPICH2 MPI implementation and the ZeptoOS Linux operating system. JETS represents an advance over the few known examples of multilevel many-parallel-task scheduling systems: it more efficiently schedules and launches many short-duration parallel application invocations; it overcomes the challenges of coupling the user processes of each multiprocessing application invocation via the messaging fabric; and it concurrently manages many application executions in various stages. We report here on the JETS architecture and its performance on both synthetic benchmarks and an MPTC application in molecular dynamics.
S. Jha, M. Cole, D. S. Katz, M. Parashar, O. Rana, J. Weissman, "Distributed computing practice for large-scale science and engineering applications," Journal of Concurrency and Computation: Practice and Experience (CCPE), V.25(11), 2013. DOI: 10.1002/cpe.2897
(This paper is not open access)
Abstract: It is generally accepted that the ability to develop large-scale distributed applications has lagged seriously behind other developments in cyberinfrastructure. In this paper, we provide insight into how such applications have been developed and an understanding of why developing applications for distributed infrastructure is hard. Our approach is unique in the sense that it is centered around half a dozen existing scientific applications; we posit that these scientific applications are representative of the characteristics, requirements, as well as the challenges of the bulk of current distributed applications on production cyberinfrastructure (such as the US TeraGrid). We provide a novel and comprehensive analysis of such distributed scientific applications. Specifically, we survey existing models and methods for large-scale distributed applications and identify commonalities, recurring structures, patterns and abstractions. We find that there are many ad hoc solutions employed to develop and execute distributed applications, which result in a lack of generality and the inability of distributed applications to be extensible and independent of infrastructure details. In our analysis, we introduce the notion of application vectors: a novel way of understanding the structure of distributed applications. Important contributions of this paper include identifying patterns that are derived from a wide range of real distributed applications, as well as an integrated approach to analyzing applications, programming systems and patterns, resulting in the ability to provide a critical assessment of the current practice of developing, deploying and executing distributed applications. Gaps and omissions in the state of the art are identified, and directions for future research are outlined.
Z. Zhang, D. S. Katz, J. Wozniak, A. Espinosa, I Foster, "Design and Analysis of Data Management in Scalable Parallel Scripting," Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC12), 2012. DOI: 10.1109/SC.2012.44
(This paper is not open access)
Abstract: We seek to enable efficient large-scale parallel execution of applications in which a shared filesystem abstraction is used to couple many tasks. Such parallel scripting (many-task computing, MTC) applications suffer poor performance and utilization on large parallel computers because of the volume of filesystem I/O and a lack of appropriate optimizations in the shared filesystem. Thus, we design and implement a scalable MTC data management system that uses aggregated compute node local storage for more efficient data movement strategies. We co-design the data management system with the data-aware scheduler to enable dataflow pattern identification and automatic optimization. The framework reduces the time to solution of parallel stages of an astronomy data analysis application, Montage, by 83.2% on 512 cores; decreases the time to solution of a seismology application, CyberShake, by 7.9% on 2,048 cores; and delivers BLAST performance better than mpiBLAST at various scales up to 32,768 cores, while preserving the flexibility of the original BLAST application.
K. Chard, S. Caton, O. Rana, and D. S. Katz, "A Social Content Delivery Network for Scientific Cooperation: Vision, Design, and Architecture," Proceedings of the Third International Workshop on Data Intensive Computing in the Clouds (DataCloud 2012) (in conjunction with SC12), 2012. DOI: 10.1109/SC.Companion.2012.128
(This paper is not open access)
Abstract: Data volumes have increased so significantly that we need to carefully consider how we interact with, share, and analyze data to avoid bottlenecks. In contexts such as eScience and scientific computing, a large emphasis is placed on collaboration, resulting in many well-known challenges in ensuring that data is in the right place at the right time and accessible by the right users. Yet these simple requirements create substantial challenges for the distribution, analysis, storage, and replication of potentially "large" datasets. Additional complexity is added through constraints such as budget, data locality, usage, and available local storage. In this paper, we propose a "socially driven" approach to address some of the challenges within (academic) research contexts by defining a Social Data Cloud and underpinning Content Delivery Network: a Social CDN (SCDN). Our approach leverages digitally encoded social constructs via social network platforms that we use to represent (virtual) research communities. Ultimately, the S-CDN builds upon the intrinsic incentives of members of a given scientific community to address their data challenges collaboratively and in proven trusted settings. We define the design and architecture of a SCDN and investigate its feasibility via a coauthorship case study as first steps to illustrate its usefulness.
Z. Yun, Z. Lei, G. Allen, D. S. Katz, J. Ramanujam, "DA-TC: a novel application execution model in multicluster systems," Cluster Computing, v.15(4), 2012. DOI: 10.1007/s10586-012-0228-5
(This paper is not open access)
Abstract: The availability of a large number of separate clusters has given rise to the field of multicluster systems in which these resources are coupled to obtain their combined benefits to solve large-scale compute-intensive applications. However, it is challenging to achieve automatic load balancing of the jobs across these participating autonomic systems. We developed a novel user space execution model named DA-TC to address the workload allocation techniques for the applications with large number of sequential jobs in multicluster systems. Through this model, we can achieve dynamic load balancing for task assignment, and slower resources become beneficial factors rather than bottlenecks for application execution. The effectiveness of this strategy is demonstrated through theoretical analysis. This model is also evaluated through extensive experimental studies and the results show that when compared with the traditional method, the proposed DA-TC model can significantly improve the performance of application execution in terms of application turnaround time and system reliability in multicluster circumstances.
D. S. Katz and G. Allen, "Experiences and Lessons from the Center for Computation & Technology," Research Trends, (30), pp. 13-17, September 2012.
(This paper is not open access)
M. Santcroos, S. D. Olabarriaga, S. Jha, and D. S. Katz, "Pilot Abstractions for Compute, Data, and Network," Collaborative Research using eScience Infrastructure and High Speed Networks Workshop, in conjunction with 8th IEEE International Conference on eScience, 2012. DOI: 10.1109/eScience.2012.6404459
(This paper is not open access)
Abstract: Scientific experiments in a variety of domains are producing increasing amounts of data that need to be processed efficiently. Distributed Computing Infrastructures are increasingly important in fulfilling these large-scale computational requirements.
T. G. Armstrong, J. M. Wozniak, M. Wilde, K. Maheshwari, D. S. Katz, M. Ripeanu, E. L. Lusk, and I. T. Foster, "ExM: High Level Dataflow Programming for Extreme-Scale Systems," 4th USENIX Workshop on Hot Topics in Parallelism (HotPar '12), 2012.
(This paper is not open access)
Abstract: We present here the ExM (extreme-scale many-task) programming and execution model as a practical solution to the challenges of programing the higher-level logic of complex parallel applications on current petascale and future exascale computing systems. ExM provides an expressive, high-level functional programming model that yields massive concurrency through implicit, automated parallelism. It comprises a judicious integration of dataflow constructs, highly parallel function evaluation, and extremely scalable task generation. It directly addresses the intertwined programmability and scalability requirements of systems with massive concurrency, while providing a programming model that may be attractive and feasible for systems of much lower scale. We describe here the benefits of the ExM programming and execution model, its potential applications, and the performance of its current implementation.
J. M. Wozniak, T. G. Armstrong, K. Maheshwari, E. L. Lusk, D. S. Katz, M. Wilde and I. T. Foster, "Turbine: A Distributed-Memory Dataflow Engine for Extreme-Scale Many-Task Applications," Proceedings of Scalable Workflow Enactment Engines and Technologies (SWEET), SIGMOD 2012, 2012. DOI: 10.1145/2443416.2443421
(This paper is not open access)
Abstract: Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge. One approach is to structure applications with an upper-layer of many loosely-coupled coarse-grained tasks, each comprising a tightly coupled parallel function or program. "Many-task" programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tighly-coupled parallelism at the lower level via multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and inter-task data movement is a significant performance challenge. In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with automated self-distribution, and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems.
K. Maheshwari, A. Espinosa, D. S. Katz, M. Wilde, Z. Zhang, I. Foster, S. Callaghan, and P. Maechling, "Job and Data Clustering for Aggregate Use of Multiple Production Cyberinfrastructures," Proceedings of Fifth International Workshop on Data Intensive Distributed Computting (DIDC'12), pp. 3-11, 2012. DOI: 10.1145/2286996.2287000
(This paper is not open access)
Abstract: In this paper, we address the challenges of reducing the time-to-solution of the data intensive earthquake simulation workflow "CyberShake" by supplementing the high-performance parallel computing (HPC) resources on which it typically runs with distributed, heterogeneous resources that can be obtained opportunistically from grids and clouds. We seek to minimize time to solution by maximizing the amount of work that can be efficiently done on the distributed resources. We identify data movement as the main bottleneck in effectively utilizing the combined local and distributed resources. We address this by analyzing the I/O characteristics of the application, processor acquisition rate (from a pilot-job service), and the data movement throughput of the infrastructure. With these factors in mind, we explore a combination of strategies including partitioning of computation (over HPC and distributed resources) and job clustering. We validate our approach with a theoretical study and with preliminary measurements on the Ranger HPC system and distributed Open Science Grid resources.
E. Vairavanathan, S. Al-Kiswany, L. Costa, M. Ripeanu, Z. Zhang, D. S. Katz, M. Wilde, "A Workflow-Aware Storage System: An Opportunity Study," Proceedings of the 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 326-334, 2012. DOI: 10.1109/CCGrid.2012.109
(This paper is not open access)
Abstract: This paper evaluates the potential gains a workflow-aware storage system can bring. Two observations make us believe such storage system is crucial to efficiently support workflow-based applications: First, workflows generate irregular and application-dependent data access patterns. These patterns render existing storage systems unable to harness all optimization opportunities as this often requires conflicting optimization options or even conflicting design decision at the level of the storage system. Second, when scheduling, workflow runtime engines make suboptimal decisions as they lack detailed data location information.
This paper discusses the feasibility, and evaluates the potential performance benefits brought by, building a workflow-aware storage system that supports per-file access optimizations and exposes data location. To this end, this paper presents approaches to determine the application-specific data access patterns, and evaluates experimentally the performance gains of a workflow-aware storage approach. Our evaluation using synthetic benchmarks shows that a workflow-aware storage system can bring significant performance gains: up to 7x performance gain compared to the distributed storage system - MosaStore and up to 16x compared to a central, well provisioned, NFS server.
Z. Zhang, D. S. Katz, M. Ripeanu, M. Wilde, I. Foster, "AME: An Anyscale Many-Task Computing Engine," Proceedings of the 6th Workshop on Workflows in Support of Large-Scale Science (WORKS'11), 2011. DOI: 10.1145/2110497.2110513
(This paper is not open access)
Abstract: Many-Task Computing (MTC) is a new application category that encompasses increasingly popular applications in biology, economics, and statistics. The high inter-task parallelism and data-intensive processing capabilities of these applications pose new challenges to existing supercomputer hardware-software stacks. These challenges include resource provisioning; task dispatching, dependency resolution, and load balancing; data management; and resilience.
This paper examines the characteristics of MTC applications which create these challenges, and identifies related gaps in the middleware that supports these applications on extreme-scale systems. Based on this analysis, we propose AME, an Anyscale MTC Engine, which addresses the scalability aspects of these gaps. We describe the AME framework and present performance results for both synthetic benchmarks and real applications. Our results show that AME's dispatching performance linearly scales up to 14,120 tasks/second on 16,384 cores with high efficiency. The overhead of the intermediate data management scheme does not increase significantly up to 16,384 cores. AME eliminates 73% of the file transfer between compute nodes and the global filesystem for the Montage astronomy application running on 2,048 cores. Our results indicate that AME scales well on today's petascale machines, and is a strong candidate for exascale machines.
M. Wilde, N. Hategan, J. M. Wozniak, B. Clifford, D. S. Katz, I. Foster, "Swift: A language for distributed parallel scripting," Parallel Computing, v.37(9), pp. 633-652, 2011. DOI: 10.1016/j.parco.2011.05.005
(This paper is not open access)
Abstract: Scientists, engineers, and statisticians must execute domain-specific application programs many times on large collections of file-based data. This activity requires complex orchestration and data management as data is passed to, from, and among application invocations. Distributed and parallel computing resources can accelerate such processing, but their use further increases programming complexity. The Swift parallel scripting language reduces these complexities by making file system structures accessible via language constructs and by allowing ordinary application programs to be composed into powerful parallel scripts that can efficiently utilize parallel and distributed resources. We present Swift’s implicitly parallel and deterministic programming model, which applies external applications to file collections using a functional style that abstracts and simplifies distributed parallel execution.
D. S. Katz, D. Hart, C. Jordan, A. Majumdar, J.P. Navarro, W. Smith, J. Towns, V. Welch, N. Wilkins-Diehr, "Cyberinfrastructure Usage Modalities on the TeraGrid," 2011 High-Performance Grid and Cloud Computing Workshop, Proceedings of 2011 IPDPS Workshops, pp. 927-934, 2011. DOI: 10.1109/IPDPS.2011.239
(This paper is not open access)
Abstract: This paper is intended to explain how the TeraGrid would like to be able to measure "usage modalities." We would like to (and are beginning to) measure these modalities to understand what objectives our users are pursuing, how they go about achieving them, and why, so that we can make changes in the TeraGrid to better support them.
S. Jha, D. S. Katz, A. Luckow, A. Merzky, and K. Stamou, "Understanding Scientific Applications for Cloud Environments," in Cloud Computing: Principles and Paradigms, (Editors: R. Buyya, J. Broberg, and A. Goscinski,) pp. 345-371, Wiley, New York, 2011.
D. S. Katz, S. Callaghan, R. Harkness, S. Jha, K. Kurowski, S. Manos, S. Pamidighantam, M. Pierce, B. Plale, C. Song, J. Towns, "Science on the TeraGrid," Computational Methods in Science and Technology (CMST), v.Special Issue 2010, pp. 81-97, 2010.
(is not open access)
Abstract: The TeraGrid is an advanced, integrated, nationally-distributed, open, user-driven, US cyberinfrastructure that enables and supports leading edge scientific discovery and promotes science and technology education. It comprises supercomputing resources, storage systems, visualization resources, data collections, software, and science gateways, integrated by software systems and high bandwidth networks, coordinated through common policies and operations, and supported by technology experts. This paper discusses the TeraGrid itself, examples of the science that is occurring on the TeraGrid today, and applications that are being developed to perform science in the future.
P. Balaji, W. Feng, H. Lin, J. Archuleta, S. Matsuoka, A. Warren, J. Setubal, E. Lusk, R. Thakur, I. Foster, D. S. Katz, S. Jha, K. Shinpaugh, S. Coghlan, D. Reed, "Global-scale Distributed I/O with ParaMEDIC," Concurrency and Computation: Practice and Experience, v.22(16), pp. 2266-2281, 2010. DOI: 10.1002/cpe.1590
(This paper is not open access)
Abstract: Achieving high performance for distributed I/O on a wide-area network continues to be an elusive holy grail. Despite enhancements in network hardware as well as software stacks, achieving high-performance remains a challenge. In this paper, our worldwide team took a completely new and non-traditional approach to distributed I/O, called ParaMEDIC: Parallel Metadata Environment for Distributed I/O and Computing, by utilizing application-specific transformation of data to orders of magnitude smaller metadata before performing the actual I/O. Specifically, this paper details our experiences in deploying a large-scale system to facilitate the discovery of missing genes and constructing a genome similarity tree by encapsulating the mpiBLAST sequence-search algorithm into ParaMEDIC. The overall project involved nine computational sites spread across the U.S. and generated more than a petabyte of data that was 'teleported' to a large-scale facility in Tokyo for storage.
J. C. Jacob, D. S. Katz, G. B. Berriman, J. Good, A. C. Laity, E. Deelman, C. Kesselman, G. Singh, M.-H. Su, T. A. Prince, R. Williams, "Montage: A Grid Portal and Software Toolkit for Science-Grade Astronomical Image Mosaicking," International Journal of Computational Science and Engineering, v. 4(2), pp. 73-87, 2009. DOI: 10.1504/IJCSE.2009.026999
(This paper is open access in the US)
Abstract: Montage is a portable software toolkit for constructing custom, science-grade mosaics by composing multiple astronomical images. The mosaics constructed by Montage preserve the astrometry (position) and photometry (intensity) of the sources in the input images. The mosaic to be constructed is specified by the user in terms of a set of parameters, including dataset and wavelength to be used, location and size on the sky, coordinate system and projection, and spatial sampling rate. Many astronomical datasets are massive, and are stored in distributed archives that are, in most cases, remote with respect to the available computational resources. Montage can be run on both single- and multi-processor computers, including clusters and grids. Standard grid tools are used to run Montage in the case where the data or computers used to construct a mosaic are located remotely on the Internet. This paper describes the architecture, algorithms, and usage of Montage as both a software toolkit and as a grid portal. Timing results are provided to show how Montage performance scales with number of processors on a cluster computer. In addition, we compare the performance of two methods of running Montage in parallel on a grid.
S. Jha, D. S. Katz, M. Parashar, O. Rana, and J. Weissman, "Critical Perspectives on Large-Scale Distributed Applications and Production Grids," (Best Paper Award Winner), Proceedings of the 10th IEEE/ACM International Conference on Grid Computing (Grid 2009), 2009. DOI: 10.1109/GRID.2009.5353064
(This paper is not open access)
Abstract: It is generally accepted that the ability to develop large-scale distributed applications that are extensible and independent of infrastructure details has lagged seriously behind other developments in cyberinfrastructure. As the sophistication and scale of distributed infrastructure increases, the complexity of successfully developing and deploying distributed applications increases both quantitatively and in qualitatively newer ways. In this paper we trace the evolution of a representative set of "state-of-the-art" distributed applications and production infrastructure; in doing so we aim to provide insight into the evolving sophistication of distributed applications – from simple generalizations of legacy static high-performance to applications composed of multiple loosely-coupled and dynamic components. The ultimate aim of this work is to highlight that even accounting for the fact that developing applications for distributed infrastructure is a difficult undertaking, there are suspiciously few novel and interesting distributed applications that utilize production Grid infrastructure. Along the way, we aim to provide an appreciation for the fact that developing distributed applications and the theory and practise of production Grid infrastructure have often not progressed in phase. Progress in the next phase and generation of distributed applications will require stronger coupling between the design and implementation of production infrastructure and the theory of distributed applications, including but not limited to explicit support for distributed application usage modes and advances that enable distributed applications to scale-out.
D. S. Katz, G. Allen, R. Cortez, C. Cruz-Neira, R. Gottumukkala, Z. D. Greenwood, L. Guice, S. Jha, R. Kolluru, T. Kosar, L. Leger, H. Liu, C. McMahon, J. Nabrzyski, B. Rodriguez-Milla, E. Seidel, G. Speyrer, M. Stubblefield, B. Voss, and S. Whittenburg, "Louisiana: A Model for Advancing Regional e-Research through Cyberinfrastructure," Philosophical Transactions of the Royal Society A, v. 367, pp. 2459-2469, 2009. DOI: 10.1098/rsta.2009.0037,
(This paper is not open access)
Abstract: Louisiana researchers and universities are leading a concentrated, collaborative effort to advance statewide e-Research through a new cyberinfrastructure: computing systems, data storage systems, advanced instruments and data repositories, visualization environments, and people, all linked together by software and high performance networks. This effort has led to a set of interlinked projects that have started making a significant difference to the state and has created an environment that encourages increased collaboration, leading to new e-Research. This paper describes the overall effort, the new projects and environment, and the results to-date.
P. Chakraborty, S. Jha, and D. S. Katz, "Novel Submission Modes for Tightly-Coupled Jobs Across Distributed Resources for Reduced Time-to-Solution," Philosophical Transactions of the Royal Society A, v. 367, pp. 2545-2556, 2009. DOI: 10.1098/rsta.2009.0054,
(This paper is not open access)
Abstract: The problems of scheduling a single parallel job across a large scale distributed system are well known and surprisingly difficult to solve. In addition, because of the issues involved with distributed submission like co-reserving resources, managing accounts and certificates simultaneously on multiple machines etc., the vast number of HPC-application users have been happy to remain restricted to submitting jobs to single machines. Meanwhile, the need to simulate larger and more complex physical systems continues to grow, with a concomitant increase in the number of cores required to solve the resulting scientific problems. One might reduce the demand on load per machines, and eventually the wait-time on queue by decomposing the problem to utilise two resources in such circumstances, even though there might be a reduction in the peak performance. This motivates the question: can otherwise monolithic jobs running on single resources be distributed over more than one machine such that there is an overall reduction in the time-to-solution? In this paper, we briefly discuss the development and performance of a parallel molecular dynamics code and its generalisation to work on multiple distributed machines (using MPICH-G2). We benchmark and validate the performance of our simulations over multiple input-data sets of varying size. The primary aim of this work however, is to show that the time-to-solution can be reduced by sacrificing some peak performance and distributing over multiple machines.
P. Balaji, W. Feng, H. Lin, J. Archuleta, S. Matsuoka, A. Warren, J. Setubal, E. Lusk, R. Thakur, I. Foster, D. S. Katz, S. Jha, K. Shinpaugh, S. Coghlan and D. Reed, "Distributed I/O with ParaMEDIC: Experiences with a Worldwide Supercomputer," Proceedings of 23rd Annual International Supercomputing Conference (ISC), (Outstanding Paper Award), 2008.
(This paper is not open access)
Abstract: Achieving high performance for distributed I/O on a wide-area network continues to be an elusive holy grail. Despite enhancements in network hardware as well as software stacks, achieving high-performance remains a challenge. In this paper, our worldwide team took a completely new and non-traditional approach to distributed I/O, called ParaMEDIC: Parallel Metadata Environment for Distributed I/O and Computing, by utilizing application-speciﬁc transformation of data to orders-of-magnitude smaller meta-data before performing the actual I/O. Speciﬁcally, this paper details our experiences in deploying a large-scale system to facilitate the discovery of missing genes and constructing a genome similarity tree by encapsulating the mpiBLAST sequence-search algorithm into ParaMEDIC. The overall project involved nine different computational sites spread across the U.S. generating more than a petabyte of data, that was "teleported" to a large-scale facility in Tokyo for storage.
G. Singh, M.-H. Su, K. Vahi, E. Deelman, B. Berriman, J. Good, D. S. Katz, and G. Mehta, "Workflow Task Clustering for Best Effort Systems with Pegasus," Proceedings of 15th Mardi Gras Conference, 2008. DOI: 10.1145/1341811.1341822
(This paper is not open access)
Abstract: Many scientific workflows are composed of fine computational granularity tasks, yet they are composed of thousands of them and are data intensive in nature, thus requiring resources such as the TeraGrid to execute efficiently. In order to improve the performance of such applications, we often employ task clustering techniques to increase the computational granularity of workflow tasks. The goal is to minimize the completion time of the workflow by reducing the impact of queue wait times. In this paper, we examine the performance impact of the clustering techniques using the Pegasus workflow management system. Experiments performed using an astronomy workflow on the NCSA TeraGrid cluster show that clustering can achieve a significant reduction in the workflow completion time (up to 97%).
L. Battestilli, A. Hutanu, G. Karmous-Edwards, D. S. Katz, J. MacLaren, J. Mambretti, J. H. Moore, S.-J. Park, H. G. Perros, S. Sundar, S. Tanwir, S. R. Thorpe, and Y. Xin, "EnLIGHTened Computing: An Architecture for Co-allocating Network, Compute, and other Grid Resources for High-End Applications," Proceedings of IEEE Honet'07, Dubai, UAE, 2007. DOI: 10.1109/HONET.2007.4600261
(This paper is not open access)
Abstract: Many emerging high performance applications require distributed infrastructure that is significantly more powerful and flexible than traditional Grids. Such applications require the optimization, close integration, and control of all Grid resources, including networks. The EnLIGHTened (ENL) Computing Project has designed an architectural framework that allows Grid applications to dynamically request (in-advance or on-demand) any type of Grid resource: computers, storage, instruments, and deterministic, high-bandwidth network paths, including lightpaths. Based on application requirements, the ENL middleware communicates with Grid resource managers and, when availability is verified, co-allocates all the necessary resources. ENL’s Domain Network Manager controls all network resource allocations to dynamically setup and delete dedicated circuits using Generalized Multiprotocol Label Switching (GMPLS) control plane signaling. In order to make optimal brokering decisions, the ENL middleware uses near-real-time performance information about Grid resources. A prototype of this architectural framework on a national-scale testbed implementation has been used to demonstrate a small number of applications. Based on this, a set of changes for the middleware have been laid out and are being implemented.
G. Bruce Berriman, J. C. Good, A. C. Laity, J. C. Jacob, D. S. Katz, E. Deelman, G. Singh, and M.-H. Su, "Montage: An Astronomical Image Mosaic Engine," The National Virtual Observatory Book (eds.: M. J. Graham, M. J. Fitzpatrick, and T. A. McGlynn,) chapter 19, APS Conference Series, v. 382, 2007.
G. Singh, K. Vahi, A. Ramakrishnan, G. Mehta, E. Deelman, H. Zhao, R. Sakellariou, K. Blackburn, D. Brown, S. Fairhurst, D. Meyers, B. Berriman, J. Good, and D. S. Katz, "Optimizing Workflow Data Footprint," Scientific Programming, v. 15(4), pp. 249-268, 2007.
(This paper is not open access)
Abstract: In this paper we examine the issue of optimizing disk usage and scheduling large-scale scientific workflows onto distributed resources where the workflows are data-intensive, requiring large amounts of data storage, and the resources have limited storage resources. Our approach is two-fold: we minimize the amount of space a workflow requires during execution by removing data files at runtime when they are no longer needed and we demonstrate that workflows may have to be restructured to reduce the overall data footprint of the workflow. We show the results of our data management and workflow restructuring solutions using a Laser Interferometer Gravitational-Wave Observatory (LIGO) application and an astronomy application, Montage, running on a large-scale production grid-the Open Science Grid. We show that although reducing the data footprint of Montage by 48% can be achieved with dynamic data cleanup techniques, LIGO Scientific Collaboration workflows require additional restructuring to achieve a 56% reduction in data space usage. We also examine the cost of the workflow restructuring in terms of the application's runtime.
G. B. Berriman, E. Deelman, J. Good, J. C. Jacob, D. S. Katz, A. C. Laity, T. A. Prince, G. Singh, and M.-H. Su, "Generating Complex Astronomy Workflows," Workflows for e-Science, (Editors: I. J. Taylor, E. Deelman, D. B. Gannon, and M. Shields,) pp. 19-38, Springer London, 2007.
(This paper is not open access)
Abstract: Astronomy has a rich heritage of discovery using image data sets that cover the full range of the electromagnetic spectrum. Image data sets in one frequency range have often been studied in isolation from those in other frequency ranges. This is mostly a consequence of the diverse properties of the data collections themselves. Images are delivered in different coordinate systems, map projections, spatial samplings, and image sizes, and the pixels themselves are rarely co-registered on the sky. Moreover, the spatial extent of many astronomically important structures, such as clusters of galaxies and star formation regions, is often substantially greater than that of individual images.
D. S. Katz, J. C. Jacob, P. P. Li, Y. Chao, G. Allen, "Data-Oriented Distributed Computing for Science: Reality and Possibilities," On the Move to Meaningful Internet Systems 2006: CoopIS, DOA, GADA, and ODBASE, Lecture Notes in Computer Science, v. 4276, pp. 1119-1124, 2006. DOI: 10.1007/11914952_2
(This paper is not open access)
Abstract: As is becoming commonly known, there is an explosion happening in the amount of scientific data that is publicly available. One challenge is how to make productive use of this data. This talk will discuss some parallel and distributed computing projects, centered around virtual astronomy, but also including other scientific data-oriented realms. It will look at some specific projects from the past, including Montage, Grist, OurOcean, and SCOOP, and will discuss the distributed computing, Grid, and Web-service technologies that have successfully been used in these projects.
G. B. Berriman, J. C. Good, A. C. Laity, J. C. Jacob, D. S. Katz, E. Deelman, G. Singh, M.-H. Su, and T. Prince, "The Design and Applications of Montage: An Astronomical Image Mosaic Engine," Proceedings of the 2006 Earth Science Technology Conference (ESTC-06), 2006.
(This paper is not open access)
Abstract: Montage is a portable toolkit for constructing custom, science-grade mosaics by composing multiple astronomical images. The mosaics constructed by Montage preserve the astrometry (position) and photometry (intensity) of the sources in the input images. The mosaic to be constructed is specified by the user in terms of a set of parameters, including dataset and wavelength to be used, location and size on the sky, coordinate system and projection, and spatial sampling rate. Many astronomical datasets are massive, and are stored in distributed archives that are, in most cases, remote with respect to the available computational resources. The paper describes scientific applications of Montage by NASA projects and researchers, who run the software on both single- and multi-processor computers, including clusters and grids. Standard grid tools are used to run Montage in the case where the data or computers used to construct a mosaic are located remotely on the Internet. This paper describes the architecture, algorithms, and performance of Montage as both a software toolkit and as a grid portal.
D. E. Bernholdt, B. A. Allan, R. Armstrong, F. Bertrand, K. Chiu, T. L. Dahlgren, K. Damevski, W. R. Elwasif, T. G. W. Epperly, M. Govindaraju, D. S. Katz, J. A. Kohl, M. Krishnan, G. Kumfert, J. W. Larson, S. Lefantzi, M. J. Lewis, A. D. Malony, L. C. McInnes, J. Nieplocha, B. Norris, S. G. Parker, J. Ray, S. Shende, T. L. Windus, and S. Zhou, "A Component Architecture for High-Performance Scientific Computing," International Journal of High Performance Computing Applications, v. 20(2), pp. 163-202, Summer 2006. DOI: 10.1177/1094342006064488
(This paper is not open access)
Abstract: The Common Component Architecture (CCA) provides a means for software developers to manage the complexity of large-scale scientific simulations and to move toward a plug-and-play environment for high-performance computing. In the scientific computing context, component models also promote collaboration using independently developed software, thereby allowing particular individuals or groups to focus on the aspects of greatest interest to them. The CCA supports parallel and distributed computing as well as local high-performance connections between components in a language-independent manner. The design places minimal requirements on components and thus facilitates the integration of existing code into the CCA environment. The CCA model imposes minimal overhead to minimize the impact on application performance. The focus on high performance distinguishes the CCA from most other component models. The CCA is being applied within an increasing range of disciplines, including combustion research, global climate simulation, and computational chemistry.
A. C. Laity, G. B. Berriman, J. C. Good, D. S. Katz, J. C. Jacob, L. Brieger, R. Moore, R. Williams, E. Deelman, G. Singh, and M.-H. Su, "An All-Sky 2MASS Mosaic Constructed on the TeraGrid," Astronomical Data Analysis Software & Systems (ADASS) XVI, 2006.
(This paper is not open access)
Abstract: The Montage mosaic engine supplies on-request image mosaic services for the NVO astronomical community. A companion paper describes scientific applications of Montage. This paper describes one application in detail: the generation at SDSC of a mosaic of the 2MASS All-sky Image Atlas on the NSF TeraGrid. The goals of the project are: to provide a value-added 2MASS product that combines overlapping images to improve sensitivity; to demonstrate applicability of computing at-scale to astronomical missions and surveys, especially projects such as LSST; and to demonstrate the utility of the NVO Hyperatlas format. The numerical processing of an 8-TB 32-bit survey to produce a 64-bit 20-TB output atlas presented multiple scaleability and operational challenges. An MPI Python module, MYMPI, was used to manage the alternately sequential and parallel steps of the Montage process. This allowed us to parallelize all steps of the mosaic process: that of many, sequential steps executing simultaneously for independent mosaics and that of a single MPI parallel job executing on many CPUs for a single mosaic. The Storage Resource Broker (SRB) developed at SDSC has been used to archive the output results in the Hyperatlas. The 2MASS mosaics are now being assessed for scientific quality. The input images consist of 4,121,440 files, each 2MB in size. The input files that fall on mosaic boundaries are opened, read, and used multiple times in the processing of adjacent mosaics, so that a total of 14 TB in 6,275,494 files are actually opened and read in the creation of mosaics across the entire survey. Around 130,000 CPU-hours were used to complete the mosaics. The output consists of 1734 6-degree plates for each of 3 bands. Each of the 5202 mosaics is roughly 4 GB in size, and each has been tiled into a 12x12 array of 26-MB files for ease of handling. The total size is about 20 TB in 750,000 tiles.
E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz, "Pegasus: a Framework for Mapping Complex Scientific Workflows onto Distributed Systems," Scientific Programming, v.13(3), pp. 219-237, November 2005. doi: 10.1155/2005/128026
(This paper is not open access)
Abstract: This paper describes the Pegasus framework that can be used to map complex scientific workflows onto distributed resources. Pegasus enables users to represent the workflows at an abstract level without needing to worry about the particulars of the target execution systems. The paper describes general issues in mapping applications and the functionality of Pegasus. We present the results of improving application performance through workflow restructuring.
D. S. Katz, N. Anagnostou, G. B. Berriman, E. Deelman, J. Good, J. C. Jacob, C. Kesselman, A. Laity, T. A. Prince, G. Singh, M.-H. Su, and R. Williams, "Astronomical Image Mosaicking on a Grid: Initial Experiences," Engineering the Grid - Status and Perspective, (Editors: B. Di Martino, J. Dongarra, A. Hoisie, L. Yang, and H. Zima,) American Scientific Publishers, 2006.
(This chapter is not open access)
Abstract: This chapter discusses some grid experiences in solving the problem of generating large astronomical image mosaics by composing multiple small images, from the team that has developed Montage (http://montage.ipac.caltech.edu/). The problem of generating these mosaics is complex in that individual images must be projected into a common coordinate space, overlaps between images calculated, the images processed so that the backgrounds match, and images composed while using a variety of techniques to handle the presence of multiple pixels in the same output space. To accomplish these tasks, a suite of software tools called Montage has been developed. The modules in this suite can be run on a single processor computer using a simple shell script, and can additionally be run using a combination of parallel approaches. These include running MPI versions of some modules, and using standard grid tools. In the latter case, processing workflows are automatically generated, and appropriate data sources are located and transferred to a variety of parallel processing environments for execution. As a result, it is now possible to generate large-scale mosaics on-demand in timescales that support iterative, scientific exploration. In this chapter, we describe Montage, how it was modified to execute in the grid environment, the tools that were used to support its execution, as well as performance results.
D. S. Katz, G. B. Berriman, E. Deelman, J. Good, J. C. Jacob, C. Kesselman, A. C. Laity, T. A. Prince, G. Singh, and M.-H. Su, "A Comparison of Two Methods for Building Astronomical Image Mosaics on a Grid," Proceedings of 34th International Conference on Parallel Processing Workshops , pp. 85-94, 2005. DOI: 10.1109/ICPPW.2005.6
(This paper is open access in the US)
Abstract: This paper compares two methods for running an application composed of a set of modules on a grid. The set of modules (collectively called Montage) generates large astronomical image mosaics by composing multiple small images. The workflow that describes a particular run of Montage can be expressed as a directed acyclic graph (DAG), or as a short sequence of parallel (MPI) and sequential programs. In the first case, Pegasus can be used to run the workflow. In the second case, a short shell script that calls each program can be run. In this paper, we discuss the Montage modules, the workflow run for a sample job, and the two methods of actually running the workflow. We examine the run time for each method and compare the portions that differ between the two methods.
G. Singh, E. Deelman, G. Mehta, K. Vahi, M.H.-Su, G. B. Berriman, G. Good, J. Jacob, D. S. Katz, A. Lazzarini, K. Blackburn, and S. Koranda, "The Pegasus Portal: Web Based Grid Computing," Proceedings of the The 20th Annual ACM Symposium on Applied Computing (SAC 2005), pp. 680-686, 2005. DOI: 10.1145/1066677.1066834
(This paper is not open access)
Abstract: Pegasus is a planning framework for mapping abstract workflows for execution on the Grid. This paper presents the implementation of a web-based portal for submitting workflows to the Grid using Pegasus. The portal also includes components for generating abstract workflows based on a metadata description of the desired data products and application-specific services. We describe our experiences in using this portal for two Grid applications. A major contribution of our work is in introducing several components that can be useful for Grid portals and hence should be included in Grid portal development toolkits.
A. C. Laity, N. Anagnostou, G. B. Berriman, J. C. Good, J. C. Jacob, D. S. Katz, and T. Prince "Montage: An Astronomical Image Mosaic Service for the NVO," Astronomical Data Analysis Software & Systems (ADASS) XIV, 2004.
(This paper is not open access)
Abstract: Montage is a software system for generating astronomical image mosaics according to user-specified size, rotation, WCS-compliant projection and coordinate system, with background modeling and rectification capabilities. Its architecture has been described in the proceedings of ADASS XII and XIII. It has been designed as a toolkit, with independent modules for image reprojection, background rectification and coaddition, and will run on workstations, clusters and grids. The primary limitation of Montage thus far has been in the projection algorithm. It uses a spherical trigonometry approach that is general at the expense of speed. The reprojection algorithm has now been made 30 times faster for commonly used tangent plane to tangent plane reprojections that cover up to several square degrees, through modification of a custom algorithm first derived by the Spitzer Space Telescope. This focus session will describe this algorithm, demonstrate the generation of mosaics in real time, and describe applications of the software. In particular, we will highlight one case study which shows how Montage is supporting the generation of science-grade mosaics of images measured with the Infrared Array Camera aboard the Spitzer Space Telescope.
J. C. Jacob, R. Williams, J. Babu, S. G. Djorgovski, M. J. Graham, D. S. Katz, A. Mahabal, C. D. Miller, R. Nichol, D. E. Vanden Berk, and H. Walia, "Grist: Grid Data Mining for Astronomy," Astronomical Data Analysis Software & Systems (ADASS) XIV, 2004.
(This paper is open access in the US)
Abstract: The Grist project is developing a grid-technology based system as a research environment for astronomy with massive and complex datasets. This knowledge extraction system will consist of a library of distributed grid services controlled by a workflow system, compliant with standards emerging from the grid computing, web services, and virtual observatory communities. This new technology is being used to find high redshift quasars, study peculiar variable objects, search for transients in real time, and fit SDSS QSO spectra to measure black hole masses. Grist services are also a component of the "hyperatlas" project to serve high-resolution multi-wavelength imagery over the Internet. In support of these science and outreach objectives, the Grist framework will provide the enabling fabric to tie together distributed grid services in the areas of data access, federation, mining, subsetting, source extraction, image mosaicking, statistics, and visualization.
G. B. Berriman, E. Deelman, J. Good, J. Jacob, D. S. Katz, C. Kesselman, A. Laity, T. A. Prince, G. Singh, M. Su, "Montage: A Grid Enabled Engine for Delivering Custom Science-Grade Mosaics on Demand," Proceedings of the SPIE Conference on Astronomical Telescopes and Instrumentation, SPIE, 2004. DOI: 10.1117/12.550551
(This paper is not open access)
Abstract: This paper describes the design of a grid-enabled version of Montage, an astronomical image mosaic service, suitable for large scale processing of the sky. All the re-projection jobs can be added to a pool of tasks and performed by as many processors as are available, exploiting the parallelization inherent in the Montage architecture. We show how we can describe the Montage application in terms of an abstract workflow so that a planning tool such as Pegasus can derive an executable workflow that can be run in the Grid environment. The execution of the workflow is performed by the workflow manager DAGMan and the associated Condor-G. The grid processing will support tiling of images to a manageable size when the input images can no longer be held in memory. Montage will ultimately run operationally on the Teragrid. We describe science applications of Montage, including its application to science product generation by Spitzer Legacy Program teams and large-scale, all-sky image processing projects.
D. S. Katz, A. Bergou, G. B. Berriman, G. L. Block, J. Collier, D. W. Curkendall, J. Good, L. Husman, J. C. Jacob, A. Laity, P. P. Li, C. Miller, T. Prince, H. Siegel, and R. Williams, "Accessing and Visualizing Scientific Spatiotemporal Data," Proceedings of the 16th International Conference on Scientific and Statistical Database Management, pp. 107-110, 2004. DOI: 10.1109/SSDM.2004.1311198
(This paper is open access in the US)
Abstract: This paper discusses work done by JPL's Parallel Applications Technologies Group in helping scientists access and visualize very large data sets through the use of multiple computing resources, such as parallel supercomputers, clusters, and grids. These tools do one or more of the following tasks: visualize local data sets for local users, visualize local data sets for remote users, and access and visualize remote data sets. The tools are used for various types of data, including remotely sensed image data, digital elevation models, astronomical surveys, etc. The paper attempts to pull some common elements out of these tools that may be useful for others who have to work with similarly large data sets.
J. C. Jacob, D. S. Katz, T. Prince, G. B. Berriman, J. C. Good, A. C. Laity, E. Deelman, G. Singh, and M.-H. Su, "The Montage Architecture for Grid-Enabled Science Processing of Large, Distributed Datasets," Proceedings of the 2004 Earth Science Technology Conference (ESTC-04), 2004.
(This paper is open access)
Abstract: Montage is an Earth Science Technology Office (ESTO) Computational Technologies (CT) Round III Grand Challenge project that will deploy a portable, compute-intensive, custom astronomical image mosaicking service for the National Virtual Observatory (NVO). Although Montage is developing a compute- and data-intensive service for the astronomy community, we are also helping to address a problem that spans both Earth and space science: how to efficiently access and process multi-terabyte, distributed datasets. In both communities, the datasets are massive, and are stored in distributed archives that are, in most cases, remote with respect to the available computational resources. Therefore, use of state-of-the-art computational grid technologies is a key element of the Montage portal architecture. This paper describes the aspects of the Montage design that are applicable to both the Earth and space science communities.
G. B. Berriman, J. C. Good, A. C. Laity A. Bergou, J. C. Jacob, D. S. Katz, E. Deelman, C. Kesselman, G. Singh, M.-H. Su, and R. Williams, "Montage: A Grid Enabled Image Mosaic Service for the National Virtual Observatory," Astronomical Data Analysis Software & Systems (ADASS) XIII, 2003.
(This paper is not open access)
Abstract: The architecture of Montage, which delivers custom science-grade astronomical images, was presented at ADASS XII. That architecture has been tested on 2MASS images computed on single processor Linux machines that hold all image data in memory. This year, we describe the design of a grid-enabled version of Montage, suitable for large scale processing of the sky. It exploits to the maximum the parallelization inherent in the Montage architecture, whereby image re-projections are performed in parallel. All the re-projection jobs can be added to a pool of tasks and performed by as many processors as are available. We show how we can describe the Montage application in terms of an abstract workflow so that a planning tool such as Pegasus can derive an executable workflow that can be run in the Grid environment. The execution of the workflow is performed by the workflow manager DAGMan and the associated Condor-G. The grid processing will support tiling of images to a manageable size when the input images can no longer be held in memory. When fully tested, Montage will ultimately run operationally on the Teragrid. We will present processing metrics and describe how Montage is being used, including its application to science product generation by SIRTF Legacy Program teams and large-scale image processing pro jects such as Atlasmaker (this conference).
E. Ciocca, I. Koren, Z. Koren, C. M. Krishna, and D. S. Katz, "Application-Level Fault Tolerance and Detection in the Orbital Thermal Imaging Spectrometer," Proceedings of the 2004 Pacific Rim International Symposium on Dependable Computing, pp. 43-48, 2004. DOI: 10.1109/PRDC.2004.1276551
(This paper is not open access)
Abstract: Systems that operate in extremely volatile environments, such as orbiting satellites, must be designed with a strong emphasis on fault tolerance. Rather than rely solely on the system hardware, it may be benecial to entrust some of the fault handling to software at the application level, which can utilize semantic information and software communication channels to achieve fault tolerance with considerably less power and performance overhead. This paper details the implementation and evaluation of such a software-level approach, Application-Level Fault Tolerance and Detection (ALFTD) into the Orbital Thermal Imaging Spectrometer (OTIS).
J. W. Larson, B. Norris, E. T. Ong, D. E. Bernholdt, J. B. Drake, W. R. El Wasif, M. W. Ham, C. E. Rasmussen, G. Kumfert, D. S. Katz, S. Zhou, C. DeLuca, and N. S. Collins, "Components, The Common Component Architecture, and the Climate/Ocean/Weather Community," Proceedings of the 20th International Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, 84th American Meteorological Society Annual Meeting, 2004.
(This paper is not open access)
Abstract: Earth system and environmental models present the scientist/programmer with multiple challenges in software design, development, and maintenance, overall system integration, and performance. We describe how work in the industrial sector of software engineering - namely component-based software engineering - can be brought to bear to address issues of software complexity. We explain how commercially developed component solutions are inadequate to address the performance needs of the Earth system modeling community. We describe a component-based approach called the Common Component Architecture that has as its goal the creation of a component paradigm that is compatible with the requirements of high-performance computing applications. We outline the relationship and ongoing collaboration between CCA and major Climate/Weather/Ocean community software projects. We present examples of work in progress that uses CCA, and discuss long-term plans for the CCA-climate/weather/ocean collaboration.
M. Turmon, R. Granat, D. S. Katz, and J. Z. Lou, "Tests and Tolerances for High-Performance Software-Implemented Fault Detection," IEEE Transactions on Computers, v.52(5), pp. 579-591, May 2003. DOI: 10.1109/TC.2003.1197125
(This paper is open access in the US)
Abstract: We describe and test a software approach to fault detection in common numerical algorithms. Such result checking or algorithm-based fault tolerance (ABFT) methods may be used, for example, to overcome single-event upsets in computational hardware or to detect errors in complex, high-efficiency implementations of the algorithms. Following earlier work, we use checksum methods to validate results returned by a numerical subroutine operating subject to unpredictable errors in data. We consider common matrix and Fourier algorithms which return results satisfying a necessary condition having a linear form: the checksum tests compliance with this condition. We discuss the theory and practice of setting numerical tolerances to separate errors caused by a fault from those inherent in finite-precision numerical calculations. We concentrate on comprehensively defining and evaluating tests having various accuracy/computation burden tradeoffs, and emphasize average-case algorithm behavior rather then using worst-case upper bounds on error.
D. S. Katz and R. R. Some, "NASA Advances Robotic Space Exploration," IEEE Computer, v. 36(1), pp. 52-61, January 2003. DOI: 10.1109/MC.2003.1160056
(This paper is open access in the US)
Abstract: NASA's successful exploration of space has uncovered vast amounts of new knowledge about the Earth, the solar system and its other planets, and the stellar spaces beyond. To continue gaining new knowledge has required - and will continue to require - new capabilities in onboard processing hardware, system software, and applications such as autonomy.
For example, initial robotic space exploration missions functioned, for the most part, as large flying cameras. These instruments have evolved over time to include more sophisticated imaging radar, multispectral imagers, spectrometers, gravity wave detectors, a host of prepositioned sensors and, most recently, rovers.
D. S. Katz, E. R. Tisdale, and C. D. Norton, "The Common Component Architecture (CCA) Applied to Sequential and Parallel Computational Electromagnetic Applications," Recent Advances in Computational Science & Engineering: Proceedings of the International Conference on Scientific & Engineering Computation (IC-SEC) 2002, pp. 353-356, Imperial College Press, 2002.
(This paper is open access in the US)
Abstract: The development of large-scale multi-disciplinary scientific applications for high-performance computers today involves managing the interaction between portions of the application developed by different groups. The CCA (Common Component Architecture) Forum is developing a component architecture specification to address high-performance scientific computing, emphasizing scalable (possibly-distributed) parallel computations. This paper presents an examination of the CCA software in sequential and parallel electromagnetics applications using unstructured adaptive mesh refinement (AMR). The CCA learning curve and the process for modifying Fortran 90 code (a driver routine and an AMR library) into two components are described. The performance of the original applications and the componentized versions are measured and shown to be comparable.
G. B. Berriman, D. Curkendall, J. Good, J. Jacob, D. S. Katz, M. Kong, S. Monkewitz, R. Moore, T. Prince, R. Williams, "An Architecture for Access to Compute Intensive Image Mosaic and Cross-Identification Services in the NVO," Proceedings of the SPIE Conference on Astronomical Telescopes and Instrumentation, SPIE, pp. 91-102, 2002.
(This paper is not open access)
Abstract: The National Virtual Observatory (NVO) will provide on-demand access to data collections, data fusion services and compute intensive applications. The paper describes the development of a framework that will support two key aspects of these objectives: a compute engine that will deliver custom image mosaics, and a "request management system," based on an e-business applications server, for job processing, including monitoring, failover and status reporting. We will develop this request management system to support a diverse range of astronomical requests, including services scaled to operate on the emerging computational grid infrastructure. Data requests will be made through existing portals to demonstrate the system: the NASA/IPAC Extragalactic Database (NED), the On-Line Archive Science Information Services (OASIS) at the NASA/IPAC Infrared Science Archive (IRSA); the Virtual Sky service at Caltechs Center for Advanced Computing Research (CACR), and the yourSky mosaic server at the Jet Propulsion Laboratory (JPL).
D. S. Katz, E. R. Tisdale, and C. D. Norton, "A Study of the Common Component Architecture (CCA) Forum Software," Fifth Annual Workshop on High-Performance Embedded Computing (HPEC), 2002.
(This paper is open access)