| 1. |
- Householder, John Ethan, et al.
(author)
-
One sixth of Amazonian tree diversity is dependent on river floodplains
- 2024
-
In: NATURE ECOLOGY & EVOLUTION. - 2397-334X.
-
Journal article (peer-reviewed)abstract
- Amazonia's floodplain system is the largest and most biodiverse on Earth. Although forests are crucial to the ecological integrity of floodplains, our understanding of their species composition and how this may differ from surrounding forest types is still far too limited, particularly as changing inundation regimes begin to reshape floodplain tree communities and the critical ecosystem functions they underpin. Here we address this gap by taking a spatially explicit look at Amazonia-wide patterns of tree-species turnover and ecological specialization of the region's floodplain forests. We show that the majority of Amazonian tree species can inhabit floodplains, and about a sixth of Amazonian tree diversity is ecologically specialized on floodplains. The degree of specialization in floodplain communities is driven by regional flood patterns, with the most compositionally differentiated floodplain forests located centrally within the fluvial network and contingent on the most extraordinary flood magnitudes regionally. Our results provide a spatially explicit view of ecological specialization of floodplain forest communities and expose the need for whole-basin hydrological integrity to protect the Amazon's tree diversity and its function.
|
|
| 2. |
- ter Steege, Hans, et al.
(author)
-
Mapping density, diversity and species-richness of the Amazon tree flora
- 2023
-
In: COMMUNICATIONS BIOLOGY. - 2399-3642. ; 6:1
-
Journal article (peer-reviewed)abstract
- Using 2.046 botanically-inventoried tree plots across the largest tropical forest on Earth, we mapped tree species-diversity and tree species-richness at 0.1-degree resolution, and investigated drivers for diversity and richness. Using only location, stratified by forest type, as predictor, our spatial model, to the best of our knowledge, provides the most accurate map of tree diversity in Amazonia to date, explaining approximately 70% of the tree diversity and species-richness. Large soil-forest combinations determine a significant percentage of the variation in tree species-richness and tree alpha-diversity in Amazonian forest-plots. We suggest that the size and fragmentation of these systems drive their large-scale diversity patterns and hence local diversity. A model not using location but cumulative water deficit, tree density, and temperature seasonality explains 47% of the tree species-richness in the terra-firme forest in Amazonia. Over large areas across Amazonia, residuals of this relationship are small and poorly spatially structured, suggesting that much of the residual variation may be local. The Guyana Shield area has consistently negative residuals, showing that this area has lower tree species-richness than expected by our models. We provide extensive plot meta-data, including tree density, tree alpha-diversity and tree species-richness results and gridded maps at 0.1-degree resolution. A study mapping the tree species richness in Amazonian forests shows that soil type exerts a strong effect on species richness, probably caused by the areas of these forest types. Cumulative water deficit, tree density and temperature seasonality affect species richness at a regional scale.
|
|
| 3. |
- Beal, Jacob, et al.
(author)
-
Robust estimation of bacterial cell count from optical density
- 2020
-
In: Communications Biology. - : Springer Science and Business Media LLC. - 2399-3642. ; 3:1
-
Journal article (peer-reviewed)abstract
- Optical density (OD) is widely used to estimate the density of cells in liquid culture, but cannot be compared between instruments without a standardized calibration protocol and is challenging to relate to actual cell count. We address this with an interlaboratory study comparing three simple, low-cost, and highly accessible OD calibration protocols across 244 laboratories, applied to eight strains of constitutive GFP-expressing E. coli. Based on our results, we recommend calibrating OD to estimated cell count using serial dilution of silica microspheres, which produces highly precise calibration (95.5% of residuals <1.2-fold), is easily assessed for quality control, also assesses instrument effective linear range, and can be combined with fluorescence calibration to obtain units of Molecules of Equivalent Fluorescein (MEFL) per cell, allowing direct comparison and data fusion with flow cytometry measurements: in our study, fluorescence per cell measurements showed only a 1.07-fold mean difference between plate reader and flow cytometry data.
|
|
| 4. |
- Gonzalo P., Rodrigo, 1980-, et al.
(author)
-
Enabling workflow-aware scheduling on HPC systems
- 2017
-
In: HPDC '17. - : ACM Digital Library. - 9781450346993 ; , s. 3-14
-
Conference paper (other academic/artistic)abstract
- Scientific workflows are increasingly common in the workloads of current High Performance Computing (HPC) systems. However, HPC schedulers do not incorporate workflow-specific mechanisms beyond the capacity to declare dependencies between their jobs. Thus, workflows are run as sets of batch jobs with dependencies, which induces long intermediate wait times and, consequently, long workflow turnaround times. Alternatively, to reduce their turnaround time, workflows may be submitted as single pilot jobs that are allocated their maximum required resources for their entire runtime. Pilot jobs achieve shorter turnaround times but reduce the HPC system's utilization because resources may idle during the workflow's execution. We present a workflow-aware scheduling (WoAS) system that enables existing scheduling algorithms to exploit fine-grained information on a workflow's resource requirements and structure without modification. The current implementation of WoAS is integrated into Slurm, a widely used HPC batch scheduler. We evaluate the system using a simulator using real and synthetic workflows and a synthetic baseline workload that captures job patterns observed over three years of workload data from Edison, a large supercomputer hosted at the National Energy Research Scientific Computing Center. Our results show that WoAS reduces workflow turnaround times and improves system utilization without significantly slowing down conventional jobs.
|
|
| 5. |
- Rodrigo, Gonzalo P., 1980-, et al.
(author)
-
A2L2 : an application aware flexible HPC scheduling model for low-latency allocation
- 2015
-
In: VTDC '15. - New York, NY, USA : ACM Digital Library. - 9781450335737 ; , s. 11-19
-
Conference paper (peer-reviewed)abstract
- High-performance computing (HPC) is focused on providing large-scale compute capacity to scientific applications. HPC schedulers tend to be optimized for large parallel batch jobs and, as such, often overlook the requirements of other scientific applications. In this work, we propose a cloud-inspired HPC scheduling model that aims to capture application performance and requirement models (Application Aware - A2) and dynamically resize malleable application resource allocations to be able to support applications with critical performance or deadline requirements. (Low Latency allocation - L2). The proposed model incorporates measures to improve data-intensive applications performance on HPC systems and is derived from a set of cloud scheduling techniques that are identified as applicable in HPC environments. The model places special focus on dynamically malleable applications; data-intensive applications that support dynamic resource allocation without incurring severe performance penalties; which are proposed for fine-grained back-filling and dynamic resource allocation control without job preemption.
|
|
| 6. |
- Fox, William, et al.
(author)
-
E-HPC : A Library for Elastic Resource Management in HPC Environments
- 2017
-
In: 12th Workshop on Workflows in Support of Large-Scale Science (WORKS). - New York, NY, USA : Association for Computing Machinery (ACM). - 9781450351294
-
Conference paper (peer-reviewed)abstract
- Next-generation data-intensive scientific workflows need to support streaming and real-time applications with dynamic resource needs on high performance computing (HPC) platforms. The static resource allocation model on current HPC systems that was designed for monolithic MPI applications is insufficient to support the elastic resource needs of current and future workflows. In this paper, we discuss the design, implementation and evaluation of Elastic-HPC (E-HPC), an elastic framework for managing resources for scientific workflows on current HPC systems. E-HPC considers a resource slot for a workflow as an elastic window that might map to different physical resources over the duration of a workflow. Our framework uses checkpoint-restart as the underlying mechanism to migrate workflow execution across the dynamic window of resources. E-HPC provides the foundation necessary to enable dynamic resource allocation of HPC resources that are needed for streaming and real-time workflows. E-HPC has negligible overhead beyond the cost of checkpointing. Additionally, E-HPC results in decreased turnaround time of workflows compared to traditional model of resource allocation for workflows, where resources are allocated per stage of the workflow. Our evaluation shows that E-HPC improves core hour utilization for common workflow resource use patterns and provides an effective framework for elastic expansion of resources for applications with dynamic resource needs.
|
|
| 7. |
- Gonzalo P., Rodrigo, 1980-
(author)
-
HPC scheduling in a brave new world
- 2017
-
Doctoral thesis (other academic/artistic)abstract
- Many breakthroughs in scientific and industrial research are supported by simulations and calculations performed on high performance computing (HPC) systems. These systems typically consist of uniform, largely parallel compute resources and high bandwidth concurrent file systems interconnected by low latency synchronous networks. HPC systems are managed by batch schedulers that order the execution of application jobs to maximize utilization while steering turnaround time. In the past, demands for greater capacity were met by building more powerful systems with more compute nodes, greater transistor densities, and higher processor operating frequencies. Unfortunately, the scope for further increases in processor frequency is restricted by the limitations of semiconductor technology. Instead, parallelism within processors and in numbers of compute nodes is increasing, while the capacity of single processing units remains unchanged. In addition, HPC systems’ memory and I/O hierarchies are becoming deeper and more complex to keep up with the systems’ processing power. HPC applications are also changing: the need to analyze large data sets and simulation results is increasing the importance of data processing and data-intensive applications. Moreover, composition of applications through workflows within HPC centers is becoming increasingly important. This thesis addresses the HPC scheduling challenges created by such new systems and applications. It begins with a detailed analysis of the evolution of the workloads of three reference HPC systems at the National Energy Research Supercomputing Center (NERSC), with a focus on job heterogeneity and scheduler performance. This is followed by an analysis and improvement of a fairshare prioritization mechanism for HPC schedulers. The thesis then surveys the current state of the art and expected near-future developments in HPC hardware and applications, and identifies unaddressed scheduling challenges that they will introduce. These challenges include application diversity and issues with workflow scheduling or the scheduling of I/O resources to support applications. Next, a cloud-inspired HPC scheduling model is presented that can accommodate application diversity, takes advantage of malleable applications, and enables short wait times for applications. Finally, to support ongoing scheduling research, an open source scheduling simulation framework is proposed that allows new scheduling algorithms to be implemented and evaluated in a production scheduler using workloads modeled on those of a real system. The thesis concludes with the presentation of a workflow scheduling algorithm to minimize workflows’ turnaround time without over-allocating resources.
|
|
| 8. |
- Gonzalo P., Rodrigo, 1980-, et al.
(author)
-
ScSF : a scheduling simulation framework
- 2018
-
In: Job Scheduling Strategies for Parallel Processing. - Cham : Springer. - 9783319773971 - 9783319773988 ; , s. 152-173
-
Conference paper (peer-reviewed)abstract
- High-throughput and data-intensive applications are increasingly present, often composed as workflows, in the workloads of current HPC systems. At the same time, trends for future HPC systems point towards more heterogeneous systems with deeper I/O and memory hierarchies. However, current HPC schedulers are designed to support classical large tightly coupled parallel jobs over homogeneous systems. Therefore, There is an urgent need to investigate new scheduling algorithms that can manage the future workloads on HPC systems. However, there is a lack of appropriate models and frameworks to enable development, testing, and validation of new scheduling ideas.In this paper, we present an open-source scheduler simulation framework (ScSF) that covers all the steps of scheduling research through simulation. ScSF provides capabilities for workload modeling, workload generation, system simulation, comparative workload analysis, and experiment orchestration. The simulator is designed to be run over a distributed computing infrastructure enabling to test at scale. We describe in detail a use case of ScSF to develop new techniques to manage scientific workflows in a batch scheduler. In the use case, such technique was implemented in the framework scheduler. For evaluation purposes, 1728 experiments, equivalent to 33 years of simulated time, were run in a deployment of ScSF over a distributed infrastructure of 17 compute nodes during two months. Finally, the experimental results were analyzed in the framework to judge that the technique minimizes workflows’ turnaround time without over-allocating resources. Finally, we discuss lessons learned from our experiences that will help future researchers.
|
|
| 9. |
|
|
| 10. |
- Rodrigo, Gonzalo P, 1980-, et al.
(author)
-
HPC System Lifetime Story : Workload Characterization and Evolutionary Analyses on NERSC Systems
- 2015
-
In: Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing (HDPC). - New York, NY, USA : ACM Digital Library. - 9781450335508 ; , s. 57-60
-
Conference paper (peer-reviewed)abstract
- High performance computing centers have traditionally served monolithic MPI applications. However, in recent years, many of the large scientific computations have included high throughput and data-intensive jobs. HPC systems have mostly used batch queue schedulers to schedule these workloads on appropriate resources. There is a need to understand future scheduling scenarios that can support the diverse scientific workloads in HPC centers. In this paper, we analyze the workloads on two systems (Hopper and Carver) at the National Energy Research Scientific Computing (NERSC) Center. Specifically, we present a trend analysis towards understanding the evolution of the workload over the lifetime of the two systems.
|
|
