- Argonne National Laboratory has completed the largest cosmic simulation to date, covering 28.6 billion light-years.
- The simulation utilized 8,100 computing nodes and approximately 50,000 GPUs, running for about four days.
- The research focuses on understanding cosmic expansion and its effects on galaxy formation.
- These simulations help extract more reliable cosmic measurement results.
Researchers at Argonne National Laboratory have completed two of the largest cosmological simulations in history, modeling 13.4 trillion dark matter particles across a cosmic region of 28.6 billion light-years. These simulations ran on Argonne's supercomputer Aurora, utilizing 8,100 computing nodes and approximately 50,000 graphics processing units (GPUs). The team completed both runs and most of the data analysis in about four days, creating a massive dataset aimed at helping scientists study the early moments of the universe and explain observations from major astronomical surveys.
The Challenge of Understanding Cosmic Expansion
The project, led by Argonne computational scientist Nicholas Frontiere, focuses on understanding cosmic expansion, a rapid expansion period shortly after the Big Bang. Although expansion is a core aspect of modern cosmology, the physics driving it remains uncertain.
According to the researchers, expansion stretches tiny quantum fluctuations across vast distances in the early universe. Over billions of years, gravity amplifies these small density variations, helping to form the large-scale cosmic web observed today, including galaxies and dark matter. Dark matter cannot be directly observed through ordinary light, but its gravitational effects help shape the distribution of galaxies. These simulations use particles to track the density and velocity of dark matter, allowing scientists to model the evolution of cosmic structures. The team is particularly interested in primordial non-Gaussianity: subtle deviations from the simple statistical distributions predicted by the standard inflation model.
Detecting these deviations could help differentiate between various inflationary explanations and reveal more information about the universe's initial conditions. However, any potential signals are expected to be weak and may be obscured by cosmic variance, which is the statistical uncertainty arising when observations cover only a limited portion of the universe. Therefore, researchers need to combine the vast amount of data with simulations of sufficient resolution to capture the structure of dark matter halos and galaxy formation.
The Importance of Aurora's Computational Power
Aurora's computational power is crucial in this regard. The machine features 10,624 nodes and 63,744 GPUs, equipped with high-capacity memory and a storage system designed for handling large data flows. These simulations generated over 85 PB of data through Aurora's distributed asynchronous object storage system, primarily for checkpointing. These checkpoints allow computations to resume after interruptions rather than restarting from scratch. The team retained about 8 PB of data for scientific analysis. They utilized Argonne's Hardware/Hybrid Accelerated Cosmology Code (HACC), which is designed to run large-scale cosmological simulations across different supercomputing architectures.
Unlike earlier projects conducted on the Frontier system at Oak Ridge National Laboratory, which simulated gravity along with gas, stars, and other processes, Aurora's simulations focused solely on gravity itself. This approach enabled researchers to cover a larger volume of the universe with more particles.
The two simulations used the same cosmological model and non-Gaussian signals, but their initial phases were reversed. Analyzing them together helps offset some random variations and isolate potential physical phenomena. The signal strength is also concealed, allowing researchers to test analytical methods without knowing the expected answers. The synthetic observations produced will help scientists validate workflows and identify potential biases, which can then be applied to real data from projects like the Dark Energy Spectroscopic Instrument (DESI) and NASA's SPHEREx mission, both supported by the Department of Energy.
By generating realistic simulated observations in the sky, these simulations may assist researchers in extracting more reliable measurement results from these surveys. Their scale also provides scientists with a powerful testing platform to examine subtle features of the early universe.
Simulations Bridging the Gap Between Theory and Observation
The project received computational time through the Department of Energy's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program. As DESI, SPHEREx, and other surveys collect increasingly detailed observational data, simulations of this scale help bridge the gap between theoretical models and the universe as observed by astronomers.
Project Specifications Computing Nodes 10,624 Number of GPUs 63,744 Data Generated Over 85 PB Data Retained Approximately 8 PB
The Contribution of Large Simulations to Cosmological Research
This large-scale cosmic simulation not only showcases Argonne National Laboratory's breakthroughs in computational capability but also highlights the complexities of understanding cosmic expansion. By simulating the behavior of dark matter, scientists can delve deeper into the conditions of the early universe and potentially uncover key factors influencing galaxy formation. As observational data increases, these simulations will become essential tools for linking theory with actual observations, further advancing the field of cosmology.