Doe Science news source
The DOE Science News Source is a Newswise initiative to promote research news from the Office of Science of the DOE to the public and news media.
  • 2017-08-24 13:05:11
  • Article ID: 680037

Big Bang - The Movie

  • Credit: Lindsey Bleem, Nan Li, and the HACC team/Argonne National Laboratory; Mike Gladders/University of Chicago

    A simulated sky image of galaxies produced by running Argonne-developed high-performance computing codes and then running a galaxy formation model. Argonne has collaborated with the University of Illinois, teaming up two supercomputers to perform simulation and data analysis of extremely large-scale, computationally intensive models of the universe.

  • Credit: Joe Insley and Silvio Rizzi/Argonne National Laboratory.

    Dark matter halo distribution from the Outer Rim simulation carried out on 32 racks of Mira with the Hardware/Hybrid Accelerated Cosmology Code (HACC). Shown is the full simulation volume with halos above a certain mass cut. The halos are colored by mass, the heavier halos are shown in red. These halos host the galaxies that we observe with large telescopes.

If you have ever had to wait those agonizing minutes in front of a computer for a movie or large file to load, you’ll likely sympathize with the plight of cosmologists at the U.S. Department of Energy’s (DOE) Argonne National Laboratory. But instead of watching TV dramas, they are trying to transfer, as fast and as accurately as possible, the huge amounts of data that make up movies of the universe – computationally demanding and highly intricate simulations of how our cosmos evolved after the Big Bang.

In a new approach to enable scientific breakthroughs, researchers linked together supercomputers at the Argonne Leadership Computing Facility (ALCF) and at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign (UI). This link enabled scientists to transfer massive amounts of data and to run two different types of demanding computations in a coordinated fashion – referred to technically as a workflow.

“We talk about building the ‘universe in the lab,’ and simulations are a huge component of that.” - Katrin Heitmann, Argonne cosmologist

What distinguishes the new work from typical workflows is the scale of the computation, the associated data generation and transfer and the scale and complexity of the final analysis. Researchers also tapped the unique capabilities of each supercomputer: They performed cosmological simulations on the ALCF’s Mira supercomputer, and then sent huge quantities of data to UI’s Blue Waters, which is better suited to perform the required data analysis tasks because of its processing power and memory balance.

For cosmology, observations of the sky and computational simulations go hand in hand, as each informs the other. Cosmological surveys are becoming ever more complex as telescopes reach deeper into space and time, mapping out the distributions of galaxies at farther and farther distances, at earlier epochs of the evolution of the universe.

The very nature of cosmology precludes carrying out controlled lab experiments, so scientists rely instead on simulations to provide a unique way to create a virtual cosmological laboratory. “The simulations that we run are a backbone for the different kinds of science that can be done experimentally, such as the large-scale experiments at different telescope facilities around the world,” said Argonne cosmologist Katrin Heitmann. “We talk about building the ‘universe in the lab,’ and simulations are a huge component of that.”

Not just any computer is up to the immense challenge of generating and dealing with datasets that can exceed many petabytes a day, according to Heitmann. “You really need high-performance supercomputers that are capable of not only capturing the dynamics of trillions of different particles, but also doing exhaustive analysis on the simulated data,” she said. “And sometimes, it’s advantageous to run the simulation and do the analysis on different machines.”

Typically, cosmological simulations can only output a fraction of the frames of the computational movie as it is running because of data storage restrictions. In this case, Argonne sent every data frame to NCSA as soon it was generated, allowing Heitmann and her team to greatly reduce the storage demands on the ALCF file system. “You want to keep as much data around as possible,” Heitmann said. “In order to do that, you need a whole computational ecosystem to come together: the fast data transfer, having a good place to ultimately store that data and being able to automate the whole process.”

In particular, Argonne transferred the data produced immediately to Blue Waters for analysis. The first challenge was to set up the transfer to sustain the bandwidth of one petabyte per day.

Once Blue Waters performed the first pass of data analysis, it reduced the raw data – with high fidelity – into a manageable size. At that point, researchers sent the data to a distributed repository at Argonne, the Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory and the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory. Cosmologists can access and further analyze the data through a system built by researchers in Argonne’s Mathematics and Computer Science Division in collaboration with Argonne’s High Energy Physics Division.

Argonne and University of Illinois built one such central repository on the Supercomputing ’16 conference exhibition floor in November 2016, with memory units supplied by DDN Storage. The data moved over 1,400 miles to the conference’s SciNet network. The link between the computers used high-speed networking through the Department of Energy’s Energy Science Network (ESnet). Researchers sought, in part, to take full advantage of the fast SciNET infrastructure to do real science; typically it is used for demonstrations of technology rather than solving real scientific problems.

“External data movement at high speeds significantly impacts a supercomputer’s performance,” said Brandon George, systems engineer at DDN Storage. “Our solution addresses that issue by building a self-contained data transfer node with its own high-performance storage that takes in a supercomputer’s results and the responsibility for subsequent data transfers of said results, leaving supercomputer resources free to do their work more efficiently.”

The full experiment ran successfully for 24 hours without interruption and led to a valuable new cosmological data set that Heitmann and other researchers started to analyze on the SC16 show floor.

Argonne senior computer scientist Franck Cappello, who led the effort, likened the software workflow that the team developed to accomplish these goals to an orchestra. In this “orchestra,” Cappello said, the software connects individual sections, or computational resources, to make a richer, more complex sound.

He added that his collaborators hope to improve the performance of the software to make the production and analysis of extreme-scale scientific data more accessible. “The SWIFT workflow environment and the Globus file transfer service were critical technologies to provide the effective and reliable orchestration and the communication performance that were required by the experiment,” Cappello said.

“The idea is to have data centers like we have for the commercial cloud. They will hold scientific data and will allow many more people to access and analyze this data, and develop a better understanding of what they’re investigating,” said Cappello, who also holds an affiliate position at NCSA and serves as director of the international Joint Laboratory on Extreme Scale Computing, based in Illinois. “In this case, the focus was cosmology and the universe. But this approach can aid scientists in other fields in reaching their data just as well.”

Argonne computer scientist Rajkumar Kettimuthu and David Wheeler, lead network engineer at NCSA, were instrumental in establishing the configuration that actually reached this performance. Maxine Brown from University of Illinois provided the Sage environment to display the analysis result at extreme resolution. Justin Wozniak from Argonne developed the whole workflow environment using SWIFT to orchestrate and perform all operations.

The Argonne Leadership Computing Facility, the Oak Ridge Leadership Computing Facility, the Energy Science Network and the National Energy Research Scientific Computing Center are DOE Office of Science User Facilities. Blue Waters is the largest leadership-class supercomputer funded by the National Science Foundation. Part of this work was funded by DOE’s Office of Science.

The National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign provides supercomputing and advanced digital resources for the nation’s science enterprise. At NCSA, University of Illinois faculty, staff, students, and collaborators from around the globe use advanced digital resources to address research grand challenges for the benefit of science and society. NCSA has been advancing one third of the Fortune 50 for more than 30 years by bringing industry, researchers, and students together to solve grand challenges at rapid speed and scale.

The National Science Foundation (NSF) is an independent federal agency that supports fundamental research and education across all fields of science and engineering. In fiscal year (FY) 2016, its budget is $7.5 billion. NSF funds reach all 50 states through grants to nearly 2,000 colleges, universities and other institutions. Each year, NSF receives more than 48,000 competitive proposals for funding and makes about 12,000 new funding awards. NSF also awards about $626 million in professional and service contracts yearly.

Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation's first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America's scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the U.S. Department of Energy's Office of Science.

The U.S. Department of Energy's Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit the Office of Science website.

  • Filters

  • × Clear Filters

High-Speed Movie Aids Scientists Who Design Glowing Molecules

In a recent experiment conducted at the Department of Energy's SLAC National Accelerator Laboratory, a research team used bright, ultrafast X-ray pulses from SLAC's X-ray free-electron laser to create a high-speed movie of a fluorescent protein in action. With that information, the scientists began to design a marker that switches more easily, a quality that can improve resolution during biological imaging.

Biomass-Produced Electricity in the US Possible, but It'll Cost

If the U.S. wants to start using wood pellets to produce energy, either the government or power customers will have to pay an extra cost, a new University of Georgia study has found.

Scientists Make Atoms-Thick Post-It Notes for Solar Cells and Circuits

In a study published Sept. 20 in Nature, UChicago and Cornell University researchers describe an innovative method to make stacks of semiconductors just a few atoms thick. The technique offers scientists and engineers a simple, cost-effective method to make thin, uniform layers of these materials, which could expand capabilities for devices from solar cells to cell phones.

Titan Helps Researchers Suck Mystery Out of Cell's 'Vacuum Cleaners'

In cancer cells, a membrane transport protein called P-glycoprotein, or Pgp, actively pumps anticancer drugs out of the cell, contributing to multidrug resistance. Recently, a team led by computational biophysicist Emad Tajkhorshid from the University of Illinois at Urbana-Champaign (UIUC) used the Titan supercomputer to uncover new details about Pgp that could help the drug discovery community manipulate Pgp function.

Laser-Free Method of Ion Cooling Has Range of Potential Uses

Prof. Daniel Zajfman's universal ion trap cools to a tenth of a degree above absolute zero. The new method does not depend on the type or the weight of the ion and, thus, might be used to investigate the properties of large biological molecules or nanoparticles, among other things.

Tiny Lasers from a Gallery of Whispers

Whispering gallery mode resonators rely on a phenomenon similar to an effect observed in circular galleries, and the same phenomenon applies to light. When light is stored in ring-shaped or spherical active resonators, the waves superimpose in such a way that it can result in laser light. This week in APL Photonics, investigators report a new type of dye-doped WGM micro-laser that produces light with tunable wavelengths.

Copper Catalyst Yields High Efficiency CO2-to-Fuels Conversion

Berkeley Lab scientists have developed a new electrocatalyst that can directly convert carbon dioxide into multicarbon fuels and alcohols using record-low inputs of energy. The work is the latest in a round of studies coming out of Berkeley Lab tackling the challenge of a creating a clean chemical manufacturing system that can put carbon dioxide to good use.

Solar-to-Fuel System Recycles CO2 to Make Ethanol and Ethylene

Berkeley Lab scientists have harnessed the power of photosynthesis to convert carbon dioxide into fuels and alcohols at efficiencies far greater than plants. The achievement marks a significant advance in the effort to move toward sustainable sources of fuel.

New Evidence for Small, Short-Lived Drops of Early Universe Quark-Gluon Plasma?

UPTON, NY--Particles emerging from even the lowest energy collisions of small deuterons with large heavy nuclei at the Relativistic Heavy Ion Collider (RHIC)--a U.S. Department of Energy Office of Science User Facility for nuclear physics research at DOE's Brookhaven National Laboratory--exhibit behavior scientists associate with the formation of a soup of quarks and gluons, the fundamental building blocks of nearly all visible matter.

New Insights Into Nanocrystal Growth in Liquid

PNNL researchers have measured the forces that cause certain crystals to assemble, revealing competing factors that researchers might be able to control. The work has a variety of implications in both discovery and applied science. In addition to providing insights into the formation of minerals and semiconductor nanomaterials, it might also help scientists understand soil as it expands and contracts through wetting and drying cycles.

  • Filters

  • × Clear Filters

PPPL Physicist Francesca Poli Named ITER Scientist Fellow

Article describes new ITER Scientist Fellow.

Los Alamos Gains Role in High-Performance Computing for Materials Program

A new high-performance computing initiative announced this week by the U.S. Department of Energy will help U.S. industry accelerate the development of new or improved materials for use in severe environments.

UK Commits $88 Million to LBNF/DUNE in First-Ever Umbrella Science Agreement with U.S.

The UK has committed $88 million to the Long-Baseline Neutrino Facility and Deep Underground Neutrino Experiment as part of an umbrella science and technology agreement with the United States.

Wayne State Receives $1.2 Million NSF Grant to Develop Autonomous Battery Operating System

Researchers at Wayne State University led by Nathan Fisher, associate professor of computer science in the College of Engineering, received a $1.2 million grant from the National Science Foundation to address the need for effective, integrative battery operating systems that provide sustained and reliable power.

UAH leads effort that secures $20 million grant from the National Science Foundation

A partnership comprising nine universities in Alabama, including The University of Alabama in Huntsville (UAH) as the lead institution, has been awarded a $20 million, five-year grant by the National Science Foundation's Experimental Program to Stimulate Competitive Research (EPSCoR).

Sandia Labs Wins 5 Regional Technology Transfer Awards

Sandia National Laboratories won five awards from the 2017 Federal Laboratory Consortium for its work to develop and commercialize innovative technologies.

Tulane Receives Grant to Reduce Auto Emissions

Members of Tulane University's Shantz Lab will work with industrial scientists to assist in the development of next-generation materials designed to reduce harmful automotive emissions. The three-year old lab and its group of students have received a grant and equipment resources from SACHEM, Inc., a chemical science company.

Lab Leads New Effort in Materials Development

Lawrence Livermore National Lab will be part of a multi-lab effort to apply high-performance computing to US-based industry's discovery, design, and development of materials for severe environments under a new initiative announced by the Department of Energy (DOE) on Sept. 19.

ORNL Innovation Crossroads Program Opens Second Round of Energy Entrepreneurial Fellowships

Entrepreneurs are invited to apply for the second round of Oak Ridge National Laboratory's Innovation Crossroads program.

Los Alamos Recognized as Top Diversity Employer

For the second straight year, Los Alamos National Laboratory was recognized as a top diversity employer by LATINA Style and STEM Workforce Diversity magazine.

  • Filters

  • × Clear Filters

Fungi: Gene Activator Role Discovered

Specific modifications to fungi DNA may hold the secret to turning common plant degradation agents into biofuel producers.

First Look at a Living Cell Membrane

Neutrons provide the solution to nanoscale examination of living cell membrane and confirm the existence of lipid rafts.

High Yield Biomass Conversion Strategy Ready for Commercialization

Researchers convert 80 percent of biomass into high-value products with strategy that's ready for commercialization.

Consequences of Drought Stress on Biofuels

Switchgrass cultivated during a year of severe drought inhibited microbial fermentation and resulting biofuel production.

Clay Minerals and Metal Oxides Change How Uranium Travels Through Sediments

Montmorillonite clays prevent uranium from precipitating from liquids, letting it travel with groundwater.

Tundra Loses Carbon with Rapid Permafrost Thaw

Seven-year-study shows plant growth does not sustainably balance carbon losses from solar warming and permafrost thaw.

Crystals Grow by Twisting, Aligning and Snapping Together

Van der Waals force, which that enables tiny crystals to grow, could be used to design new materials.

Vitamin B12 Fuels Microbial Growth

Scarce compound, vitamin B12, is key for cellular metabolism and may help shape microbial communities that affect environmental cycles and bioenergy production.

Carbon in Floodplain Unlikely to Cycle into the Atmosphere

Microbes leave a large fraction of carbon in anoxic sediments untouched, a key finding for understanding how watersheds influence Earth's ecosystem.

Bacterial Cell Wall Changes Produce More Fatty Molecules

New strategy greatly increases the production and secretion of biofuel building block lipids in bacteria able to grow at industrial scales.


Thursday September 21, 2017, 03:05 PM

From Science to Finance: SLAC Summer Interns Forge New Paths in STEM

SLAC National Accelerator Laboratory

Thursday September 07, 2017, 02:05 PM

Students Discuss 'Cosmic Opportunities' at 45th Annual SLAC Summer Institute

SLAC National Accelerator Laboratory

Thursday August 31, 2017, 05:05 PM

Binghamton University Opens $70 Million Smart Energy Building

Binghamton University, State University of New York

Wednesday August 23, 2017, 05:05 PM

Widening Horizons for High Schoolers with Code

Argonne National Laboratory

Saturday May 20, 2017, 12:05 PM

Rensselaer Polytechnic Institute Graduates Urged to Embrace Change at 211th Commencement

Rensselaer Polytechnic Institute (RPI)

Monday May 15, 2017, 01:05 PM

ORNL, University of Tennessee Launch New Doctoral Program in Data Science

Oak Ridge National Laboratory

Friday April 07, 2017, 11:05 AM

Champions in Science: Profile of Jonathan Kirzner

Department of Energy, Office of Science

Wednesday April 05, 2017, 12:05 PM

High-Schooler Solves College-Level Security Puzzle From Argonne, Sparks Interest in Career

Argonne National Laboratory

Tuesday March 28, 2017, 12:05 PM

Champions in Science: Profile of Jenica Jacobi

Department of Energy, Office of Science

Friday March 24, 2017, 10:40 AM

Great Neck South High School Wins Regional Science Bowl at Brookhaven Lab

Brookhaven National Laboratory

Wednesday February 15, 2017, 04:05 PM

Middle Schoolers Test Their Knowledge at Science Bowl Competition

Argonne National Laboratory

Friday January 27, 2017, 04:00 PM

Haslam Visits ORNL to Highlight State's Role in Discovering Tennessine

Oak Ridge National Laboratory

Tuesday November 08, 2016, 12:05 PM

Internship Program Helps Foster Development of Future Nuclear Scientists

Oak Ridge National Laboratory

Friday May 13, 2016, 04:05 PM

More Than 12,000 Explore Jefferson Lab During April 30 Open House

Thomas Jefferson National Accelerator Facility

Monday April 25, 2016, 05:05 PM

Giving Back to National Science Bowl

Ames Laboratory

Friday March 25, 2016, 12:05 PM

NMSU Undergrad Tackles 3D Particle Scattering Animations After Receiving JSA Research Assistantship

Thomas Jefferson National Accelerator Facility

Tuesday February 02, 2016, 10:05 AM

Shannon Greco: A Self-Described "STEM Education Zealot"

Princeton Plasma Physics Laboratory

Monday November 16, 2015, 04:05 PM

Rare Earths for Life: An 85th Birthday Visit with Mr. Rare Earth

Ames Laboratory

Tuesday October 20, 2015, 01:05 PM

Meet Robert Palomino: 'Give Everything a Shot!'

Brookhaven National Laboratory

Tuesday April 22, 2014, 11:30 AM

University of Utah Makes Solar Accessible

University of Utah

Wednesday March 06, 2013, 03:40 PM

Student Innovator at Rensselaer Polytechnic Institute Seeks Brighter, Smarter, and More Efficient LEDs

Rensselaer Polytechnic Institute (RPI)

Friday November 16, 2012, 10:00 AM

Texas Tech Energy Commerce Students, Community Light up Tent City

Texas Tech University

Wednesday November 23, 2011, 10:45 AM

Don't Get 'Frosted' Over Heating Your Home This Winter

Temple University

Wednesday July 06, 2011, 06:00 PM

New Research Center To Tackle Critical Challenges Related to Aircraft Design, Wind Energy, Smart Buildings

Rensselaer Polytechnic Institute (RPI)

Friday April 22, 2011, 09:00 AM

First Polymer Solar-Thermal Device Heats Home, Saves Money

Wake Forest University

Friday April 15, 2011, 12:25 PM

Like Superman, American University Will Get Its Energy from the Sun

American University

Thursday February 10, 2011, 05:00 PM

ARRA Grant to Help Fund Seminary Building Green Roof

University of Chicago

Tuesday December 07, 2010, 05:00 PM

UC San Diego Installing 2.8 Megawatt Fuel Cell to Anchor Energy Innovation Park

University of California San Diego

Monday November 01, 2010, 12:50 PM

Rensselaer Smart Lighting Engineering Research Center Announces First Deployment of New Technology on Campus

Rensselaer Polytechnic Institute (RPI)

Friday September 10, 2010, 12:40 PM

Ithaca College Will Host Regional Clean Energy Summit

Ithaca College

Tuesday July 27, 2010, 10:30 AM

Texas Governor Announces $8.4 Million Award to Create Renewable Energy Institute

Texas Tech University

Friday May 07, 2010, 04:20 PM

Creighton University to Offer New Alternative Energy Program

Creighton University

Wednesday May 05, 2010, 09:30 AM

National Engineering Program Seeks Subject Matter Experts in Energy

JETS Junior Engineering Technical Society

Wednesday April 21, 2010, 12:30 PM

Students Using Solar Power To Create Sustainable Solutions for Haiti, Peru

Rensselaer Polytechnic Institute (RPI)

Wednesday March 03, 2010, 07:00 PM

Helping Hydrogen: Student Inventor Tackles Challenge of Hydrogen Storage

Rensselaer Polytechnic Institute (RPI)

Thursday February 04, 2010, 02:00 PM

Turning Exercise into Electricity

Furman University

Thursday November 12, 2009, 12:45 PM

Campus Leaders Showing the Way to a Sustainable, Clean Energy Future

National Wildlife Federation (NWF)

Tuesday November 03, 2009, 04:20 PM

Furman University Receives $2.5 Million DOE Grant for Geothermal Project

Furman University

Thursday September 17, 2009, 02:45 PM

Could Sorghum Become a Significant Alternative Fuel Source?

Salisbury University

Wednesday September 16, 2009, 11:15 AM

Students Navigating the Hudson River With Hydrogen Fuel Cells

Rensselaer Polytechnic Institute (RPI)

Wednesday September 16, 2009, 10:00 AM

College Presidents Flock to D.C., Urge Senate to Pass Clean Energy Bill

National Wildlife Federation (NWF)

Wednesday July 01, 2009, 04:15 PM

Northeastern Announces New Professional Master's in Energy Systems

Northeastern University

Friday October 12, 2007, 09:35 AM

Kansas Rural Schools To Receive Wind Turbines

Kansas State University

Thursday August 17, 2006, 05:30 PM

High Gas Prices Here to Stay, Says Engineering Professor

Rowan University

Wednesday May 17, 2006, 06:45 PM

Time Use Expert's 7-Year Fight for Better Gas Mileage

University of Maryland, College Park

Showing results

0-4 Of 2215