Doe Science news source
The DOE Science News Source is a Newswise initiative to promote research news from the Office of Science of the DOE to the public and news media.
  • 2017-09-19 15:35:36
  • Article ID: 681413

A TOAST for Next Generation CMB Experiments

Berkeley Lab Cosmology Software Scales Up to 658,784 Knights Landing Cores

  • Credit: Julian Borrill, Berkeley Lab

    Cumulative daily maps of the sky temperature and polarization at each frequency showing how the atmosphere and noise integrate down over time. The year-long campaign spanned 129 observation-days during which the ACTpol SS patch was available for a 13-hour constant elevation scan. To make these maps, the signal, noise, and atmosphere observations were combined (including percent level detector calibration error), filtered with a 3rd order polynomial, and binned into pixels.

  • Credit: Roy Kaltschmidt, Berkeley Lab

    Cori supercomputer at NERSC

  • Credit: Julian Borrill, Berkeley Lab

    The telescope's view through one realization of turbulent, wind-blown, atmospheric water vapor. The volume of atmosphere being simulated depended on (a) the scan width and duration and (b) the wind speed and direction, both of which changed every 20 minutes. The entire observation used about 5000 such realizations.

The Cosmic Microwave Background (CMB) is the oldest light ever observed and is a wellspring of information about our cosmic past. This ancient light began its journey across space when the universe was just 380,000 years old. Today it fills the cosmos with microwaves. By parsing its subtle features with telescopes and supercomputers, cosmologists have gained insights about both the properties of our Universe and of fundamental physics. 

Despite all that we’ve learned from the CMB so far, there is still much about the universe that remains a mystery. Next-generation experiments like CMB Stage-4 (CMB-S4) will probe this primordial light at even higher sensitivity to learn more about the evolution of space and time and the nature of matter. But before this can happen scientists need to ensure that their data analysis infrastructure will be able to handle the information deluge.

That’s where researchers in Lawrence Berkeley National Laboratory’s (Berkeley Lab’s) Computational Cosmology Center (C3) come in. They recently achieved a critical milestone in preparation for upcoming CMB experiments: scaling their data simulation and reduction framework TOAST (Time Ordered Astrophysics Scalable Tools) to run on all 658,784 Intel Knights Landing (KNL) Xeon Phi processor cores on the National Energy Research Scientific Computing Center’s (NERSC’s) Cori system.  

The team also extended TOAST’s capabilities to support ground-based telescope observations, including implementing a module to simulate the noise introduced by looking through the atmosphere, which must then be removed to get a clear picture of the CMB. All of these achievements were made possible with funding from Berkeley’s Laboratory Directed Research and Development (LDRD) program. 

“Over the next 10 years, the CMB community is expecting a 1,000-fold increase in the volume of data being gathered and analyzed—better than Moore’s Law scaling, even as we enter an era of energy-constrained computing,” says Julian Borrill, a cosmologist in Berkeley Lab’s Computational Research Division (CRD) and head of C3. “This means that we’ve got to sit at the bleeding edge of computing just to keep up with the data volume.”

TOAST: Balancing Scientific Accessibility and Performance

To ensure that they are making the most of the latest in computing technology, the C3 team worked closely with staff from NERSC, Intel and Cray to get their TOAST code to run on all of Cori supercomputer’s 658,784 KNL processors. This collaboration is part of the NERSC Exascale Science Applications Program (NESAP), which helps science code teams adapt their software to take advantage of Cori’s manycore architecture and could be a stepping-stone to next generation exascale supercomputers.

“In the CMB community, telescope properties differ greatly, and up until now each group typically had its own approach to processing data. To my knowledge, TOAST is the first attempt to create a tool that is useful for the entire CMB community,” says Ted Kisner, a Computer Systems Engineer in C3 and one of the lead TOAST developers.

“TOAST has a modular design that allows it to adapt to any detector or telescope quite easily,” says Rollin Thomas, a big data architect at NERSC who helped the team scale TOAST on Cori. “So instead of having a lot of different people independently re-inventing the wheel for each new experiment, thanks to C3 there is now a tool that the whole community can embrace.”    

According to Kisner, the challenges to building a tool that can be used by the entire CMB community were both technical and sociological. Technically, the framework had to perform well at high concurrency on a variety of systems, including supercomputers, desktop workstations and laptops. It also had to be flexible enough to interface with different data formats and other software tools. Sociologically, parts of the framework that researchers interact with frequently had to be written in a high-level programming language that many scientists are familiar with.

The C3 team achieved a balance between computing performance and accessibility by creating a hybrid application. Parts of the framework are written in C and C++ to ensure that it can run efficiently on supercomputers, but it also includes a layer written in Python, so that researchers can easily manipulate the data and prototype new analysis algorithms.

“Python is a tremendously popular and important programming language, it’s easy to learn and scientists value how productive it makes them. For many scientists and graduate students, this is the only programming language they know,” says Thomas. “By making Python the interface to TOAST, the C3 team essentially opens up HPC resources and experiments to scientists that would otherwise be struggling with big data and not have access to supercomputers. It also helps scientists focus their customization efforts at parts of the code where differences between experiments matter the most, and re-use lower-level algorithms common across all the experiments.”

To ensure that all of TOAST could effectively scale up to 658,784 KNL cores, Thomas and his colleagues at NERSC helped the team launch their software on Cori with Shifter—an open-source, software package developed at NERSC to help supercomputer users easily and securely run software packaged as Linux Containers. Linux container solutions, like Shifter, allow an application to be packaged with its entire software stack including libraries, binaries and scripts as well as defining other run-time parameters like environment variables.  This makes it easy for a user to repeatedly and reliably run applications even at large-scales.

“This collaboration is a great example of what NERSC’s NESAP for data program can do for science,” says Thomas. “By fostering collaborations between the C3 team and Intel engineers, we increased their productivity on KNL. Then, we got them to scale up to 658,784 KNL cores with Shifter. This is the biggest Shifter job done for science so far.” 

With this recent hero run, the cosmologists also accomplished an important scientific milestone: simulating and mapping 50,000 detectors observing 20 percent of the sky at 7 frequencies for 1 year. That’s the scale of data expected to be collected by the Simons Observatory, which is an important stepping-stone to CMB-S4.

"Collaboration with NERSC is essential for Intel Python engineers – this is unique opportunity for us to scale Python and other tools to hundreds thousands of cores," says Sergey Maidanov, Software Engineering Manager at Intel. "TOAST was among a few applications where multiple tools helped to identify and address performance scaling bottlenecks, from Intel® MKL and Intel® VTune™ Amplifier to Intel® Trace Analyzer and Collector and other tools. Such a collaboration helps us to set the direction for our tools development."

Accounting for the Atmosphere

The C3 team originally deployed TOAST at NERSC nearly a decade ago primarily to support data analysis for Planck, a space-based mission that observed the sky for four years with 72 detectors. By contrast, CMB-S4 will scan the sky with a suite of ground-based telescopes, fielding a total of around 500,000 detectors for about five years beginning in the mid 2020s.

In preparation for these ground-based observations, the C3 team recently added an atmospheric simulation module that naturally generates correlated atmospheric noise for all detectors, even detectors on different telescopes in the same location. This approach allows researchers to test new analysis algorithms on much more realistic simulated data.

“As each detector observes the microwave sky through the atmosphere it captures a lot of thermal radiation from water vapor, producing extremely correlated noise fluctuations between the detectors,” says Reijo Keskitalo, a C3 computer systems engineer who led the atmospheric simulation model development. 

Keskitalo notes that previous efforts by the CMB community typically simulated the correlated atmospheric noise for each detector separately. The problem with this approach is it can’t scale to the huge numbers of detectors expected for experiments like CMB-S4. But by simulating the common atmosphere observed by all the detectors once, the novel C3 method ensures that the simulations are both tractable and realistic.

“For satellite experiments like Planck, the atmosphere isn’t an issue. But when you are observing the CMB with ground-based telescopes, the atmospheric noise problem is malignant because it doesn’t average out with more detectors. Ultimately, we needed a tool that would simulate something that looks like the atmosphere because you don’t get a realistic idea of experiment performance without it,” says Keskitalo.

“The ability to simulate and reduce the extraordinary data volume with sufficient precision and speed will be absolutely critical to achieving CMB-S4’s science goals,” says Borrill.

In the short term, tens of realizations are needed to develop the mission concept, he adds. In the medium term, hundreds of realizations are required for detailed mission design and the validation and verification of the analysis pipelines. Long term, tens of thousands of realizations will be vital for the Monte Carlo methods used to obtain the final science results.   

“CMB-S4 will be a large, distributed collaboration involving at least 4 DOE labs. We will continue to use NERSC – which has supported the CMB community for 20 years now – and, given our requirements, likely need the Argonne Leadership Class Facility (ALCF) systems too. There will inevitably be several generations of HPC architecture over the lifetime of this effort, and our recent work is a stepping stone that allows us to take full advantage of the Xeon Phi based systems currently being deployed at NERSC,” says Borrill.

The work was funded through Berkeley Lab’s LDRD program designed to seed innovative science and new research directions. NERSC and ALCF are both DOE Office of Science User Facilities.

The Office of Science of the U.S. Department of Energy supports Berkeley Lab. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit


The telescope's view through one realization of turbulent, wind-blown, atmospheric water vapor. The volume of atmosphere being simulated depended on (a) the scan width and duration and (b) the wind speed and direction, both of which changed every 20 minutes. The entire observation used about 5000 such realizations.

  • Filters

  • × Clear Filters

National MagLab's Latest Magnet Snags World Record, Marks New Era of Scientific Discovery

The Florida State University-headquartered National High Magnetic Field Laboratory has shattered another world record with the testing of a 32-tesla magnet -- 33 percent stronger than what had previously been the world's strongest superconducting magnet used for research and more than 3,000 times stronger than a small refrigerator magnet.

Clearing the Air

A greater understanding of the dynamics of chemical reactions is leading to better models of atmospheric chemistry. Through this work, scientists are gaining insight into a key chemical able to break down some major air pollutants.

The Wet Road to Fast and Stable Batteries

An international team of scientists --- including several researchers from the U.S. Department of Energy's (DOE) Argonne National Laboratory -- - has discovered an anode battery material with superfast charging and stable operation over many thousands of cycles.

Light Perfects Interfaces

Shining light on a growing semiconductor modifies its interface with the surface and could improve the optical properties of each.

Advance in Light Filtering Technology Has Implications for LCD Screens, Lasers and Beyond

Vector polarizers are a light filtering technology hidden behind the operation of many optical systems. They can be found, for instance, in sunglasses, LCD screens, microscopes, microprocessors, laser machining and more. Optical physicists published details of their new vector polarizer design this week in APL Photonics. The newly proposed design is a major advance in polarization technology because it enables flexible filtering of a wide range of light sources and generation of new light states.

Accelerating the Self-Assembly of Nanoscale Patterns for Next-Generation Materials

Scientists have come up with a way to massively speed up the ordering process for self-assembling materials. The resulting ultra-small, well-ordered patterns could be used in the fabrication of microelectronics, antireflective surfaces, magnetic data storage systems, and fluid-flow devices.

Beta of Neurodata Without Borders Software Now Available

Neuroscientists can now explore a beta version of the new Neurodata Without Borders: Neurophysiology (NWB:N 2.0) software and offer input to developers before it is fully released next year.

Scientists Discover Path to Improving Game-Changing Battery Electrode

Researchers from Stanford University, two Department of Energy national labs and the battery manufacturer Samsung created a comprehensive picture of how the same chemical processes that give cathodes their high capacity are also linked to changes in atomic structure that sap performance.

ESnet's Petascale DTN Project Speeds up Data Transfers between Leading HPC Centers

A new Petascale Data Transfer Node project aims to to achieve regular disk-to-disk, end-to-end transfer rates of one petabyte per week between major supercomputing facilities, which translates to achievable throughput rates of about 15 Gbps on real world science data sets.

Underappreciated Microbes Now Get Credit for Holding Down Two Jobs in Soil

Soil microbes work as both decomposers and synthesizers of carbon compounds in soil, offering new answers with impacts to crops and eco-health.

  • Filters

  • × Clear Filters

LLNL Releases Newly Declassified Nuclear Test Videos

Researchers at Lawrence Livermore National Laboratory (LLNL) released 62 newly declassified videos today of atmospheric nuclear tests films that have never before been seen by the public.

NAU Researchers Join DOE Project to Study the Soil Microbiome and Its Effect on Carbon Persistence

NAU Regents' Professor Bruce Hungate, director of the Center for Ecosystem Science and Society (Ecoss), recently joined a new initiative lead by LLNL to study how the soil microbiome controls the mechanisms that regulate the stabilization of the organic matter in soil.

Four Scientists Win the Los Alamos Medal

Los Alamos National Laboratory will award four former researchers with the Los Alamos Medal for their scientific contributions.

Stewart Prager Honored with FPA Distinguished Career Award

Announcement of Fusion Power Associates career award for Stewart Prager

WVU Physicists Among Collaborators Granted $7 Million to Form U.S. Department of Energy Center of Excellence

Scientists pause each afternoon at Kirtland Air Force Base in Sandia National Laboratories in Albuquerque, New Mexico, awaiting the daily lightning flash and unmistakable floor jolt that accompanies a Z shot

US Dept. Of Energy Grant to Advance Combined Heat and Power Systems in the Midwest

The University of Illinois at Chicago has received a five-year, $4.2 million grant from the U.S. Department of Energy to help industrial, commercial, institutional and utility entities evaluate and install highly efficient combined heat and power (CHP) technologies.CHP, also known as cogeneration, is a single system that produces both thermal energy and electricity.

Applications Open: ECS Toyota Young Investigator Fellowship 2018-2019

ECS, in a continued partnership with the Toyota Research Institute of North America (TRINA), a division of Toyota Motor Engineering & Manufacturing North America, Inc. (TEMA), is requesting proposals from young professors and scholars pursuing innovative electrochemical research in green energy technology.

Successful Startup Founder to Lead Entrepreneurship Program at Argonne

John Carlisle has been named the director of Chain Reaction Innovations (CRI), a program aimed at accelerating job creation through innovation, based at the U.S. Department of Energy's Argonne National Laboratory.

Department of Energy Supports Argonne Nuclear Technologies

This fall, U.S. Department of Energy Secretary Rick Perry announced nearly $4.7 million in funding for the department's Argonne National Laboratory across 16 projects in three divisions. Four of those TCF awards, representing more than $1 million in funds, are slated for Argonne's Nuclear Engineering division.

Southern Research Develops Gasifier Technology to Unlock Coal's Potential

Southern Research has been selected to receive nearly $1.7 million in U.S. Department of Energy funding to develop a new, cost-efficient gasifier capable of converting low-grade coal into synthesis gas (syngas) that can be used in a number of applications.

  • Filters

  • × Clear Filters

Molecular Mousetraps Capture More Nuclear Waste

Cage-like molecules with internal chemical hooks remove three times more hazardous radioactive iodine compounds than current methods.

New Quantum Liquid Crystal--In the Driver's Seat

Lasers reveal a new state of matter--the first 3-D quantum liquid crystal.

Chemical "Pressure" Tuning Magnetic Properties

Unexpectedly, a little chemical substitution stabilizes unusual magnetic phase of vortexes called skyrmions.

Stirring up a Quantum Spin Liquid with Disorder

New, unexpected paradigm discovered: Disorder may actually promote an exotic quantum state, with potential for ultrafast computing.

Light Perfects Interfaces

Shining light on a growing semiconductor modifies its interface with the surface and could improve the optical properties of each.

Underappreciated Microbes Now Get Credit for Holding Down Two Jobs in Soil

Soil microbes work as both decomposers and synthesizers of carbon compounds in soil, offering new answers with impacts to crops and eco-health.

Energy, Economy, and the Earth: The Benefits of Creating Feedback Loops

Scientists reduce uncertainties in future climate prediction by directly coupling an energy-economy model to an Earth system model.

How Grasslands Regulate Their Productivity in Response to Droughts

Scientists show that grasslands are more sensitive to changes in the amount of moisture in the air than to changes in precipitation.

Building Confidence in Hydrologic Models

Scientists evaluate seven hydrologic models to understand how each model agrees and differs.

El Nino and Liquid Water Clouds Contribute to Antarctic Melt in 2015-2016

Atmospheric Radiation Measurement (ARM) observations provide clues on atmospheric contributions to an Antarctic melt event.


Tuesday October 03, 2017, 01:05 PM

Stairway to Science

Argonne National Laboratory

Thursday September 28, 2017, 12:05 PM

After-School Energy Rush

Argonne National Laboratory

Thursday September 28, 2017, 10:05 AM

Bringing Diversity Into Computational Science Through Student Outreach

Brookhaven National Laboratory

Thursday September 21, 2017, 03:05 PM

From Science to Finance: SLAC Summer Interns Forge New Paths in STEM

SLAC National Accelerator Laboratory

Thursday September 07, 2017, 02:05 PM

Students Discuss 'Cosmic Opportunities' at 45th Annual SLAC Summer Institute

SLAC National Accelerator Laboratory

Thursday August 31, 2017, 05:05 PM

Binghamton University Opens $70 Million Smart Energy Building

Binghamton University, State University of New York

Wednesday August 23, 2017, 05:05 PM

Widening Horizons for High Schoolers with Code

Argonne National Laboratory

Saturday May 20, 2017, 12:05 PM

Rensselaer Polytechnic Institute Graduates Urged to Embrace Change at 211th Commencement

Rensselaer Polytechnic Institute (RPI)

Monday May 15, 2017, 01:05 PM

ORNL, University of Tennessee Launch New Doctoral Program in Data Science

Oak Ridge National Laboratory

Friday April 07, 2017, 11:05 AM

Champions in Science: Profile of Jonathan Kirzner

Department of Energy, Office of Science

Wednesday April 05, 2017, 12:05 PM

High-Schooler Solves College-Level Security Puzzle From Argonne, Sparks Interest in Career

Argonne National Laboratory

Tuesday March 28, 2017, 12:05 PM

Champions in Science: Profile of Jenica Jacobi

Department of Energy, Office of Science

Friday March 24, 2017, 10:40 AM

Great Neck South High School Wins Regional Science Bowl at Brookhaven Lab

Brookhaven National Laboratory

Wednesday February 15, 2017, 04:05 PM

Middle Schoolers Test Their Knowledge at Science Bowl Competition

Argonne National Laboratory

Friday January 27, 2017, 04:00 PM

Haslam Visits ORNL to Highlight State's Role in Discovering Tennessine

Oak Ridge National Laboratory

Tuesday November 08, 2016, 12:05 PM

Internship Program Helps Foster Development of Future Nuclear Scientists

Oak Ridge National Laboratory

Friday May 13, 2016, 04:05 PM

More Than 12,000 Explore Jefferson Lab During April 30 Open House

Thomas Jefferson National Accelerator Facility

Monday April 25, 2016, 05:05 PM

Giving Back to National Science Bowl

Ames Laboratory

Friday March 25, 2016, 12:05 PM

NMSU Undergrad Tackles 3D Particle Scattering Animations After Receiving JSA Research Assistantship

Thomas Jefferson National Accelerator Facility

Tuesday February 02, 2016, 10:05 AM

Shannon Greco: A Self-Described "STEM Education Zealot"

Princeton Plasma Physics Laboratory

Monday November 16, 2015, 04:05 PM

Rare Earths for Life: An 85th Birthday Visit with Mr. Rare Earth

Ames Laboratory

Tuesday October 20, 2015, 01:05 PM

Meet Robert Palomino: 'Give Everything a Shot!'

Brookhaven National Laboratory

Tuesday April 22, 2014, 11:30 AM

University of Utah Makes Solar Accessible

University of Utah

Wednesday March 06, 2013, 03:40 PM

Student Innovator at Rensselaer Polytechnic Institute Seeks Brighter, Smarter, and More Efficient LEDs

Rensselaer Polytechnic Institute (RPI)

Friday November 16, 2012, 10:00 AM

Texas Tech Energy Commerce Students, Community Light up Tent City

Texas Tech University

Wednesday November 23, 2011, 10:45 AM

Don't Get 'Frosted' Over Heating Your Home This Winter

Temple University

Wednesday July 06, 2011, 06:00 PM

New Research Center To Tackle Critical Challenges Related to Aircraft Design, Wind Energy, Smart Buildings

Rensselaer Polytechnic Institute (RPI)

Friday April 22, 2011, 09:00 AM

First Polymer Solar-Thermal Device Heats Home, Saves Money

Wake Forest University

Friday April 15, 2011, 12:25 PM

Like Superman, American University Will Get Its Energy from the Sun

American University

Thursday February 10, 2011, 05:00 PM

ARRA Grant to Help Fund Seminary Building Green Roof

University of Chicago

Tuesday December 07, 2010, 05:00 PM

UC San Diego Installing 2.8 Megawatt Fuel Cell to Anchor Energy Innovation Park

University of California San Diego

Monday November 01, 2010, 12:50 PM

Rensselaer Smart Lighting Engineering Research Center Announces First Deployment of New Technology on Campus

Rensselaer Polytechnic Institute (RPI)

Friday September 10, 2010, 12:40 PM

Ithaca College Will Host Regional Clean Energy Summit

Ithaca College

Tuesday July 27, 2010, 10:30 AM

Texas Governor Announces $8.4 Million Award to Create Renewable Energy Institute

Texas Tech University

Friday May 07, 2010, 04:20 PM

Creighton University to Offer New Alternative Energy Program

Creighton University

Wednesday May 05, 2010, 09:30 AM

National Engineering Program Seeks Subject Matter Experts in Energy

JETS Junior Engineering Technical Society

Wednesday April 21, 2010, 12:30 PM

Students Using Solar Power To Create Sustainable Solutions for Haiti, Peru

Rensselaer Polytechnic Institute (RPI)

Wednesday March 03, 2010, 07:00 PM

Helping Hydrogen: Student Inventor Tackles Challenge of Hydrogen Storage

Rensselaer Polytechnic Institute (RPI)

Thursday February 04, 2010, 02:00 PM

Turning Exercise into Electricity

Furman University

Thursday November 12, 2009, 12:45 PM

Campus Leaders Showing the Way to a Sustainable, Clean Energy Future

National Wildlife Federation (NWF)

Tuesday November 03, 2009, 04:20 PM

Furman University Receives $2.5 Million DOE Grant for Geothermal Project

Furman University

Thursday September 17, 2009, 02:45 PM

Could Sorghum Become a Significant Alternative Fuel Source?

Salisbury University

Wednesday September 16, 2009, 11:15 AM

Students Navigating the Hudson River With Hydrogen Fuel Cells

Rensselaer Polytechnic Institute (RPI)

Wednesday September 16, 2009, 10:00 AM

College Presidents Flock to D.C., Urge Senate to Pass Clean Energy Bill

National Wildlife Federation (NWF)

Wednesday July 01, 2009, 04:15 PM

Northeastern Announces New Professional Master's in Energy Systems

Northeastern University

Friday October 12, 2007, 09:35 AM

Kansas Rural Schools To Receive Wind Turbines

Kansas State University

Thursday August 17, 2006, 05:30 PM

High Gas Prices Here to Stay, Says Engineering Professor

Rowan University

Showing results

0-4 Of 2215