Doe Science news source
The DOE Science News Source is a Newswise initiative to promote research news from the Office of Science of the DOE to the public and news media.
  • 2017-11-28 12:05:18
  • Article ID: 685820

High-Performance Computing Cuts Particle Collision Data Prep Time

New approach to raw data reconstruction has potential to turn particle tracks into physics discoveries faster

  • Credit: NERSC

    Cori, the newest supercomputer at the National Energy Research Scientific Computing Center (NERSC), is a Cray XC40 with a peak performance of about 30 petaflops.

  • Credit: Brookhaven National Laboratory

    Mark Lukascsyk, Jérôme Lauret, and Levente Hajdu standing beside a tape silo at the RHIC & ATLAS Computing Facility at Brookhaven National Laboratory. Data sets from RHIC runs are stored on tape and were transferred from Brookhaven to NERSC.

  • Credit: Brookhaven National Laboratory

    In high-throughput computing, a workload made up of data from many STAR collisions is processed event-by-event in a sequential manner to give physicists "reconstructed data" —the product they need to fully analyze the data. High-performance computing breaks the workload into smaller chunks that can be run through separate CPUs to speed up the data reconstruction. In this simple illustration, breaking a workload of 15 events into three chunks of five events processed in parallel yields the same product in one-third the time as the high-throughput method. Using 32 CPUs on a supercomputer like Cori can greatly reduce the time it takes to transform the raw data from a real STAR dataset, with many millions of events, into useful information physicists can analyze to make discoveries.

For the first time, scientists have used high-performance computing (HPC) to reconstruct the data collected by a nuclear physics experiment—an advance that could dramatically reduce the time it takes to make detailed data available for scientific discoveries.

The demonstration project used the Cori supercomputer at the National Energy Research Scientific Computing Center (NERSC), a high-performance computing center at Lawrence Berkeley National Laboratory in California, to reconstruct multiple datasets collected by the STAR detector during particle collisions at the Relativistic Heavy Ion Collider (RHIC), a nuclear physics research facility at Brookhaven National Laboratory in New York. By running multiple computing jobs simultaneously on the allotted supercomputing cores, the team transformed 4.73 petabytes of raw data into 2.45 petabytes of “physics-ready” data in a fraction of the time it would have taken using in-house high-throughput computing resources, even with a two-way transcontinental data journey.

“The reason why this is really fantastic,” said Brookhaven physicist Jérôme Lauret, who manages STAR’s computing needs, “is that these high-performance computing resources are elastic. You can call to reserve a large allotment of computing power when you need it—for example, just before a big conference when physicists are in a rush to present new results.” According to Lauret, preparing raw data for analysis typically takes many months, making it nearly impossible to provide such short-term responsiveness. “But with HPC, perhaps you could condense that many months production time into a week. That would really empower the scientists!”

The accomplishment showcases the synergistic capabilities of RHIC and NERSC—U.S. Department of Energy (DOE) Office of Science User Facilities located at DOE-run national laboratories on opposite coasts—connected by one of the most extensive high-performance data-sharing networks in the world, DOE’s Energy Sciences Network (ESnet), another DOE Office of Science User Facility.

“This is a key usage model of high-performance computing for experimental data, demonstrating that researchers can get their raw data processing or simulation campaigns done in a few days or weeks at a critical time instead of spreading out over months on their own dedicated resources,” said Jeff Porter, a member of the data and analytics services team at NERSC.

Billions of data points

To make physics discoveries at RHIC, scientists must sort through hundreds of millions of collisions between ions accelerated to very high energy. STAR, a sophisticated, house-sized electronic instrument, records the subatomic debris streaming from these particle smashups. In the most energetic events, many thousands of particles strike detector components, producing firework-like displays of colorful particle tracks. But to figure out what these complex signals mean, and what they can tell us about the intriguing form of matter created in RHIC’s collisions, scientists need detailed descriptions of all the particles and the conditions under which they were produced. They must also compare huge statistical samples from many different types of collision events.

Cataloging that information requires sophisticated algorithms and pattern recognition software to combine signals from the various readout electronics, and a seamless way to match that data with records of collision conditions. All the information must then be packaged in a way that physicists can use for their analyses.

Since RHIC started running in the year 2000, this raw data processing, or reconstruction, has been carried out on dedicated computing resources at the RHIC and ATLAS Computing Facility (RACF) at Brookhaven. High-throughput computing (HTC) clusters crunch the data, event-by-event, and write out the coded details of each collision to a centralized mass storage space accessible to STAR physicists around the world.

But the challenge of keeping up with the data has grown with RHIC’s ever-improving collision rates and as new detector components have been added. In recent years, STAR’s annual raw data sets have reached billions of events with data sizes in the multi-Petabyte range. So the STAR computing team investigated the use of external resources to meet the demand for timely access to physics-ready data.  

Many cores make light work

Unlike the high-throughput computers at the RACF, which analyze events one-by-one, HPC resources like those at NERSC break large problems into smaller tasks that can run in parallel. So the first challenge was to “parallelize” the processing of STAR event data.

“We wrote workflow programs that achieved the first level of parallelization—event parallelization,” Lauret said. That means they submit fewer jobs made of many events that can be processed simultaneously on the many HPC computing cores.

 

“Imagine building a city with 100 homes. If this was done in high-throughput fashion, each home would have one builder doing all the tasks in sequence—building the foundation, the walls, and so on,” Lauret said. “But with HPC we change the paradigm. Instead of one worker per house we have 100 workers per house, and each worker has a task—building the walls or the roof. They work in parallel, at the same time, and we assemble everything together at the end. With this approach, we will build that house 100 times faster.”

Of course, it takes some creativity to think about how such problems can be broken up into tasks that can run simultaneously instead of sequentially, Lauret added.

HPC also saves time matching raw detector signals with data on the environmental conditions during each event. To do this, the computers must access a “condition database”—a record of the voltage, temperature, pressure, and other detector conditions that must be accounted for in understanding the behavior of the particles produced in each collision. In event-by-event, high-throughput reconstruction, the computers call up the database to retrieve data for every single event. But because HPC cores share some memory, events that occur close in time can use the same cached condition data. Fewer calls to the database means faster data processing.

Networking teamwork

Another challenge in migrating the task of raw data reconstruction to an HPC environment was just getting the data from New York to the supercomputers in California and back. Both the input and output datasets are huge. The team started small with a proof-of-principle experiment—just a few hundred jobs—to see how their new workflow programs would perform.

“We had a lot of assistance from the networking professionals at Brookhaven,” said Lauret, “particularly Mark Lukascsyk, one of our network engineers, who was so excited about the science and helping us make discoveries.” Colleagues in the RACF and ESnet also helped identify hardware issues and developed solutions as the team worked closely with Jeff Porter, Mustafa Mustafa, and others at NERSC to optimize the data transfer and the end-to-end workflow.

Start small, scale up

After fine-tuning their methods based on the initial tests, the team started scaling up to using 6,400 computing cores at NERSC, then up and up and up.

“6,400 cores is already half of the size of the resources available for data reconstruction at RACF,” Lauret said. “Eventually we went to 25,600 cores in our most recent test.” With everything ready ahead of time for an advance-reservation allotment of time on the Cori supercomputer, “we did this test for a few days and got an entire data production done in no time,” Lauret said.According to Porter at NERSC, “This model is potentially quite transformative, and NERSC has worked to support such resource utilization by, for example, linking its center-wide high-performant disk system directly to its data transfer infrastructure and allowing significant flexibility in how job slots can be scheduled.”  

The end-to-end efficiency of the entire process—the time the program was running (not sitting idle, waiting for computing resources) multiplied by the efficiency of using the allotted supercomputing slots and getting useful output all the way back to Brookhaven—was 98 percent.

“We’ve proven that we can use the HPC resources efficiently to eliminate backlogs of unprocessed data and resolve temporary resource demands to speed up science discoveries,” Lauret said.

He’s now exploring ways to generalize the workflow to the Open Science Grid—a global consortium that aggregates computing resources—so the entire community of high-energy and nuclear physicists can make use of it.

This work was supported by the DOE Office of Science.

This animation shows a series of collision events at STAR, each with thousands of particle tracks and the signals registered as some of those particles strike various detector components. It should give you an idea of how complex the challenge is to reconstruct a complete record of every single particle and the conditions under which it was created so scientists can compare hundreds of millions of events to look for trends and make discoveries.

 

 

Brookhaven National Laboratory is supported by the Office of Science of the U.S. Department of Energy. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.

Follow @BrookhavenLab on Twitter or find us on Facebook.

X
X
X
  • Filters

  • × Clear Filters

Researchers Achieve HD Video Streaming at 10,000 Times Lower Power

Engineers at the University of Washington have developed a new HD video streaming method that doesn't need to be plugged in. Their prototype skips the power-hungry components and has something else, like a smartphone, process the video instead.

Lawrence Livermore Issues Combined State-by-State Energy and Water Use Flow Charts

For the first time, Lawrence Livermore National Laboratory (LLNL) has issued state-by-state energy and water flow charts in one location so that analysts and policymakers can find all the information they need in one place.

Battery's Hidden Layer Revealed

An international team led by Argonne National Laboratory makes breakthrough in understanding the chemistry of the microscopically thin layer that forms between the liquid electrolyte and solid electrode in lithium-ion batteries. The results are being used in improving the layer and better predicting battery lifetime.

Ramp Compression of Iron Provides Insight into Core Conditions of Large Rocky Exoplanets

A team of researchers from Lawrence Livermore National Laboratory (LLNL), Princeton University, Johns Hopkins University and the University of Rochester have provided the first experimentally based mass-radius relationship for a hypothetical pure iron planet at super-Earth core conditions. This discovery can be used to evaluate plausible compositional space for large, rocky exoplanets, forming the basis of future planetary interior models, which in turn can be used to more accurately interpret observation data from the Kepler space mission and aid in identifying planets suitable for habitability.

Getting Magnesium Ions to Pick Up the Pace

Magnesium ions move very fast to enable a new class of battery materials.

Valleytronics Discovery Could Extend Limits of Moore's Law

Research appearing today in Nature Communications finds useful new information-handling potential in samples of tin(II) sulfide (SnS), a candidate "valleytronics" transistor material that might one day enable chipmakers to pack more computing power onto microchips. 

Scientists Use Machine Learning to Speed Discovery of Metallic Glass

SLAC and its collaborators are transforming the way new materials are discovered. In a new report, they combine artificial intelligence and accelerated experiments to discover potential alternatives to steel in a fraction of the time.

Seeing How Next-Generation Batteries Power-Up

Scientists directly see how the atoms in a magnesium-based battery fit into the structure of electrodes.

A Heavyweight Solution for Lighter-Weight Combat Vehicles

Researchers at Pacific Northwest National Laboratory have developed and successfully tested a novel process - called Friction Stir Dovetailing - that joins thick plates of aluminum to steel. The new process will be used to make lighter-weight military vehicles that are more agile and fuel efficient.

How to Turn Light Into Atomic Vibrations

Converting laser light into nuclear vibrations is key to switching a material's properties on and off for future electronics.


  • Filters

  • × Clear Filters

Five Leading Liberal Arts Colleges Partner to Create New Solar Energy Facility in Maine

Amherst, Bowdoin, Hampshire, Smith and Williams colleges have formed a partnership that will allow them to offset 46,000 megawatt hours per year of their collective electrical needs--enough to power 5,000 New England homes--with electricity created at a solar power facility to be built in Maine.

Argonne Selects Innovators From Across Nation to Grow Startups

Argonne announces second cohort of Chain Reaction Innovations.

Brookhaven Lab Materials Physicist Yimei Zhu Receives 2018 Distinguished Scientist Award from the Microscopy Society of America

How do complex atomic and electronic interactions impact material properties? Using electron microscopy instrumentation and methods he developed, Yimei Zhu has been investigating this question for the past 30 years. The Microscopy Society of America is now recognizing his contributions.

SLAC Produces First Electron Beam with Superconducting Electron Gun

Accelerator scientists at the Department of Energy's SLAC National Accelerator Laboratory are testing a new type of electron gun for a future generation of instruments that take snapshots of the atomic world in never-before-seen quality and detail, with applications in chemistry, biology, energy and materials science.

U.S., India Sign Agreement Providing for Neutrino Physics Collaboration at Fermilab and in India

Earlier today, April 16, 2018, U.S. Secretary of Energy Rick Perry and India's Atomic Energy Secretary Dr. Sekhar Basu signed an agreement in New Delhi to expand the two countries' collaboration on world-leading science and technology projects. It opens the way for jointly advancing cutting-edge neutrino science projects under way in both countries: the Long-Baseline Neutrino Facility (LBNF) with the international Deep Underground Neutrino Experiment (DUNE) hosted at the U.S. Department of Energy's Fermilab and the India-based Neutrino Observatory (INO).

Nanomaterials Expert Ganpati Ramanath Named Fellow of Materials Research Society

Nanomaterials expert Ganpati Ramanath, the John Tod Horton '52 Professor of Materials Science and Engineering at Rensselaer Polytechnic Institute, has been named a fellow of the Materials Research Society (MRS) "for developing creative approaches to realize new nanomaterials via chemically directed nanostructure synthesis and assembly and for tailoring interfaces in electronics and energy applications using molecular nanolayers."

Doing the Neutron Dance

Two materials scientists, Suzanne te Velthuis and Stephan Rosenkranz, have been named fellows of the Neutron Scattering Society of America (NSSA).

Hirohisa Tanaka Joins SLAC to Push Limits of Neutrino Physics

Accomplished neutrino physicist Hirohisa Tanaka has joined the Department of Energy's SLAC National Accelerator Laboratory as a professor of particle physics and astrophysics. He oversees a group at the lab that is preparing for research with the future Deep Underground Neutrino Experiment (DUNE) at the Long-Baseline Neutrino Facility (LBNF). The experiment will give scientists unprecedented opportunities to learn more about neutrinos - fundamental particles with mysterious properties that could play crucial roles in the evolution of the universe.

University Teams to Compete in Department of Energy's 2018 National Cyber Defense Competition

The U.S. Department of Energy is proud to announce the 29 university teams selected to compete in the third annual Cyber Defense Competition (CDC), taking place April 6-7, 2018.


  • Filters

  • × Clear Filters

Getting Magnesium Ions to Pick Up the Pace

Magnesium ions move very fast to enable a new class of battery materials.

Seeing How Next-Generation Batteries Power-Up

Scientists directly see how the atoms in a magnesium-based battery fit into the structure of electrodes.

Worm-Inspired Tough Materials

Scientists mimic a worm's lethal jaw to design and form resilient materials.

How to Turn Light Into Atomic Vibrations

Converting laser light into nuclear vibrations is key to switching a material's properties on and off for future electronics.

Superacids Are Good Medicine for Super Thin Semiconductors

Scientists demonstrated that powerful acids heal certain structural defects in synthetic films.

Tubular Science Improves Polymer Solar Cells

Novel engineered polymers assemble buckyballs into columns using a conventional coating process.

Fast! Hard X-Ray Flash Breaks Speed Record

Lasting just a few hundred billionths of a billionth of a second, these bursts offer new tool to study chemistry and magnetism.

Scientists Have Overestimated Meteor Sizes

First demonstration of high-pressure metastability mapping with ultrafast X-ray diffraction shows objects aren't as large as previously thought.

Rewriting Resistance: Genetic Changes Increase Crops' Biomass and Sugar Release

Using genetic engineering, scientists improve biomass growth and conversion in woody and grassy feedstocks.

Measuring the Glow of Plants From Below

Novel observations suggest a great potential of measuring global gross primary production via solar-induced fluorescence.


Spotlight

Thursday April 12, 2018, 07:05 PM

The Race for Young Scientific Minds

Argonne National Laboratory

Wednesday March 14, 2018, 02:05 PM

Q&A: Al Ashley Reflects on His Efforts to Diversify SLAC and Beyond

SLAC National Accelerator Laboratory

Thursday February 15, 2018, 12:05 PM

Insights on Innovation in Energy, Humanitarian Aid Highlight UVA Darden's Net Impact Week

University of Virginia Darden School of Business

Friday February 09, 2018, 11:05 AM

Ivy League Graduate, Writer and Activist with Dyslexia Visits CSUCI to Reframe the Concept of Learning Disabilities

California State University, Channel Islands

Wednesday January 17, 2018, 12:05 PM

Photographer Adam Nadel Selected as Fermilab's New Artist-in-Residence for 2018

Fermi National Accelerator Laboratory (Fermilab)

Wednesday January 17, 2018, 12:05 PM

Fermilab Computing Partners with Argonne, Local Schools for Hour of Code

Fermi National Accelerator Laboratory (Fermilab)

Wednesday December 20, 2017, 01:05 PM

Q&A: Sam Webb Teaches X-Ray Science from a Remote Classroom

SLAC National Accelerator Laboratory

Monday December 18, 2017, 01:05 PM

The Future of Today's Electric Power Systems

Rensselaer Polytechnic Institute (RPI)

Monday December 18, 2017, 12:05 PM

Supporting the Development of Offshore Wind Power Plants

Rensselaer Polytechnic Institute (RPI)

Tuesday October 03, 2017, 01:05 PM

Stairway to Science

Argonne National Laboratory

Thursday September 28, 2017, 12:05 PM

After-School Energy Rush

Argonne National Laboratory

Thursday September 28, 2017, 10:05 AM

Bringing Diversity Into Computational Science Through Student Outreach

Brookhaven National Laboratory

Thursday September 21, 2017, 03:05 PM

From Science to Finance: SLAC Summer Interns Forge New Paths in STEM

SLAC National Accelerator Laboratory

Thursday September 07, 2017, 02:05 PM

Students Discuss 'Cosmic Opportunities' at 45th Annual SLAC Summer Institute

SLAC National Accelerator Laboratory

Thursday August 31, 2017, 05:05 PM

Binghamton University Opens $70 Million Smart Energy Building

Binghamton University, State University of New York

Wednesday August 23, 2017, 05:05 PM

Widening Horizons for High Schoolers with Code

Argonne National Laboratory

Saturday May 20, 2017, 12:05 PM

Rensselaer Polytechnic Institute Graduates Urged to Embrace Change at 211th Commencement

Rensselaer Polytechnic Institute (RPI)

Monday May 15, 2017, 01:05 PM

ORNL, University of Tennessee Launch New Doctoral Program in Data Science

Oak Ridge National Laboratory

Friday April 07, 2017, 11:05 AM

Champions in Science: Profile of Jonathan Kirzner

Department of Energy, Office of Science

Wednesday April 05, 2017, 12:05 PM

High-Schooler Solves College-Level Security Puzzle From Argonne, Sparks Interest in Career

Argonne National Laboratory

Tuesday March 28, 2017, 12:05 PM

Champions in Science: Profile of Jenica Jacobi

Department of Energy, Office of Science

Friday March 24, 2017, 10:40 AM

Great Neck South High School Wins Regional Science Bowl at Brookhaven Lab

Brookhaven National Laboratory

Wednesday February 15, 2017, 04:05 PM

Middle Schoolers Test Their Knowledge at Science Bowl Competition

Argonne National Laboratory

Friday January 27, 2017, 04:00 PM

Haslam Visits ORNL to Highlight State's Role in Discovering Tennessine

Oak Ridge National Laboratory

Tuesday November 08, 2016, 12:05 PM

Internship Program Helps Foster Development of Future Nuclear Scientists

Oak Ridge National Laboratory

Friday May 13, 2016, 04:05 PM

More Than 12,000 Explore Jefferson Lab During April 30 Open House

Thomas Jefferson National Accelerator Facility

Monday April 25, 2016, 05:05 PM

Giving Back to National Science Bowl

Ames Laboratory

Friday March 25, 2016, 12:05 PM

NMSU Undergrad Tackles 3D Particle Scattering Animations After Receiving JSA Research Assistantship

Thomas Jefferson National Accelerator Facility

Tuesday February 02, 2016, 10:05 AM

Shannon Greco: A Self-Described "STEM Education Zealot"

Princeton Plasma Physics Laboratory

Monday November 16, 2015, 04:05 PM

Rare Earths for Life: An 85th Birthday Visit with Mr. Rare Earth

Ames Laboratory

Tuesday October 20, 2015, 01:05 PM

Meet Robert Palomino: 'Give Everything a Shot!'

Brookhaven National Laboratory

Tuesday April 22, 2014, 11:30 AM

University of Utah Makes Solar Accessible

University of Utah

Wednesday March 06, 2013, 03:40 PM

Student Innovator at Rensselaer Polytechnic Institute Seeks Brighter, Smarter, and More Efficient LEDs

Rensselaer Polytechnic Institute (RPI)

Friday November 16, 2012, 10:00 AM

Texas Tech Energy Commerce Students, Community Light up Tent City

Texas Tech University

Wednesday November 23, 2011, 10:45 AM

Don't Get 'Frosted' Over Heating Your Home This Winter

Temple University

Wednesday July 06, 2011, 06:00 PM

New Research Center To Tackle Critical Challenges Related to Aircraft Design, Wind Energy, Smart Buildings

Rensselaer Polytechnic Institute (RPI)

Friday April 22, 2011, 09:00 AM

First Polymer Solar-Thermal Device Heats Home, Saves Money

Wake Forest University

Friday April 15, 2011, 12:25 PM

Like Superman, American University Will Get Its Energy from the Sun

American University

Thursday February 10, 2011, 05:00 PM

ARRA Grant to Help Fund Seminary Building Green Roof

University of Chicago

Tuesday December 07, 2010, 05:00 PM

UC San Diego Installing 2.8 Megawatt Fuel Cell to Anchor Energy Innovation Park

University of California San Diego

Monday November 01, 2010, 12:50 PM

Rensselaer Smart Lighting Engineering Research Center Announces First Deployment of New Technology on Campus

Rensselaer Polytechnic Institute (RPI)

Friday September 10, 2010, 12:40 PM

Ithaca College Will Host Regional Clean Energy Summit

Ithaca College

Tuesday July 27, 2010, 10:30 AM

Texas Governor Announces $8.4 Million Award to Create Renewable Energy Institute

Texas Tech University

Friday May 07, 2010, 04:20 PM

Creighton University to Offer New Alternative Energy Program

Creighton University

Wednesday May 05, 2010, 09:30 AM

National Engineering Program Seeks Subject Matter Experts in Energy

JETS Junior Engineering Technical Society

Wednesday April 21, 2010, 12:30 PM

Students Using Solar Power To Create Sustainable Solutions for Haiti, Peru

Rensselaer Polytechnic Institute (RPI)

Wednesday March 03, 2010, 07:00 PM

Helping Hydrogen: Student Inventor Tackles Challenge of Hydrogen Storage

Rensselaer Polytechnic Institute (RPI)

Thursday February 04, 2010, 02:00 PM

Turning Exercise into Electricity

Furman University





Showing results

0-4 Of 2215