- 2017-11-29 11:05:27
- Article ID: 685892
Scaling Deep Learning for Science
ORNL-designed algorithm leverages Titan to create high-performing deep neural networks
Now, researchers are eager to apply this computational technique—commonly referred to as deep learning—to some of science’s most persistent mysteries. But because scientific data often looks much different from the data used for animal photos and speech, developing the right artificial neural network can feel like an impossible guessing game for nonexperts. To expand the benefits of deep learning for science, researchers need new tools to build high-performing neural networks that don’t require specialized knowledge.
Using the Titan supercomputer, a research team led by Robert Patton of the US Department of Energy’s(DOE’s) Oak Ridge National Laboratory (ORNL) has developed an evolutionary algorithm capable of generating custom neural networks that match or exceed the performance of handcrafted artificial intelligence systems. Better yet, by leveraging the GPU computing power of the Cray XK7 Titan—the leadership-class machine managed by the Oak Ridge Leadership Computing Facility, a DOE Office of Science User Facility at ORNL—these auto-generated networks can be produced quickly, in a matter of hours as opposed to the months needed using conventional methods.
The research team’s algorithm, called MENNDL (Multinode Evolutionary Neural Networks for Deep Learning), is designed to evaluate, evolve, and optimize neural networks for unique datasets. Scaled across Titan’s 18,688 GPUs, MENNDL can test and train thousands of potential networks for a science problem simultaneously, eliminating poor performers and averaging high performers until an optimal network emerges. The process eliminates much of the time-intensive, trial-and-error tuning traditionally required of machine learning experts.
“There’s no clear set of instructions scientists can follow to tweak networks to work for their problem,” said research scientist Steven Young, a member of ORNL’s Nature Inspired Machine Learning team. “With MENNDL, they no longer have to worry about designing a network. Instead, the algorithm can quickly do that for them, while they focus on their data and ensuring the problem is well-posed.”
Pinning down parameters
Inspired by the brain’s web of neurons, deep neural networks are a relatively old concept in neuroscience and computing, first popularized by two University of Chicago researchers in the 1940s. But because of limits in computing power, it wasn’t until recently that researchers had success in training machines to independently interpret data.
Today’s neural networks can consist of thousands or millions of simple computational units—the “neurons”—arranged in stacked layers, like the rows of figures spaced across a foosball table. During one common form of training, a network is assigned a task (e.g., to find photos with cats) and fed a set of labeled data (e.g., photos of cats and photos without cats). As the network pushes the data through each successive layer, it makes correlations between visual patterns and predefined labels, assigning values to specific features (e.g., whiskers and paws). These values contribute to the weights that define the network’s model parameters. During training, the weights are continually adjusted until the final output matches the targeted goal. Once the network learns to perform from training data, it can then be tested against unlabeled data.
Although many parameters of a neural network are determined during the training process, initial model configurations must be set manually. These starting points, known as hyperparameters, include variables like the order, type, and number of layers in a network.
Finding the optimal set of hyperparameters can be the key to efficiently applying deep learning to an unusual dataset. “You have to experimentally adjust these parameters because there’s no book you can look in and say, ‘These are exactly what your hyperparameters should be,’” Young said. “What we did is use this evolutionary algorithm on Titan to find the best hyperparameters for varying types of datasets.”
Unlocking that potential, however, required some creative software engineering by Patton’s team. MENNDL homes in on a neural network’s optimal hyperparameters by assigning a neural network to each Titan node. The team designed MENNDL to use a deep learning framework called Caffe to carry out the computation, relying on the parallel computing Message Passing Interface standard to divide and distribute data among nodes. As Titan works through individual networks, new data is fed to the system’s nodes asynchronously, meaning once a node completes a task, it’s quickly assigned a new task independent of the other nodes’ status. This ensures that the 27-petaflop Titan stays busy combing through possible configurations.
“Designing the algorithm to really work at that scale was one of the challenges,” Young said. “To really leverage the machine, we set up MENNDL to generate a queue of individual networks to send to the nodes for evaluation as soon as computing power becomes available.”
To demonstrate MENNDL’s versatility, the team applied the algorithm to several datasets, training networks to identify sub-cellular structures for medical research, classify satellite images with clouds, and categorize high-energy physics data. The results matched or exceeded the performance of networks designed by experts.
Networking neutrinos
One science domain in which MENNDL is already proving its value is neutrino physics. Neutrinos, ghost-like particles that pass through your body at a rate of trillions per second, could play a major role in explaining the formation of the early universe and the nature of matter—if only scientists knew more about them.
Large detectors at DOE’s Fermi National Accelerator Laboratory (Fermilab) use high-intensity beams to study elusive neutrino reactions with ordinary matter. The devices capture a large sample of neutrino interactions that can be transformed into basic images through a process called “reconstruction.” Like a slow-motion replay at a sporting event, these reconstructions can help physicists better understand neutrino behavior.
“They almost look like a picture of the interaction,” said Gabriel Perdue, an associate scientist at Fermilab.
Perdue leads an effort to integrate neural networks into the classification and analysis of detector data. The work could improve the efficiency of some measurements, help physicists understand how certain they can be about their analyses, and lead to new avenues of inquiry.
Teaming up with Patton’s team under a 2016 Director’s Discretionary application on Titan, Fermilab researchers produced a competitive classification network in support of a neutrino scattering experiment called MINERvA (Main Injector Experiment for v-A). The task, known as vertex reconstruction, required a network to analyze images and precisely identify the location where neutrinos interact with the detector—a challenge for events that produce many particles.
In only 24 hours, MENNDL produced optimized networks that outperformed handcrafted networks—an achievement that would have taken months for Fermilab researchers. To identify the high-performing network, MENNDL evaluated approximately 500,000 neural networks. The training data consisted of 800,000 images of neutrino events, steadily processed on 18,000 of Titan’s nodes.
“You need something like MENNDL to explore this effectively infinite space of possible networks, but you want to do it efficiently,” Perdue said. “What Titan does is bring the time to solution down to something practical.”
Having recently been awarded another allocation under the Advanced Scientific Computing Research Leadership Computing Challenge program, Perdue’s team is building off its deep learning success by applying MENDDL to additional high-energy physics datasets to generate optimized algorithms. In addition to improved physics measurements, the results could provide insight into how and why machines learn.
“We’re just getting started,” Perdue said. “I think we’ll learn really interesting things about how deep learning works, and we’ll also have better networks to do our physics. The reason we’re going through all this work is because we’re getting better performance, and there’s real potential to get more.”
AI meets exascale
When Titan debuted 5 years ago, its GPU-accelerated architecture boosted traditional modeling and simulation to new levels of detail. Since then, GPUs, which excel at carrying out hundreds of calculations simultaneously, have become the go-to processor for deep learning. That fortuitous development made Titan a powerful tool for exploring artificial intelligence at supercomputer scales.
With the OLCF’s next leadership-class system, Summit, set to come online in 2018, deep learning researchers expect to take this blossoming technology even further. Summit builds on the GPU revolution pioneered by Titan and is expected to deliver more than five times the performance of its predecessor. The IBM system will contain more than 27,000 of Nvidia’s newest Volta GPUs in addition to more than 9,000 IBM Power9 CPUs. Furthermore, because deep learning requires less mathematical precision than other types of scientific computing, Summit could potentially deliver exascale-level performance for deep learning problems—the equivalent of a billion billion calculations per second.
“That means we’ll be able to evaluate larger networks much faster and evolve many more generations of networks in less time,” Young said.
In addition to preparing for new hardware, Patton’s team continues to develop MENNDL and explore other types of experimental techniques, including neuromorphic computing, another biologically inspired computing concept.
“One thing we’re looking at going forward is evolving deep learning networks from stacked layers to graphs of layers that can split and then merge later,” Young said. “These networks with branches excel at analyzing things at multiple scales, such as a closeup photograph in comparison to a wide-angle shot. When you have 20,000 GPUs available, you can actually start to think about a problem like that.”
Related Publication: Steven R. Young, Derek C. Rose, Travis Johnston, William T. Heller, Thomas P. Karnowski, Thomas E. Potok, Robert M. Patton, Gabriel Perdue, and Jonathan Miller, “Evolving Deep Networks Using HPC.” In Proceedings of the Machine Learning on HPC Environments. Paper presented at The International Conference for High Performance Computing, Networking, Storage and Analysis, Denver, Colorado (November 2017), doi: 10.1145/3146347.3146355.
Adam M. Terwilliger, Gabriel N. Perdue, David Isele, Robert M. Patton, and Steven R. Young, “Vertex Reconstruction of Neutrino Interactions Using Deep Learning.” In 2017 International Joint Conference on Neural Networks (IJCNN), IEEE (2017): 2275–2281, doi: 10.1109/IJCNN.2017.7966131.
Oak Ridge National Laboratory is supported by the US Department of Energy’s Office of Science. The single largest supporter of basic research in the physical sciences in the United States, the Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.

MORE NEWS FROM
Oak Ridge National LaboratoryParticipating Labs
- DOE Office of Science
- Argonne National Laboratory
- Oak Ridge National Laboratory
- Pacific Northwest National Laboratory
- Iowa State University, Ames Laboratory
- Brookhaven National Laboratory
- Princeton Plasma Physics Laboratory
- Lawrence Berkeley National Laboratory
- Thomas Jefferson National Accelerator Facility
- Fermi National Accelerator Laboratory (Fermilab)
- SLAC National Accelerator Laboratory

Argonne's debt to 2019 Nobel Prize for lithium-ion battery
A roar of approval rang out at the U.S. Department of Energy's (DOE's) Argonne National Laboratory upon the announcement in October that John B. Goodenough, M. Stanley Whittingham and Akira Yoshino had won the 2019 Nobel Prize in Chemistry. On December 10th in Stockholm, they received this highly coveted prize for their major contributions to the invention of the lithium-ion battery, which is a long-standing major focus of research at Argonne.

Battery collaboration meeting discusses new pathways to recycle lithium-ion batteries
At a conference held by the ReCell Center, an advanced battery recycling collaboration based at Argonne, representatives from industry, government, and academia discussed innovative approaches for lithium-ion battery recycling.

New Function for Plant Enzyme Could Lead to Green Chemistry
Scientists at Brookhaven National Laboratory have discovered a new function in a plant enzyme that could inspire the design of new chemical catalysts. The enzyme catalyzes, or initiates, one of the cornerstone chemical reactions needed to synthesize a wide array of organic molecules, including those found in lubricants, cosmetics, and those used as raw materials for making plastics.

Freeze Frame: Scientists Capture Atomic-Scale Snapshots of Artificial Proteins
Scientists at Berkeley Lab are the first to use cryo-EM (cryogenic electron microscopy), a Nobel Prize-winning technique originally designed to image proteins in solution, to image atomic changes in a synthetic soft material.

Argonne Collaboration Shows Benefits of Better Corn Residue Management Strategies
Sustainable corn stover removal can maintain soil carbon stock, according a new Argonne-led study.

Study Sheds Light on the Really Peculiar 'Normal' Phase of High-Temperature Superconductors
Experiments at SLAC and Stanford probe the normal state more accurately than ever before and discover an abrupt shift in the behavior of electrons in which they suddenly give up their individuality and behave like an electron soup.

Scientists devise catalyst that uses light to turn carbon dioxide to fuel
In a recent study from Argonne, scientists have used sunlight and a catalyst largely made of copper to transform carbon dioxide to methanol.

Science Snapshots - microbiome matchmakers, solid-liquid interfaces, undersea earthquakes
Science Snapshots from Berkeley Lab

SLAC scientists invent a way to see attosecond electron motions with an X-ray laser
Researchers at the Department of Energy's SLAC National Accelerator Laboratory have invented a way to observe the movements of electrons with powerful X-ray laser bursts just 280 attoseconds, or billionths of a billionth of a second, long.

Bank on it: Gains in one type of force produced by fusion disruptions are offset by losses in another
Simulations show that halo currents can serve as a proxy for the total force produced by vertical disruptions.

James Wilson Clark, PPPL's first deputy director for administrative operations, was a decorated World War II veteran, experienced federal administrator, and active member of the Princeton community
James W. Clark, PPPL's first deputy director for administrative operations, was a decorated World War II veteran with a long career in public service, who died Aug. 6. A memorial service in his honor will be held Dec. 21.
Department of Energy to Provide $24 Million in EPSCoR Grants for Energy-Related Research
The U.S. Department of Energy (DOE) announced a funding opportunity for up to $24 million for new grants under the Established Program to Stimulate Competitive Research (DOE EPSCoR).

University of Kentucky Grant Seeks to Turn Coal Into Carbon Fiber
UK's Center for Applied Energy Research (CAER) has received a $1.8 million U.S. Department of Energy (DOE) grant to transform coal tar pitch into high-value carbon fiber for use in aircraft, automobiles, sporting goods and other high-performance materials.

Six Berkeley Lab Scientists Named AAAS Fellows
Six scientists from the Department of Energy's Lawrence Berkeley National Laboratory (Berkeley Lab) have been named Fellows of the American Association for the Advancement of Science (AAAS).

PPPL is recognized for being green
The U.S. Department of Energy's Princeton Plasma Physics Laboratory was recognized by the U.S. Environmental Protection Agency for its green practices in reducing waste, energy, and water, and transportation, and for green purchasing and electronics recycling.

Dmitri Zakharov Recognized with the 2019 Chuck Fiori Award
The award honors Dmitri Zakharov's contributions to environmental transmission electron microscopy at Brookhaven Lab's Center for Functional Nanomaterials.

Two Argonne projects earn Secretary of Energy Honor Awards
With this year's Nobel Prize in Chemistry awarded for the development of lithium-ion batteries, directors of the Joint Center for Energy Storage Research share perspectives on the future of energy storage.

Argonne teams up with Altair to manage use of upcoming Aurora supercomputer
Argonne National Laboratory and Altair, a global technology company, have created a new scheduling system that will be employed on the Aurora supercomputer.

University of Maryland, Baltimore County wins DOE's 2019 CyberForce Competition(tm)
After a long suspenseful day, University of Maryland, Baltimore County earned the top spot as national winner of the U.S. Department of Energy's CyberForce Competition.

In its 15th year, INCITE advances open science with supercomputer grants to 47 projects
The U.S. Department of Energy's Office of Science announced allocations of supercomputer access to 47 science projects for 2020--awarding 60 percent of the available time on some of the nation's most powerful supercomputers, with the ultimate goal of accelerating discovery and innovation. In 2020, 14 projects will run on Theta and 39 projects on Summit, where six of these projects will receive an allocation on both systems.

Harvesting Energy from Light using Bio-inspired Artificial Cells
Scientists designed and connected two different artificial cells to each other to produce molecules called ATP (adenosine triphosphate).

Engineering Living Scaffolds for Building Materials
Bone and mollusk shells are composite systems that combine living cells and inorganic components. This allows them to regenerate and change structure while also being very strong and durable. Borrowing from this amazing complexity, researchers have been exploring a new class of materials called engineered living materials (ELMs).

Excavating Quantum Information Buried in Noise
Researchers developed two new methods to assess and remove error in how scientists measure quantum systems. By reducing quantum "noise" - uncertainty inherent to quantum processes - these new methods improve accuracy and precision.

How Electrons Move in a Catastrophe
Lanthanum strontium manganite (LSMO) is a widely applicable material, from magnetic tunnel junctions to solid oxide fuel cells. However, when it gets thin, its behavior changes for the worse. The reason why was not known. Now, using two theoretical methods, a team determined what happens.

When Ions and Molecules Cluster
How an ion behaves when isolated within an analytical instrument can differ from how it behaves in the environment. Now, Xue-Bin Wang at Pacific Northwest National Laboratory devised a way to bring ions and molecules together in clusters to better discover their properties and predict their behavior.

Tune in to Tetrahedral Superstructures
Shape affects how the particles fit together and, in turn, the resulting material. For the first time, a team observed the self-assembly of nanoparticles with tetrahedral shapes.

Tracing Interstellar Dust Back to the Solar System's Formation
This study is the first to confirm dust particles pre-dating the formation of our solar system. Further study of these materials will enable a deeper understanding of the processes that formed and have since altered them.

Investigating Materials that Can Go the Distance in Fusion Reactors
Future fusion reactors will require materials that can withstand extreme operating conditions, including being bombarded by high-energy neutrons at high temperatures. Scientists recently irradiated titanium diboride (TiB2) in the High Flux Isotope Reactor (HFIR) to better understand the effects of fusion neutrons on performance.

Better 3-D Imaging of Tumors in the Breast with Less Radiation
In breast cancer screening, an imaging technique based on nuclear medicine is currently being used as a successful secondary screening tool alongside mammography to improve the accuracy of the diagnosis. Now, a team is hoping to improve this imaging technique.

Microbes are Metabolic Specialists
Scientists can use genetic information to measure if microbes in the environment can perform specific ecological roles. Researchers recently analyzed the genomes of over 6,000 microbial species.
Spotlight

Barbara Garcia: A first-generation college student spends summer doing research at PPPL
Princeton Plasma Physics Laboratory

Argonne organization's scholarship fund blazes STEM pathway
Argonne National Laboratory

Brookhaven Lab, Suffolk Girl Scouts Launch Patch Program
Brookhaven National Laboratory

From an acoustic levitator to a "Neutron Bloodhound" robot, hands-on research inspires PPPL's summer interns
Princeton Plasma Physics Laboratory

Brookhaven Lab Celebrates the Bright Future of its 2019 Interns
Brookhaven National Laboratory

PPPL apprenticeship program offers young people chance to earn while they learn high-tech careers
Princeton Plasma Physics Laboratory

JSA Awards Graduate Fellowships for Research at Jefferson Lab
Thomas Jefferson National Accelerator Facility

ILSAMP Symposium showcases benefits for diverse students, STEM pipeline
Argonne National Laboratory

Integrating Scientific Computing into Science Curricula
Brookhaven National Laboratory
Students from Minnesota and Massachusetts Win DOE's 29th National Science Bowl(r)
Department of Energy, Office of Science
DOE's Science Graduate Student Research Program Selects 70 Students to Pursue Research at DOE Laboratories
Department of Energy, Office of Science

Young Women's Conference in STEM seeks to change the statistics one girl at a time
Princeton Plasma Physics Laboratory

Students team with Argonne scientists and engineers to learn about STEM careers
Argonne National Laboratory

Lynbrook High wins 2019 SLAC Regional Science Bowl competition
SLAC National Accelerator Laboratory

Equipping the next generation for a technological revolution
Argonne National Laboratory

Chemistry intern inspired by Argonne's real-world science
Argonne National Laboratory

Argonne intern streamlines the beamline
Argonne National Laboratory

Research on Light-Matter Interaction Could Lead to Improved Electronic and Optoelectronic Devices
Rensselaer Polytechnic Institute (RPI)

Innovating Our Energy Future
Oregon State University, College of Engineering

Physics graduate student takes her thesis research to a Department of Energy national lab
University of Alabama at Birmingham

"Model" students enjoy Argonne campus life
Argonne National Laboratory

Writing Code for a More Skilled and Diverse STEM Workforce
Brookhaven National Laboratory

New graduate student summer school launches at Princeton Plasma Physics Laboratory
Princeton Plasma Physics Laboratory

The Gridlock State
California State University (CSU) Chancellor's Office

Meet Jasmine Hatcher and Trishelle Copeland-Johnson
Brookhaven National Laboratory

Argonne hosts Modeling, Experimentation and Validation Summer School
Argonne National Laboratory

Undergraduate Students Extoll Benefits of National Laboratory Research Internships in Fusion and Plasma Science
Princeton Plasma Physics Laboratory

Students affected by Hurricane Maria bring their research to SLAC
SLAC National Accelerator Laboratory

Brookhaven Lab Pays Tribute to 2018 Summer Interns
Brookhaven National Laboratory

CSUMB Selected to Host Architecture at Zero Competition in 2019
California State University, Monterey Bay

From Hurricane Katrina Victim to Presidential Awardee: A SUNO Professor's Award-Winning Mentoring Efforts
Brookhaven National Laboratory

Department of Energy Invests $64 Million in Advanced Nuclear Technology
Rensselaer Polytechnic Institute (RPI)

Professor Miao Yu Named the Priti and Mukesh Chatter '82 Career Development Professor
Rensselaer Polytechnic Institute (RPI)

2018 RHIC & AGS Annual Users' Meeting: 'Illuminating the QCD Landscape'
Brookhaven National Laboratory

High-School Students Studying Carbon-Based Nanomaterials for Cancer Drug Delivery Visit Brookhaven Lab's Nanocenter
Brookhaven National Laboratory

Argonne welcomes The Martian author Andy Weir
Argonne National Laboratory

UW Professor and Clean Energy Institute Director Daniel Schwartz Wins Highest U.S. Award for STEM Mentors
University of Washington

Creating STEM Knowledge and Innovations to Solve Global Issues Like Water, Food, and Energy
Illinois Mathematics and Science Academy (IMSA)

Professor Emily Liu Receives $1.8 Million DoE Award for Solar Power Systems Research
Rensselaer Polytechnic Institute (RPI)

Celebrating 40 years of empowerment in science
Argonne National Laboratory

Introducing Graduate Students Across the Globe to Photon Science
Brookhaven National Laboratory

Students from Massachusetts and Washington Win DOE's 28th National Science Bowl(r)
Department of Energy, Office of Science

Q&A: Al Ashley Reflects on His Efforts to Diversify SLAC and Beyond
SLAC National Accelerator Laboratory

Insights on Innovation in Energy, Humanitarian Aid Highlight UVA Darden's Net Impact Week
University of Virginia Darden School of Business

Ivy League Graduate, Writer and Activist with Dyslexia Visits CSUCI to Reframe the Concept of Learning Disabilities
California State University, Channel Islands

Photographer Adam Nadel Selected as Fermilab's New Artist-in-Residence for 2018
Fermi National Accelerator Laboratory (Fermilab)
Showing results
0-4 Of 2215