- 2017-11-29 11:05:27
- Article ID: 685892
Scaling Deep Learning for Science
ORNL-designed algorithm leverages Titan to create high-performing deep neural networks
Now, researchers are eager to apply this computational technique—commonly referred to as deep learning—to some of science’s most persistent mysteries. But because scientific data often looks much different from the data used for animal photos and speech, developing the right artificial neural network can feel like an impossible guessing game for nonexperts. To expand the benefits of deep learning for science, researchers need new tools to build high-performing neural networks that don’t require specialized knowledge.
Using the Titan supercomputer, a research team led by Robert Patton of the US Department of Energy’s(DOE’s) Oak Ridge National Laboratory (ORNL) has developed an evolutionary algorithm capable of generating custom neural networks that match or exceed the performance of handcrafted artificial intelligence systems. Better yet, by leveraging the GPU computing power of the Cray XK7 Titan—the leadership-class machine managed by the Oak Ridge Leadership Computing Facility, a DOE Office of Science User Facility at ORNL—these auto-generated networks can be produced quickly, in a matter of hours as opposed to the months needed using conventional methods.
The research team’s algorithm, called MENNDL (Multinode Evolutionary Neural Networks for Deep Learning), is designed to evaluate, evolve, and optimize neural networks for unique datasets. Scaled across Titan’s 18,688 GPUs, MENNDL can test and train thousands of potential networks for a science problem simultaneously, eliminating poor performers and averaging high performers until an optimal network emerges. The process eliminates much of the time-intensive, trial-and-error tuning traditionally required of machine learning experts.
“There’s no clear set of instructions scientists can follow to tweak networks to work for their problem,” said research scientist Steven Young, a member of ORNL’s Nature Inspired Machine Learning team. “With MENNDL, they no longer have to worry about designing a network. Instead, the algorithm can quickly do that for them, while they focus on their data and ensuring the problem is well-posed.”
Pinning down parameters
Inspired by the brain’s web of neurons, deep neural networks are a relatively old concept in neuroscience and computing, first popularized by two University of Chicago researchers in the 1940s. But because of limits in computing power, it wasn’t until recently that researchers had success in training machines to independently interpret data.
Today’s neural networks can consist of thousands or millions of simple computational units—the “neurons”—arranged in stacked layers, like the rows of figures spaced across a foosball table. During one common form of training, a network is assigned a task (e.g., to find photos with cats) and fed a set of labeled data (e.g., photos of cats and photos without cats). As the network pushes the data through each successive layer, it makes correlations between visual patterns and predefined labels, assigning values to specific features (e.g., whiskers and paws). These values contribute to the weights that define the network’s model parameters. During training, the weights are continually adjusted until the final output matches the targeted goal. Once the network learns to perform from training data, it can then be tested against unlabeled data.
Although many parameters of a neural network are determined during the training process, initial model configurations must be set manually. These starting points, known as hyperparameters, include variables like the order, type, and number of layers in a network.
Finding the optimal set of hyperparameters can be the key to efficiently applying deep learning to an unusual dataset. “You have to experimentally adjust these parameters because there’s no book you can look in and say, ‘These are exactly what your hyperparameters should be,’” Young said. “What we did is use this evolutionary algorithm on Titan to find the best hyperparameters for varying types of datasets.”
Unlocking that potential, however, required some creative software engineering by Patton’s team. MENNDL homes in on a neural network’s optimal hyperparameters by assigning a neural network to each Titan node. The team designed MENNDL to use a deep learning framework called Caffe to carry out the computation, relying on the parallel computing Message Passing Interface standard to divide and distribute data among nodes. As Titan works through individual networks, new data is fed to the system’s nodes asynchronously, meaning once a node completes a task, it’s quickly assigned a new task independent of the other nodes’ status. This ensures that the 27-petaflop Titan stays busy combing through possible configurations.
“Designing the algorithm to really work at that scale was one of the challenges,” Young said. “To really leverage the machine, we set up MENNDL to generate a queue of individual networks to send to the nodes for evaluation as soon as computing power becomes available.”
To demonstrate MENNDL’s versatility, the team applied the algorithm to several datasets, training networks to identify sub-cellular structures for medical research, classify satellite images with clouds, and categorize high-energy physics data. The results matched or exceeded the performance of networks designed by experts.
Networking neutrinos
One science domain in which MENNDL is already proving its value is neutrino physics. Neutrinos, ghost-like particles that pass through your body at a rate of trillions per second, could play a major role in explaining the formation of the early universe and the nature of matter—if only scientists knew more about them.
Large detectors at DOE’s Fermi National Accelerator Laboratory (Fermilab) use high-intensity beams to study elusive neutrino reactions with ordinary matter. The devices capture a large sample of neutrino interactions that can be transformed into basic images through a process called “reconstruction.” Like a slow-motion replay at a sporting event, these reconstructions can help physicists better understand neutrino behavior.
“They almost look like a picture of the interaction,” said Gabriel Perdue, an associate scientist at Fermilab.
Perdue leads an effort to integrate neural networks into the classification and analysis of detector data. The work could improve the efficiency of some measurements, help physicists understand how certain they can be about their analyses, and lead to new avenues of inquiry.
Teaming up with Patton’s team under a 2016 Director’s Discretionary application on Titan, Fermilab researchers produced a competitive classification network in support of a neutrino scattering experiment called MINERvA (Main Injector Experiment for v-A). The task, known as vertex reconstruction, required a network to analyze images and precisely identify the location where neutrinos interact with the detector—a challenge for events that produce many particles.
In only 24 hours, MENNDL produced optimized networks that outperformed handcrafted networks—an achievement that would have taken months for Fermilab researchers. To identify the high-performing network, MENNDL evaluated approximately 500,000 neural networks. The training data consisted of 800,000 images of neutrino events, steadily processed on 18,000 of Titan’s nodes.
“You need something like MENNDL to explore this effectively infinite space of possible networks, but you want to do it efficiently,” Perdue said. “What Titan does is bring the time to solution down to something practical.”
Having recently been awarded another allocation under the Advanced Scientific Computing Research Leadership Computing Challenge program, Perdue’s team is building off its deep learning success by applying MENDDL to additional high-energy physics datasets to generate optimized algorithms. In addition to improved physics measurements, the results could provide insight into how and why machines learn.
“We’re just getting started,” Perdue said. “I think we’ll learn really interesting things about how deep learning works, and we’ll also have better networks to do our physics. The reason we’re going through all this work is because we’re getting better performance, and there’s real potential to get more.”
AI meets exascale
When Titan debuted 5 years ago, its GPU-accelerated architecture boosted traditional modeling and simulation to new levels of detail. Since then, GPUs, which excel at carrying out hundreds of calculations simultaneously, have become the go-to processor for deep learning. That fortuitous development made Titan a powerful tool for exploring artificial intelligence at supercomputer scales.
With the OLCF’s next leadership-class system, Summit, set to come online in 2018, deep learning researchers expect to take this blossoming technology even further. Summit builds on the GPU revolution pioneered by Titan and is expected to deliver more than five times the performance of its predecessor. The IBM system will contain more than 27,000 of Nvidia’s newest Volta GPUs in addition to more than 9,000 IBM Power9 CPUs. Furthermore, because deep learning requires less mathematical precision than other types of scientific computing, Summit could potentially deliver exascale-level performance for deep learning problems—the equivalent of a billion billion calculations per second.
“That means we’ll be able to evaluate larger networks much faster and evolve many more generations of networks in less time,” Young said.
In addition to preparing for new hardware, Patton’s team continues to develop MENNDL and explore other types of experimental techniques, including neuromorphic computing, another biologically inspired computing concept.
“One thing we’re looking at going forward is evolving deep learning networks from stacked layers to graphs of layers that can split and then merge later,” Young said. “These networks with branches excel at analyzing things at multiple scales, such as a closeup photograph in comparison to a wide-angle shot. When you have 20,000 GPUs available, you can actually start to think about a problem like that.”
Related Publication: Steven R. Young, Derek C. Rose, Travis Johnston, William T. Heller, Thomas P. Karnowski, Thomas E. Potok, Robert M. Patton, Gabriel Perdue, and Jonathan Miller, “Evolving Deep Networks Using HPC.” In Proceedings of the Machine Learning on HPC Environments. Paper presented at The International Conference for High Performance Computing, Networking, Storage and Analysis, Denver, Colorado (November 2017), doi: 10.1145/3146347.3146355.
Adam M. Terwilliger, Gabriel N. Perdue, David Isele, Robert M. Patton, and Steven R. Young, “Vertex Reconstruction of Neutrino Interactions Using Deep Learning.” In 2017 International Joint Conference on Neural Networks (IJCNN), IEEE (2017): 2275–2281, doi: 10.1109/IJCNN.2017.7966131.
Oak Ridge National Laboratory is supported by the US Department of Energy’s Office of Science. The single largest supporter of basic research in the physical sciences in the United States, the Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.
Participating Labs
- DOE Office of Science
- Argonne National Laboratory
- Oak Ridge National Laboratory
- Pacific Northwest National Laboratory
- Iowa State University, Ames Laboratory
- Brookhaven National Laboratory
- Princeton Plasma Physics Laboratory
- Lawrence Berkeley National Laboratory
- Thomas Jefferson National Accelerator Facility
- Fermi National Accelerator Laboratory (Fermilab)
- SLAC National Accelerator Laboratory

A Game Changer: Protein Clustering Powered by Supercomputers
New algorithm lets biologists harness massively parallel supercomputers to make sense of a protein "data deluge."

LLNL Maps Out Deployment of Carbon Capture and Sequestration for Ethanol Production
To better understand the near-term commercial potential for capturing and storing atmospheric carbon dioxide (CO2), researchers from Lawrence Livermore National Laboratory have mapped out how CO2 might be captured from existing U.S. ethanol biorefineries and permanently stored (or sequestered) underground.

Neutrons Provide Insights into Increased Performance for Hybrid Perovskite Solar Cells
Neutron scattering at Oak Ridge National Laboratory has revealed, in real time, the fundamental mechanisms behind the conversion of sunlight into energy in hybrid perovskite materials. A better understanding of this behavior will enable manufacturers to design solar cells with significantly increased efficiency.

Liquid Cell Transmission Electron Microscopy Makes a Window Into the Nanoscale
From energy materials to disease diagnostics, new microscopy techniques can provide more nuanced insight. Researchers first need to understand the effects of radiation on samples, which is possible with a new device that holds tightly sealed liquid cell samples for transmission electron microscopy.

Nanoparticle Breakthrough Could Capture Unseen Light for Solar Energy Conversion
An international team, led by Berkeley Lab scientists, has demonstrated a breakthrough in the design and function of nanoparticles that could make solar panels more efficient by converting light usually missed by solar cells into usable energy.

New Testing of Model Improves Confidence in the Performance of ITER
Article describes effect of ion and electron heating on multiscale turbulence in fusion plasmas.

Study Recommends Strong Role for National Labs in 'Second Laser Revolution'
A new study calls for the U.S. to step up its laser R&D efforts to better compete with major overseas efforts to build large, high-power laser systems, and notes progress and milestones at the Department of Energy's Berkeley Lab Laser Accelerator (BELLA) Center and other sites.

Wood Formation Model To Fuel Progress in Bioenergy, Paper, New Applications
Need stronger timber, better biofuel or new sources of green chemicals? A systems biology model built on decades of NC State research will accelerate progress on engineering trees for specific needs.

Researchers Achieve HD Video Streaming at 10,000 Times Lower Power
Engineers at the University of Washington have developed a new HD video streaming method that doesn't need to be plugged in. Their prototype skips the power-hungry components and has something else, like a smartphone, process the video instead.

Lawrence Livermore Issues Combined State-by-State Energy and Water Use Flow Charts
For the first time, Lawrence Livermore National Laboratory (LLNL) has issued state-by-state energy and water flow charts in one location so that analysts and policymakers can find all the information they need in one place.
Five Leading Liberal Arts Colleges Partner to Create New Solar Energy Facility in Maine
Amherst, Bowdoin, Hampshire, Smith and Williams colleges have formed a partnership that will allow them to offset 46,000 megawatt hours per year of their collective electrical needs--enough to power 5,000 New England homes--with electricity created at a solar power facility to be built in Maine.

Argonne Selects Innovators From Across Nation to Grow Startups
Argonne announces second cohort of Chain Reaction Innovations.

Brookhaven Lab Materials Physicist Yimei Zhu Receives 2018 Distinguished Scientist Award from the Microscopy Society of America
How do complex atomic and electronic interactions impact material properties? Using electron microscopy instrumentation and methods he developed, Yimei Zhu has been investigating this question for the past 30 years. The Microscopy Society of America is now recognizing his contributions.

SLAC Produces First Electron Beam with Superconducting Electron Gun
Accelerator scientists at the Department of Energy's SLAC National Accelerator Laboratory are testing a new type of electron gun for a future generation of instruments that take snapshots of the atomic world in never-before-seen quality and detail, with applications in chemistry, biology, energy and materials science.

U.S., India Sign Agreement Providing for Neutrino Physics Collaboration at Fermilab and in India
Earlier today, April 16, 2018, U.S. Secretary of Energy Rick Perry and India's Atomic Energy Secretary Dr. Sekhar Basu signed an agreement in New Delhi to expand the two countries' collaboration on world-leading science and technology projects. It opens the way for jointly advancing cutting-edge neutrino science projects under way in both countries: the Long-Baseline Neutrino Facility (LBNF) with the international Deep Underground Neutrino Experiment (DUNE) hosted at the U.S. Department of Energy's Fermilab and the India-based Neutrino Observatory (INO).

Nanomaterials Expert Ganpati Ramanath Named Fellow of Materials Research Society
Nanomaterials expert Ganpati Ramanath, the John Tod Horton '52 Professor of Materials Science and Engineering at Rensselaer Polytechnic Institute, has been named a fellow of the Materials Research Society (MRS) "for developing creative approaches to realize new nanomaterials via chemically directed nanostructure synthesis and assembly and for tailoring interfaces in electronics and energy applications using molecular nanolayers."

Doing the Neutron Dance
Two materials scientists, Suzanne te Velthuis and Stephan Rosenkranz, have been named fellows of the Neutron Scattering Society of America (NSSA).

Hirohisa Tanaka Joins SLAC to Push Limits of Neutrino Physics
Accomplished neutrino physicist Hirohisa Tanaka has joined the Department of Energy's SLAC National Accelerator Laboratory as a professor of particle physics and astrophysics. He oversees a group at the lab that is preparing for research with the future Deep Underground Neutrino Experiment (DUNE) at the Long-Baseline Neutrino Facility (LBNF). The experiment will give scientists unprecedented opportunities to learn more about neutrinos - fundamental particles with mysterious properties that could play crucial roles in the evolution of the universe.

William Tang Wins 2018 Global Impact Award to Advance Development of Ai Software to Help Create "a Star on Earth"
Article announces William Tang's NVIDIA award.

University Teams to Compete in Department of Energy's 2018 National Cyber Defense Competition
The U.S. Department of Energy is proud to announce the 29 university teams selected to compete in the third annual Cyber Defense Competition (CDC), taking place April 6-7, 2018.

A Game Changer: Protein Clustering Powered by Supercomputers
New algorithm lets biologists harness massively parallel supercomputers to make sense of a protein "data deluge."

Getting Magnesium Ions to Pick Up the Pace
Magnesium ions move very fast to enable a new class of battery materials.

Seeing How Next-Generation Batteries Power-Up
Scientists directly see how the atoms in a magnesium-based battery fit into the structure of electrodes.

Worm-Inspired Tough Materials
Scientists mimic a worm's lethal jaw to design and form resilient materials.

How to Turn Light Into Atomic Vibrations
Converting laser light into nuclear vibrations is key to switching a material's properties on and off for future electronics.

Superacids Are Good Medicine for Super Thin Semiconductors
Scientists demonstrated that powerful acids heal certain structural defects in synthetic films.

Tubular Science Improves Polymer Solar Cells
Novel engineered polymers assemble buckyballs into columns using a conventional coating process.

Fast! Hard X-Ray Flash Breaks Speed Record
Lasting just a few hundred billionths of a billionth of a second, these bursts offer new tool to study chemistry and magnetism.

Scientists Have Overestimated Meteor Sizes
First demonstration of high-pressure metastability mapping with ultrafast X-ray diffraction shows objects aren't as large as previously thought.

Rewriting Resistance: Genetic Changes Increase Crops' Biomass and Sugar Release
Using genetic engineering, scientists improve biomass growth and conversion in woody and grassy feedstocks.
Spotlight

Q&A: Al Ashley Reflects on His Efforts to Diversify SLAC and Beyond
SLAC National Accelerator Laboratory

Insights on Innovation in Energy, Humanitarian Aid Highlight UVA Darden's Net Impact Week
University of Virginia Darden School of Business

Ivy League Graduate, Writer and Activist with Dyslexia Visits CSUCI to Reframe the Concept of Learning Disabilities
California State University, Channel Islands

Photographer Adam Nadel Selected as Fermilab's New Artist-in-Residence for 2018
Fermi National Accelerator Laboratory (Fermilab)

Fermilab Computing Partners with Argonne, Local Schools for Hour of Code
Fermi National Accelerator Laboratory (Fermilab)

Q&A: Sam Webb Teaches X-Ray Science from a Remote Classroom
SLAC National Accelerator Laboratory

The Future of Today's Electric Power Systems
Rensselaer Polytechnic Institute (RPI)

Supporting the Development of Offshore Wind Power Plants
Rensselaer Polytechnic Institute (RPI)

Bringing Diversity Into Computational Science Through Student Outreach
Brookhaven National Laboratory

From Science to Finance: SLAC Summer Interns Forge New Paths in STEM
SLAC National Accelerator Laboratory

Students Discuss 'Cosmic Opportunities' at 45th Annual SLAC Summer Institute
SLAC National Accelerator Laboratory

Binghamton University Opens $70 Million Smart Energy Building
Binghamton University, State University of New York

Widening Horizons for High Schoolers with Code
Argonne National Laboratory

Rensselaer Polytechnic Institute Graduates Urged to Embrace Change at 211th Commencement
Rensselaer Polytechnic Institute (RPI)

Rensselaer Polytechnic Institute President's Commencement Colloquy to Address "Criticality, Incisiveness, Creativity"
Rensselaer Polytechnic Institute (RPI)

ORNL, University of Tennessee Launch New Doctoral Program in Data Science
Oak Ridge National Laboratory

Champions in Science: Profile of Jonathan Kirzner
Department of Energy, Office of Science

High-Schooler Solves College-Level Security Puzzle From Argonne, Sparks Interest in Career
Argonne National Laboratory

Champions in Science: Profile of Jenica Jacobi
Department of Energy, Office of Science

Great Neck South High School Wins Regional Science Bowl at Brookhaven Lab
Brookhaven National Laboratory

Middle Schoolers Test Their Knowledge at Science Bowl Competition
Argonne National Laboratory

Haslam Visits ORNL to Highlight State's Role in Discovering Tennessine
Oak Ridge National Laboratory

Internship Program Helps Foster Development of Future Nuclear Scientists
Oak Ridge National Laboratory

More Than 12,000 Explore Jefferson Lab During April 30 Open House
Thomas Jefferson National Accelerator Facility

NMSU Undergrad Tackles 3D Particle Scattering Animations After Receiving JSA Research Assistantship
Thomas Jefferson National Accelerator Facility

Shannon Greco: A Self-Described "STEM Education Zealot"
Princeton Plasma Physics Laboratory

Rare Earths for Life: An 85th Birthday Visit with Mr. Rare Earth
Ames Laboratory

Meet Robert Palomino: 'Give Everything a Shot!'
Brookhaven National Laboratory

Student Innovator at Rensselaer Polytechnic Institute Seeks Brighter, Smarter, and More Efficient LEDs
Rensselaer Polytechnic Institute (RPI)

Texas Tech Energy Commerce Students, Community Light up Tent City
Texas Tech University

Don't Get 'Frosted' Over Heating Your Home This Winter
Temple University

New Research Center To Tackle Critical Challenges Related to Aircraft Design, Wind Energy, Smart Buildings
Rensselaer Polytechnic Institute (RPI)

First Polymer Solar-Thermal Device Heats Home, Saves Money
Wake Forest University

Like Superman, American University Will Get Its Energy from the Sun
American University

ARRA Grant to Help Fund Seminary Building Green Roof
University of Chicago

Ithaca College in Elite Company for Environmental Leadership in Building Construction
Ithaca College

UC San Diego Installing 2.8 Megawatt Fuel Cell to Anchor Energy Innovation Park
University of California San Diego

Rensselaer Smart Lighting Engineering Research Center Announces First Deployment of New Technology on Campus
Rensselaer Polytechnic Institute (RPI)
Ithaca College Will Host Regional Clean Energy Summit
Ithaca College

Texas Governor Announces $8.4 Million Award to Create Renewable Energy Institute
Texas Tech University

Creighton University to Offer New Alternative Energy Program
Creighton University
National Engineering Program Seeks Subject Matter Experts in Energy
JETS Junior Engineering Technical Society

Students Using Solar Power To Create Sustainable Solutions for Haiti, Peru
Rensselaer Polytechnic Institute (RPI)

Helping Hydrogen: Student Inventor Tackles Challenge of Hydrogen Storage
Rensselaer Polytechnic Institute (RPI)
Showing results
0-4 Of 2215