Doe Science news source
The DOE Science News Source is a Newswise initiative to promote research news from the Office of Science of the DOE to the public and news media.
  • 2017-11-29 11:05:27
  • Article ID: 685892

Scaling Deep Learning for Science

ORNL-designed algorithm leverages Titan to create high-performing deep neural networks

  • Credit: iStock

    Inspired by the brain’s web of neurons, deep neural networks consist of thousands or millions of simple computational units. Leveraging the GPU computing power of the Cray XK7 Titan, ORNL researchers were able to auto-generate custom neural networks for science problems in a matter of hours as opposed to the months needed using conventional methods.

Deep neural networks—a form of artificial intelligence—have demonstrated mastery of tasks once thought uniquely human. Their triumphs have ranged from identifying animals in images, to recognizing human speech, to winning complex strategy games, among other successes.

Now, researchers are eager to apply this computational technique—commonly referred to as deep learning—to some of science’s most persistent mysteries. But because scientific data often looks much different from the data used for animal photos and speech, developing the right artificial neural network can feel like an impossible guessing game for nonexperts. To expand the benefits of deep learning for science, researchers need new tools to build high-performing neural networks that don’t require specialized knowledge.

Using the Titan supercomputer, a research team led by Robert Patton of the US Department of Energy’s(DOE’s) Oak Ridge National Laboratory (ORNL) has developed an evolutionary algorithm capable of generating custom neural networks that match or exceed the performance of handcrafted artificial intelligence systems. Better yet, by leveraging the GPU computing power of the Cray XK7 Titan—the leadership-class machine managed by the Oak Ridge Leadership Computing Facility, a DOE Office of Science User Facility at ORNL—these auto-generated networks can be produced quickly, in a matter of hours as opposed to the months needed using conventional methods.

The research team’s algorithm, called MENNDL (Multinode Evolutionary Neural Networks for Deep Learning), is designed to evaluate, evolve, and optimize neural networks for unique datasets. Scaled across Titan’s 18,688 GPUs, MENNDL can test and train thousands of potential networks for a science problem simultaneously, eliminating poor performers and averaging high performers until an optimal network emerges. The process eliminates much of the time-intensive, trial-and-error tuning traditionally required of machine learning experts.

“There’s no clear set of instructions scientists can follow to tweak networks to work for their problem,” said research scientist Steven Young, a member of ORNL’s Nature Inspired Machine Learning team. “With MENNDL, they no longer have to worry about designing a network. Instead, the algorithm can quickly do that for them, while they focus on their data and ensuring the problem is well-posed.”

Pinning down parameters

Inspired by the brain’s web of neurons, deep neural networks are a relatively old concept in neuroscience and computing, first popularized by two University of Chicago researchers in the 1940s. But because of limits in computing power, it wasn’t until recently that researchers had success in training machines to independently interpret data.

Today’s neural networks can consist of thousands or millions of simple computational units—the “neurons”—arranged in stacked layers, like the rows of figures spaced across a foosball table. During one common form of training, a network is assigned a task (e.g., to find photos with cats) and fed a set of labeled data (e.g., photos of cats and photos without cats). As the network pushes the data through each successive layer, it makes correlations between visual patterns and predefined labels, assigning values to specific features (e.g., whiskers and paws). These values contribute to the weights that define the network’s model parameters. During training, the weights are continually adjusted until the final output matches the targeted goal. Once the network learns to perform from training data, it can then be tested against unlabeled data.

Although many parameters of a neural network are determined during the training process, initial model configurations must be set manually. These starting points, known as hyperparameters, include variables like the order, type, and number of layers in a network.

Finding the optimal set of hyperparameters can be the key to efficiently applying deep learning to an unusual dataset. “You have to experimentally adjust these parameters because there’s no book you can look in and say, ‘These are exactly what your hyperparameters should be,’” Young said. “What we did is use this evolutionary algorithm on Titan to find the best hyperparameters for varying types of datasets.”

Unlocking that potential, however, required some creative software engineering by Patton’s team. MENNDL homes in on a neural network’s optimal hyperparameters by assigning a neural network to each Titan node. The team designed MENNDL to use a deep learning framework called Caffe to carry out the computation, relying on the parallel computing Message Passing Interface standard to divide and distribute data among nodes. As Titan works through individual networks, new data is fed to the system’s nodes asynchronously, meaning once a node completes a task, it’s quickly assigned a new task independent of the other nodes’ status. This ensures that the 27-petaflop Titan stays busy combing through possible configurations.

“Designing the algorithm to really work at that scale was one of the challenges,” Young said. “To really leverage the machine, we set up MENNDL to generate a queue of individual networks to send to the nodes for evaluation as soon as computing power becomes available.”

To demonstrate MENNDL’s versatility, the team applied the algorithm to several datasets, training networks to identify sub-cellular structures for medical research, classify satellite images with clouds, and categorize high-energy physics data. The results matched or exceeded the performance of networks designed by experts.

Networking neutrinos

One science domain in which MENNDL is already proving its value is neutrino physics. Neutrinos, ghost-like particles that pass through your body at a rate of trillions per second, could play a major role in explaining the formation of the early universe and the nature of matter—if only scientists knew more about them.

Large detectors at DOE’s Fermi National Accelerator Laboratory (Fermilab) use high-intensity beams to study elusive neutrino reactions with ordinary matter. The devices capture a large sample of neutrino interactions that can be transformed into basic images through a process called “reconstruction.” Like a slow-motion replay at a sporting event, these reconstructions can help physicists better understand neutrino behavior.

“They almost look like a picture of the interaction,” said Gabriel Perdue, an associate scientist at Fermilab.

Perdue leads an effort to integrate neural networks into the classification and analysis of detector data. The work could improve the efficiency of some measurements, help physicists understand how certain they can be about their analyses, and lead to new avenues of inquiry.

Teaming up with Patton’s team under a 2016 Director’s Discretionary application on Titan, Fermilab researchers produced a competitive classification network in support of a neutrino scattering experiment called MINERvA (Main Injector Experiment for v-A). The task, known as vertex reconstruction, required a network to analyze images and precisely identify the location where neutrinos interact with the detector—a challenge for events that produce many particles.

In only 24 hours, MENNDL produced optimized networks that outperformed handcrafted networks—an achievement that would have taken months for Fermilab researchers. To identify the high-performing network, MENNDL evaluated approximately 500,000 neural networks. The training data consisted of 800,000 images of neutrino events, steadily processed on 18,000 of Titan’s nodes.

“You need something like MENNDL to explore this effectively infinite space of possible networks, but you want to do it efficiently,” Perdue said. “What Titan does is bring the time to solution down to something practical.”

Having recently been awarded another allocation under the Advanced Scientific Computing Research Leadership Computing Challenge program, Perdue’s team is building off its deep learning success by applying MENDDL to additional high-energy physics datasets to generate optimized algorithms. In addition to improved physics measurements, the results could provide insight into how and why machines learn.

“We’re just getting started,” Perdue said. “I think we’ll learn really interesting things about how deep learning works, and we’ll also have better networks to do our physics. The reason we’re going through all this work is because we’re getting better performance, and there’s real potential to get more.”

AI meets exascale

When Titan debuted 5 years ago, its GPU-accelerated architecture boosted traditional modeling and simulation to new levels of detail. Since then, GPUs, which excel at carrying out hundreds of calculations simultaneously, have become the go-to processor for deep learning. That fortuitous development made Titan a powerful tool for exploring artificial intelligence at supercomputer scales.

With the OLCF’s next leadership-class system, Summit, set to come online in 2018, deep learning researchers expect to take this blossoming technology even further. Summit builds on the GPU revolution pioneered by Titan and is expected to deliver more than five times the performance of its predecessor. The IBM system will contain more than 27,000 of Nvidia’s newest Volta GPUs in addition to more than 9,000 IBM Power9 CPUs. Furthermore, because deep learning requires less mathematical precision than other types of scientific computing, Summit could potentially deliver exascale-level performance for deep learning problems—the equivalent of a billion billion calculations per second.

“That means we’ll be able to evaluate larger networks much faster and evolve many more generations of networks in less time,” Young said.

In addition to preparing for new hardware, Patton’s team continues to develop MENNDL and explore other types of experimental techniques, including neuromorphic computing, another biologically inspired computing concept.

“One thing we’re looking at going forward is evolving deep learning networks from stacked layers to graphs of layers that can split and then merge later,” Young said. “These networks with branches excel at analyzing things at multiple scales, such as a closeup photograph in comparison to a wide-angle shot. When you have 20,000 GPUs available, you can actually start to think about a problem like that.”

Related Publication: Steven R. Young, Derek C. Rose, Travis Johnston, William T. Heller, Thomas P. Karnowski, Thomas E. Potok, Robert M. Patton, Gabriel Perdue, and Jonathan Miller, “Evolving Deep Networks Using HPC.” In Proceedings of the Machine Learning on HPC Environments. Paper presented at The International Conference for High Performance Computing, Networking, Storage and Analysis, Denver, Colorado (November 2017), doi: 10.1145/3146347.3146355.

Adam M. Terwilliger, Gabriel N. Perdue, David Isele, Robert M. Patton, and Steven R. Young, “Vertex Reconstruction of Neutrino Interactions Using Deep Learning.” In 2017 International Joint Conference on Neural Networks (IJCNN), IEEE (2017): 2275–2281, doi: 10.1109/IJCNN.2017.7966131.

Oak Ridge National Laboratory is supported by the US Department of Energy’s Office of Science. The single largest supporter of basic research in the physical sciences in the United States, the Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.

X
X
X
  • Filters

  • × Clear Filters

The Wet Road to Fast and Stable Batteries

An international team of scientists --- including several researchers from the U.S. Department of Energy's (DOE) Argonne National Laboratory -- - has discovered an anode battery material with superfast charging and stable operation over many thousands of cycles.

Light Perfects Interfaces

Shining light on a growing semiconductor modifies its interface with the surface and could improve the optical properties of each.

Advance in Light Filtering Technology Has Implications for LCD Screens, Lasers and Beyond

Vector polarizers are a light filtering technology hidden behind the operation of many optical systems. They can be found, for instance, in sunglasses, LCD screens, microscopes, microprocessors, laser machining and more. Optical physicists published details of their new vector polarizer design this week in APL Photonics. The newly proposed design is a major advance in polarization technology because it enables flexible filtering of a wide range of light sources and generation of new light states.

Accelerating the Self-Assembly of Nanoscale Patterns for Next-Generation Materials

Scientists have come up with a way to massively speed up the ordering process for self-assembling materials. The resulting ultra-small, well-ordered patterns could be used in the fabrication of microelectronics, antireflective surfaces, magnetic data storage systems, and fluid-flow devices.

Beta of Neurodata Without Borders Software Now Available

Neuroscientists can now explore a beta version of the new Neurodata Without Borders: Neurophysiology (NWB:N 2.0) software and offer input to developers before it is fully released next year.

Scientists Discover Path to Improving Game-Changing Battery Electrode

Researchers from Stanford University, two Department of Energy national labs and the battery manufacturer Samsung created a comprehensive picture of how the same chemical processes that give cathodes their high capacity are also linked to changes in atomic structure that sap performance.

ESnet's Petascale DTN Project Speeds up Data Transfers between Leading HPC Centers

A new Petascale Data Transfer Node project aims to to achieve regular disk-to-disk, end-to-end transfer rates of one petabyte per week between major supercomputing facilities, which translates to achievable throughput rates of about 15 Gbps on real world science data sets.

Underappreciated Microbes Now Get Credit for Holding Down Two Jobs in Soil

Soil microbes work as both decomposers and synthesizers of carbon compounds in soil, offering new answers with impacts to crops and eco-health.

Energy, Economy, and the Earth: The Benefits of Creating Feedback Loops

Scientists reduce uncertainties in future climate prediction by directly coupling an energy-economy model to an Earth system model.

How Grasslands Regulate Their Productivity in Response to Droughts

Scientists show that grasslands are more sensitive to changes in the amount of moisture in the air than to changes in precipitation.


  • Filters

  • × Clear Filters

NAU Researchers Join DOE Project to Study the Soil Microbiome and Its Effect on Carbon Persistence

NAU Regents' Professor Bruce Hungate, director of the Center for Ecosystem Science and Society (Ecoss), recently joined a new initiative lead by LLNL to study how the soil microbiome controls the mechanisms that regulate the stabilization of the organic matter in soil.

Four Scientists Win the Los Alamos Medal

Los Alamos National Laboratory will award four former researchers with the Los Alamos Medal for their scientific contributions.

Stewart Prager Honored with FPA Distinguished Career Award

Announcement of Fusion Power Associates career award for Stewart Prager

WVU Physicists Among Collaborators Granted $7 Million to Form U.S. Department of Energy Center of Excellence

Scientists pause each afternoon at Kirtland Air Force Base in Sandia National Laboratories in Albuquerque, New Mexico, awaiting the daily lightning flash and unmistakable floor jolt that accompanies a Z shot

US Dept. Of Energy Grant to Advance Combined Heat and Power Systems in the Midwest

The University of Illinois at Chicago has received a five-year, $4.2 million grant from the U.S. Department of Energy to help industrial, commercial, institutional and utility entities evaluate and install highly efficient combined heat and power (CHP) technologies.CHP, also known as cogeneration, is a single system that produces both thermal energy and electricity.

Applications Open: ECS Toyota Young Investigator Fellowship 2018-2019

ECS, in a continued partnership with the Toyota Research Institute of North America (TRINA), a division of Toyota Motor Engineering & Manufacturing North America, Inc. (TEMA), is requesting proposals from young professors and scholars pursuing innovative electrochemical research in green energy technology.

Successful Startup Founder to Lead Entrepreneurship Program at Argonne

John Carlisle has been named the director of Chain Reaction Innovations (CRI), a program aimed at accelerating job creation through innovation, based at the U.S. Department of Energy's Argonne National Laboratory.

Department of Energy Supports Argonne Nuclear Technologies

This fall, U.S. Department of Energy Secretary Rick Perry announced nearly $4.7 million in funding for the department's Argonne National Laboratory across 16 projects in three divisions. Four of those TCF awards, representing more than $1 million in funds, are slated for Argonne's Nuclear Engineering division.

Southern Research Develops Gasifier Technology to Unlock Coal's Potential

Southern Research has been selected to receive nearly $1.7 million in U.S. Department of Energy funding to develop a new, cost-efficient gasifier capable of converting low-grade coal into synthesis gas (syngas) that can be used in a number of applications.

CEBAF Begins Operations following Upgrade Completion

The world's most advanced particle accelerator for investigating the quark structure of matter is gearing up to begin its first experiments following official completion of an upgrade to triple its original design energy. The Continuous Electron Beam Accelerator Facility (CEBAF) at the Department of Energy's Thomas Jefferson National Accelerator Facility is now back online and ramping up for the start of experiments.


  • Filters

  • × Clear Filters

Stirring up a Quantum Spin Liquid with Disorder

New, unexpected paradigm discovered: Disorder may actually promote an exotic quantum state, with potential for ultrafast computing.

Light Perfects Interfaces

Shining light on a growing semiconductor modifies its interface with the surface and could improve the optical properties of each.

Underappreciated Microbes Now Get Credit for Holding Down Two Jobs in Soil

Soil microbes work as both decomposers and synthesizers of carbon compounds in soil, offering new answers with impacts to crops and eco-health.

Energy, Economy, and the Earth: The Benefits of Creating Feedback Loops

Scientists reduce uncertainties in future climate prediction by directly coupling an energy-economy model to an Earth system model.

How Grasslands Regulate Their Productivity in Response to Droughts

Scientists show that grasslands are more sensitive to changes in the amount of moisture in the air than to changes in precipitation.

Building Confidence in Hydrologic Models

Scientists evaluate seven hydrologic models to understand how each model agrees and differs.

El Nino and Liquid Water Clouds Contribute to Antarctic Melt in 2015-2016

Atmospheric Radiation Measurement (ARM) observations provide clues on atmospheric contributions to an Antarctic melt event.

Designer Yeast Consumes Plant Matter and Spits Out Fatty Alcohols for Detergents and Biofuels

Highest concentration and yield of valuable chemicals reported in industrial yeast Saccharomyces cerevisiae.

Making Polymer Chemistry Click

Scientists unlock the key to efficiently make a new class of engineering polymers.

Photosynthesis without Cells: Turning Light into Fuel

An entirely human-made architecture produces hydrogen fuel using light, shows promise for transmitting energy in numerous applications.


Spotlight

Tuesday October 03, 2017, 01:05 PM

Stairway to Science

Argonne National Laboratory

Thursday September 28, 2017, 12:05 PM

After-School Energy Rush

Argonne National Laboratory

Thursday September 28, 2017, 10:05 AM

Bringing Diversity Into Computational Science Through Student Outreach

Brookhaven National Laboratory

Thursday September 21, 2017, 03:05 PM

From Science to Finance: SLAC Summer Interns Forge New Paths in STEM

SLAC National Accelerator Laboratory

Thursday September 07, 2017, 02:05 PM

Students Discuss 'Cosmic Opportunities' at 45th Annual SLAC Summer Institute

SLAC National Accelerator Laboratory

Thursday August 31, 2017, 05:05 PM

Binghamton University Opens $70 Million Smart Energy Building

Binghamton University, State University of New York

Wednesday August 23, 2017, 05:05 PM

Widening Horizons for High Schoolers with Code

Argonne National Laboratory

Saturday May 20, 2017, 12:05 PM

Rensselaer Polytechnic Institute Graduates Urged to Embrace Change at 211th Commencement

Rensselaer Polytechnic Institute (RPI)

Monday May 15, 2017, 01:05 PM

ORNL, University of Tennessee Launch New Doctoral Program in Data Science

Oak Ridge National Laboratory

Friday April 07, 2017, 11:05 AM

Champions in Science: Profile of Jonathan Kirzner

Department of Energy, Office of Science

Wednesday April 05, 2017, 12:05 PM

High-Schooler Solves College-Level Security Puzzle From Argonne, Sparks Interest in Career

Argonne National Laboratory

Tuesday March 28, 2017, 12:05 PM

Champions in Science: Profile of Jenica Jacobi

Department of Energy, Office of Science

Friday March 24, 2017, 10:40 AM

Great Neck South High School Wins Regional Science Bowl at Brookhaven Lab

Brookhaven National Laboratory

Wednesday February 15, 2017, 04:05 PM

Middle Schoolers Test Their Knowledge at Science Bowl Competition

Argonne National Laboratory

Friday January 27, 2017, 04:00 PM

Haslam Visits ORNL to Highlight State's Role in Discovering Tennessine

Oak Ridge National Laboratory

Tuesday November 08, 2016, 12:05 PM

Internship Program Helps Foster Development of Future Nuclear Scientists

Oak Ridge National Laboratory

Friday May 13, 2016, 04:05 PM

More Than 12,000 Explore Jefferson Lab During April 30 Open House

Thomas Jefferson National Accelerator Facility

Monday April 25, 2016, 05:05 PM

Giving Back to National Science Bowl

Ames Laboratory

Friday March 25, 2016, 12:05 PM

NMSU Undergrad Tackles 3D Particle Scattering Animations After Receiving JSA Research Assistantship

Thomas Jefferson National Accelerator Facility

Tuesday February 02, 2016, 10:05 AM

Shannon Greco: A Self-Described "STEM Education Zealot"

Princeton Plasma Physics Laboratory

Monday November 16, 2015, 04:05 PM

Rare Earths for Life: An 85th Birthday Visit with Mr. Rare Earth

Ames Laboratory

Tuesday October 20, 2015, 01:05 PM

Meet Robert Palomino: 'Give Everything a Shot!'

Brookhaven National Laboratory

Tuesday April 22, 2014, 11:30 AM

University of Utah Makes Solar Accessible

University of Utah

Wednesday March 06, 2013, 03:40 PM

Student Innovator at Rensselaer Polytechnic Institute Seeks Brighter, Smarter, and More Efficient LEDs

Rensselaer Polytechnic Institute (RPI)

Friday November 16, 2012, 10:00 AM

Texas Tech Energy Commerce Students, Community Light up Tent City

Texas Tech University

Wednesday November 23, 2011, 10:45 AM

Don't Get 'Frosted' Over Heating Your Home This Winter

Temple University

Wednesday July 06, 2011, 06:00 PM

New Research Center To Tackle Critical Challenges Related to Aircraft Design, Wind Energy, Smart Buildings

Rensselaer Polytechnic Institute (RPI)

Friday April 22, 2011, 09:00 AM

First Polymer Solar-Thermal Device Heats Home, Saves Money

Wake Forest University

Friday April 15, 2011, 12:25 PM

Like Superman, American University Will Get Its Energy from the Sun

American University

Thursday February 10, 2011, 05:00 PM

ARRA Grant to Help Fund Seminary Building Green Roof

University of Chicago

Tuesday December 07, 2010, 05:00 PM

UC San Diego Installing 2.8 Megawatt Fuel Cell to Anchor Energy Innovation Park

University of California San Diego

Monday November 01, 2010, 12:50 PM

Rensselaer Smart Lighting Engineering Research Center Announces First Deployment of New Technology on Campus

Rensselaer Polytechnic Institute (RPI)

Friday September 10, 2010, 12:40 PM

Ithaca College Will Host Regional Clean Energy Summit

Ithaca College

Tuesday July 27, 2010, 10:30 AM

Texas Governor Announces $8.4 Million Award to Create Renewable Energy Institute

Texas Tech University

Friday May 07, 2010, 04:20 PM

Creighton University to Offer New Alternative Energy Program

Creighton University

Wednesday May 05, 2010, 09:30 AM

National Engineering Program Seeks Subject Matter Experts in Energy

JETS Junior Engineering Technical Society

Wednesday April 21, 2010, 12:30 PM

Students Using Solar Power To Create Sustainable Solutions for Haiti, Peru

Rensselaer Polytechnic Institute (RPI)

Wednesday March 03, 2010, 07:00 PM

Helping Hydrogen: Student Inventor Tackles Challenge of Hydrogen Storage

Rensselaer Polytechnic Institute (RPI)

Thursday February 04, 2010, 02:00 PM

Turning Exercise into Electricity

Furman University

Thursday November 12, 2009, 12:45 PM

Campus Leaders Showing the Way to a Sustainable, Clean Energy Future

National Wildlife Federation (NWF)

Tuesday November 03, 2009, 04:20 PM

Furman University Receives $2.5 Million DOE Grant for Geothermal Project

Furman University

Thursday September 17, 2009, 02:45 PM

Could Sorghum Become a Significant Alternative Fuel Source?

Salisbury University

Wednesday September 16, 2009, 11:15 AM

Students Navigating the Hudson River With Hydrogen Fuel Cells

Rensselaer Polytechnic Institute (RPI)

Wednesday September 16, 2009, 10:00 AM

College Presidents Flock to D.C., Urge Senate to Pass Clean Energy Bill

National Wildlife Federation (NWF)

Wednesday July 01, 2009, 04:15 PM

Northeastern Announces New Professional Master's in Energy Systems

Northeastern University

Friday October 12, 2007, 09:35 AM

Kansas Rural Schools To Receive Wind Turbines

Kansas State University

Thursday August 17, 2006, 05:30 PM

High Gas Prices Here to Stay, Says Engineering Professor

Rowan University





Showing results

0-4 Of 2215