- 2017-06-29 08:05:06
- Article ID: 677253
Brookhaven Lab's Scientific Data and Computing Center Reaches 100 Petabytes of Recorded Data
Total reflects 17 years of experimental physics data collected by scientists to understand the fundamental nature of matter and the basic forces that shape our universe
-
Credit: Brookhaven National Laboratory
(Back row) Ognian Novakov, Christopher Pinkenburg, Jérôme Lauret, Eric Lançon, (front row) Tim Chou, David Yu, Guangwei Che, and Shigeki Misawa at Brookhaven Lab's Scientific Data and Computing Center, which houses the Oracle StorageTek tape storage system where experimental data are recorded.
-
Credit: Brookhaven National Laboratory
Inside one of the automated tape libraries at the Scientific Data and Computing Center (SDCC), Eric Lançon, director of SDCC, holds a magnetic tape cartridge. When scientists need data, a robotic arm (the piece of equipment in front of Lançon) retrieves the relevant cartridges from their slots and loads them into drives in the back of the library.
“This is a major milestone for SDCC, as it reflects nearly two decades of scientific research for the RHIC nuclear physics and ATLAS particle physics experiments, including the contributions of thousands of scientists and engineers,” said Brookhaven Lab technology architect David Yu, who leads the SDCC’s Mass Storage Group.
SDCC is at the core of a global computing network connecting more than 2,500 researchers around the world with data from the STAR and PHENIX experiments at RHIC—a DOE Office of Science User Facility at Brookhaven—and the ATLAS experiment at the Large Hadron Collider (LHC) in Europe. In these particle collision experiments, scientists recreate conditions that existed just after the Big Bang, with the goal of understanding the fundamental forces of nature—gravitational, electromagnetic, strong nuclear, and weak nuclear—and the basic structure of matter, energy, space, and time.
Big Data Revolution
The RHIC and ATLAS experiments are part of the big data revolution. These experiments involve collecting extremely large datasets that reduce statistical uncertainty to make high-precision measurements and search for extremely rare processes and particles.
For example, only one Higgs boson—an elementary particle whose energy field is thought to give mass to all the other elementary particles—is produced for every billion proton-proton collisions at the LHC. More, once produced, the Higgs boson almost immediately decays into other particles. So detecting the particle is a rare event, with around one trillion collisions required to detect a single instance. When scientists first discovered the Higgs boson at the LHC in 2012, they observed about 20 instances, recording and analyzing more than 300 trillion collisions to confirm the particle’s discovery.
At the end of 2016, the ATLAS collaboration released its first measurement of the mass of the W boson particle (another elementary particle that, together with the Z boson, is responsible for the weak nuclear force). This measurement, which is based on a sample of 15 million W boson candidates collected at LHC in 2011, has a relative precision of 240 parts per million (ppm)—a result that matches the best single-experiment measurement announced in 2007 by the Collider Detector at Fermilab collaboration, whose measurement is based on several years’ worth of collected data. A highly precise measurement is important because a deviation from the mass predicted by the Standard Model could point to new physics. More data samples are required to achieve the level of accuracy (80 ppm) that scientists need to significantly test this model.
The volume of data collected by these experiments will grow significantly in the near future as new accelerator programs deliver higher-intensity beams. The LHC will be upgraded to increase its luminosity (rate of collisions) by a factor of 10. This High-Luminosity LHC, which should be operational by 2025, will provide a unique opportunity for particle physicists to look for new and unexpected phenomena within the exabytes (one exabyte equals 1000 petabytes) of data that will be collected.
Data archiving is the first step in making available the results from such experiments. Thousands of physicists then need to calibrate and analyze the archived data and compare the data to simulations. To this end, computational scientists, computer scientists, and mathematicians in Brookhaven Lab’s Computational Science Initiative, which encompasses SDCC, are developing programming tools, numerical models, and data-mining algorithms. Part of SDCC’s mission is to provide computing and networking resources in support of these activities.
A Data Storage, Computing, and Networking Infrastructure
Housed inside SDCC are more than 60,000 computing cores, 250 computer racks, and tape libraries capable of holding up to 90,000 magnetic storage tape cartridges that are used to store, process, analyze, and distribute the experimental data. The facility provides approximately 90 percent of the computing capacity for analyzing data from the STAR and PHENIX experiments, and serves as the largest of the 12 Tier 1 computing centers worldwide that support the ATLAS experiment. As a Tier 1 center, SDCC contributes nearly 23 percent of the total computing and storage capacity for the ATLAS experiment and delivers approximately 200 terabytes of data (picture 62 million photos) per day to more than 100 data centers globally.
At SDCC, the High Performance Storage System (HPSS) has been providing mass storage services to the RHIC and LHC experiments since 1997 and 2006, respectively. This data archiving and retrieval software, developed by IBM and several DOE national laboratories, manages petabytes of data on disk and in robot-controlled tape libraries. Contained within the libraries are magnetic tape cartridges that encode the data and tape drives that read and write the data. Robotic arms load the cartridges into the drives and unload them upon request.
When ranked by the volume of data stored in a single HPSS, Brookhaven’s system is the second largest in the nation and the fourth largest in the world. Currently, the RACF operates nine Oracle robotic tape libraries that constitute the largest Oracle tape storage system in the New York tri-state area. Contained within this system are nearly 70,000 active cartridges with capacities ranging from 800 gigabytes to 8.5 terabytes, and more than 100 tape drives. As the volume of scientific data to be stored increases, more libraries, tapes, and drives can be added accordingly. In 2006, this scalability was exercised when HPSS was expanded to accommodate data from the ATLAS experiment at LHC.
“The HPSS system was deployed in the late 1990s, when the RHIC accelerator was coming on line. It allowed data from RHIC experiments to be transmitted via network to the data center for storage—a relatively new idea at the time,” said Shigeki Misawa, manager of Mass Storage and General Services at Brookhaven Lab. Misawa played a key role in the initial evaluation and configuration of HPSS, and has guided the system through significant changes in hardware (network equipment, storage systems, and servers) and operational requirements (tape drive read/write rate, magnetic tape cartridge capacity, and data transfer speed). “Prior to this system, data was recorded on magnetic tape at the experiment and physically moved to the data center,” he continued.
Over the years, SDCC’s HPSS has been augmented with a suite of optimization and monitoring tools developed at Brookhaven Lab. One of these tools is David Yu’s scheduling software that optimizes the retrieval of massive amounts of data from tape storage. Another, developed by Jérôme Lauret, software and computing project leader for the STAR experiment, is software for organizing multiple user requests to retrieve data more efficiently.
Engineers in the Mass Storage Group—including Tim Chou, Guangwei Che, and Ognian Novakov—have created other software tools customized for Brookhaven Lab’s computing environment to enhance data management and operation abilities and to improve the effectiveness of equipment usage.
STAR experiment scientists have demonstrated the capabilities of SDCC’s enhanced HPSS, retrieving more than 4,000 files per hour (a rate of 6,000 gigabytes per hour) while using a third of HPSS resources. On the data archiving side, HPSS can store data in excess of five gigabytes per second.
As demand for mass data storage spreads across Brookhaven, access to HPSS is being extended to other research groups. In the future, SDCC is expected to provide centralized mass storage services to multi-experiment facilities, such as the Center for Functional Nanomaterials and the National Synchrotron Light Source II—two more DOE Office of Science User Facilities at Brookhaven.
“The tape library system of SDCC is a clear asset for Brookhaven’s current and upcoming big data science programs,” said SDCC Director Eric Lançon. “Our expertise in the field of data archiving is acknowledged worldwide.”
Brookhaven National Laboratory is supported by the Office of Science of the U.S. Department of Energy. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.
MORE NEWS FROM
Brookhaven National Laboratory
comments 
commenting policy
Participating Labs
- DOE Office of Science
- Argonne National Laboratory
- Oak Ridge National Laboratory
- Pacific Northwest National Laboratory
- Iowa State University, Ames Laboratory
- Brookhaven National Laboratory
- Princeton Plasma Physics Laboratory
- Lawrence Berkeley National Laboratory
- Thomas Jefferson National Accelerator Facility
- Fermi National Accelerator Laboratory (Fermilab)
- SLAC National Accelerator Laboratory
Theorists probe the relationship between 'strange metals' and high-temperature superconductors
SLAC theorists have observed strange metallicity in a well-known model for simulating and describing the behavior of materials with strongly correlated electrons, which join forces to produce unexpected phenomena rather than acting independently.
The Beauty of Imperfections: Linking Atomic Defects to 2D Materials' Electronic Properties
Scientists at Berkeley Lab have revealed how atomic defects emerge in transition metal dichalcogenides, and how those defects shape the 2D material's electronic properties. Their findings could provide a versatile yet targeted platform for designing 2D materials for quantum information science.
High school students' data science contributions boost biomineralization research
Two high school students developed software to analyze images of diatoms--algae that produce silicon for constructing cell walls--to determine the differences between wild and genetically modified strains of these organisms. This work was instrumental to a research team interested in optimizing diatoms for biomineralization, the process of making materials from biological systems.
Turning Up the Heat to Create New Nanostructured Metals
The metallic thin films with 3-D interlocking nanostructures could be used in catalysis, energy storage, and biomedical sensing.
Blowing bubbles: PPPL scientist confirms novel way to launch and drive current in fusion plasmas
PPPL physicist Fatima Ebrahimi has used high-resolution computer simulations to confirm the practicality of the CHI start-up technique. The simulations show that CHI could produce electric current continuously in larger, more powerful tokamaks than exist today to produce stable fusion plasmas.
A four-way switch promises greater tunability of layered materials
A team from Oak Ridge National Laboratory and Vanderbilt University made the first experimental observation of a material phase that had been predicted but never seen.
Scientists discover "ripple" in flexible material that could improve electronic properties
Argonne scientists have discovered an intriguing new behavior in a two-dimensional material at the atomic level as it is stretched and strained, like it would be in an actual flexible device.
AI for Plant Breeding in an Ever-Changing Climate
In this Q&A, Oak Ridge National Laboratory's Dan Jacobson talks about his team's work on a genomic selection algorithm, his vision for the future of environmental genomics, and the space where simulation meets AI.
A New Parallel Strategy for Tackling Turbulence on Summit
A team at Georgia Tech created a new turbulence algorithm optimized for the Summit supercomputer. It reached a performance of less than 15 seconds of wall-clock time per time step for more than 6 trillion grid points--a new world record surpassing the prior state of the art in the field for the size of the problem.
Modeling Every Building in America Starts with Chattanooga
An ORNL team used the Titan supercomputer to model every building serviced by the Electric Power Board of Chattanooga--all 178,368 of them--and discovered that EPB could potentially save $11-$35 million per year by adjusting electricity usage during peak critical times.
Dmitri Zakharov Recognized with the 2019 Chuck Fiori Award
The award honors Dmitri Zakharov's contributions to environmental transmission electron microscopy at Brookhaven Lab's Center for Functional Nanomaterials.
Two Argonne projects earn Secretary of Energy Honor Awards
With this year's Nobel Prize in Chemistry awarded for the development of lithium-ion batteries, directors of the Joint Center for Energy Storage Research share perspectives on the future of energy storage.
Argonne teams up with Altair to manage use of upcoming Aurora supercomputer
Argonne National Laboratory and Altair, a global technology company, have created a new scheduling system that will be employed on the Aurora supercomputer.
University of Maryland, Baltimore County wins DOE's 2019 CyberForce Competition(tm)
After a long suspenseful day, University of Maryland, Baltimore County earned the top spot as national winner of the U.S. Department of Energy's CyberForce Competition.
In its 15th year, INCITE advances open science with supercomputer grants to 47 projects
The U.S. Department of Energy's Office of Science announced allocations of supercomputer access to 47 science projects for 2020--awarding 60 percent of the available time on some of the nation's most powerful supercomputers, with the ultimate goal of accelerating discovery and innovation. In 2020, 14 projects will run on Theta and 39 projects on Summit, where six of these projects will receive an allocation on both systems.
ASU solar awards eclipse other universities in latest round of DOE funding
ASU receives $9.8 million in Solar Energy Technologies Office Awards.
DOE to Provide $10 Million for New Research into Ecosystem Processes
The U.S. Department of Energy (DOE) announced a plan to provide $10 million for new observational and experimental studies aimed at improving the accuracy of today's Earth system models. Research will focus on three separate types of environments--terrestrial, watershed, and subsurface--where current models fall short of providing fully accurate representation.
ORNL to host 13 teams for DOE CyberForce Competition
Oak Ridge National Laboratory will give college students the chance to practice cybersecurity skills in a real-world setting as a host of the Department of Energy's fifth collegiate CyberForce Competition on Nov. 16.
Argonne nuclear engineer J'Tia Hart selected to Crain's Chicago Business "40 Under 40"
Argonne nuclear engineer J'Tia Hart has been named to Crain's Chicago Business's "40 Under 40" list, which recognizes young leaders in a variety of fields.
Lab-Wide Stormwater Capture, Transportation Savings and Clean-Up Efforts Win Federal Recognition
Argonne National Laboratory has won a regional Federal Green Challenge award for conserving resources and saving taxpayers' money.
Harvesting Energy from Light using Bio-inspired Artificial Cells
Scientists designed and connected two different artificial cells to each other to produce molecules called ATP (adenosine triphosphate).
Engineering Living Scaffolds for Building Materials
Bone and mollusk shells are composite systems that combine living cells and inorganic components. This allows them to regenerate and change structure while also being very strong and durable. Borrowing from this amazing complexity, researchers have been exploring a new class of materials called engineered living materials (ELMs).
Excavating Quantum Information Buried in Noise
Researchers developed two new methods to assess and remove error in how scientists measure quantum systems. By reducing quantum "noise" - uncertainty inherent to quantum processes - these new methods improve accuracy and precision.
How Electrons Move in a Catastrophe
Lanthanum strontium manganite (LSMO) is a widely applicable material, from magnetic tunnel junctions to solid oxide fuel cells. However, when it gets thin, its behavior changes for the worse. The reason why was not known. Now, using two theoretical methods, a team determined what happens.
When Ions and Molecules Cluster
How an ion behaves when isolated within an analytical instrument can differ from how it behaves in the environment. Now, Xue-Bin Wang at Pacific Northwest National Laboratory devised a way to bring ions and molecules together in clusters to better discover their properties and predict their behavior.
Tune in to Tetrahedral Superstructures
Shape affects how the particles fit together and, in turn, the resulting material. For the first time, a team observed the self-assembly of nanoparticles with tetrahedral shapes.
Tracing Interstellar Dust Back to the Solar System's Formation
This study is the first to confirm dust particles pre-dating the formation of our solar system. Further study of these materials will enable a deeper understanding of the processes that formed and have since altered them.
Investigating Materials that Can Go the Distance in Fusion Reactors
Future fusion reactors will require materials that can withstand extreme operating conditions, including being bombarded by high-energy neutrons at high temperatures. Scientists recently irradiated titanium diboride (TiB2) in the High Flux Isotope Reactor (HFIR) to better understand the effects of fusion neutrons on performance.
Better 3-D Imaging of Tumors in the Breast with Less Radiation
In breast cancer screening, an imaging technique based on nuclear medicine is currently being used as a successful secondary screening tool alongside mammography to improve the accuracy of the diagnosis. Now, a team is hoping to improve this imaging technique.
Microbes are Metabolic Specialists
Scientists can use genetic information to measure if microbes in the environment can perform specific ecological roles. Researchers recently analyzed the genomes of over 6,000 microbial species.
Spotlight
Barbara Garcia: A first-generation college student spends summer doing research at PPPL
Princeton Plasma Physics Laboratory
Argonne organization's scholarship fund blazes STEM pathway
Argonne National Laboratory
Brookhaven Lab, Suffolk Girl Scouts Launch Patch Program
Brookhaven National Laboratory
From an acoustic levitator to a "Neutron Bloodhound" robot, hands-on research inspires PPPL's summer interns
Princeton Plasma Physics Laboratory
Brookhaven Lab Celebrates the Bright Future of its 2019 Interns
Brookhaven National Laboratory
PPPL apprenticeship program offers young people chance to earn while they learn high-tech careers
Princeton Plasma Physics Laboratory
JSA Awards Graduate Fellowships for Research at Jefferson Lab
Thomas Jefferson National Accelerator Facility
ILSAMP Symposium showcases benefits for diverse students, STEM pipeline
Argonne National Laboratory
Integrating Scientific Computing into Science Curricula
Brookhaven National Laboratory
Students from Minnesota and Massachusetts Win DOE's 29th National Science Bowl(r)
Department of Energy, Office of Science
DOE's Science Graduate Student Research Program Selects 70 Students to Pursue Research at DOE Laboratories
Department of Energy, Office of Science
Young Women's Conference in STEM seeks to change the statistics one girl at a time
Princeton Plasma Physics Laboratory
Students team with Argonne scientists and engineers to learn about STEM careers
Argonne National Laboratory
Lynbrook High wins 2019 SLAC Regional Science Bowl competition
SLAC National Accelerator Laboratory
Equipping the next generation for a technological revolution
Argonne National Laboratory
Chemistry intern inspired by Argonne's real-world science
Argonne National Laboratory
Argonne intern streamlines the beamline
Argonne National Laboratory
Research on Light-Matter Interaction Could Lead to Improved Electronic and Optoelectronic Devices
Rensselaer Polytechnic Institute (RPI)
Innovating Our Energy Future
Oregon State University, College of Engineering
Physics graduate student takes her thesis research to a Department of Energy national lab
University of Alabama at Birmingham
"Model" students enjoy Argonne campus life
Argonne National Laboratory
Writing Code for a More Skilled and Diverse STEM Workforce
Brookhaven National Laboratory
New graduate student summer school launches at Princeton Plasma Physics Laboratory
Princeton Plasma Physics Laboratory
The Gridlock State
California State University (CSU) Chancellor's Office
Meet Jasmine Hatcher and Trishelle Copeland-Johnson
Brookhaven National Laboratory
Argonne hosts Modeling, Experimentation and Validation Summer School
Argonne National Laboratory
Undergraduate Students Extoll Benefits of National Laboratory Research Internships in Fusion and Plasma Science
Princeton Plasma Physics Laboratory
Students affected by Hurricane Maria bring their research to SLAC
SLAC National Accelerator Laboratory
Brookhaven Lab Pays Tribute to 2018 Summer Interns
Brookhaven National Laboratory
CSUMB Selected to Host Architecture at Zero Competition in 2019
California State University, Monterey Bay
From Hurricane Katrina Victim to Presidential Awardee: A SUNO Professor's Award-Winning Mentoring Efforts
Brookhaven National Laboratory
Department of Energy Invests $64 Million in Advanced Nuclear Technology
Rensselaer Polytechnic Institute (RPI)
Professor Miao Yu Named the Priti and Mukesh Chatter '82 Career Development Professor
Rensselaer Polytechnic Institute (RPI)
2018 RHIC & AGS Annual Users' Meeting: 'Illuminating the QCD Landscape'
Brookhaven National Laboratory
High-School Students Studying Carbon-Based Nanomaterials for Cancer Drug Delivery Visit Brookhaven Lab's Nanocenter
Brookhaven National Laboratory
Argonne welcomes The Martian author Andy Weir
Argonne National Laboratory
UW Professor and Clean Energy Institute Director Daniel Schwartz Wins Highest U.S. Award for STEM Mentors
University of Washington
Creating STEM Knowledge and Innovations to Solve Global Issues Like Water, Food, and Energy
Illinois Mathematics and Science Academy (IMSA)
Professor Emily Liu Receives $1.8 Million DoE Award for Solar Power Systems Research
Rensselaer Polytechnic Institute (RPI)
Celebrating 40 years of empowerment in science
Argonne National Laboratory
Introducing Graduate Students Across the Globe to Photon Science
Brookhaven National Laboratory
Students from Massachusetts and Washington Win DOE's 28th National Science Bowl(r)
Department of Energy, Office of Science
Q&A: Al Ashley Reflects on His Efforts to Diversify SLAC and Beyond
SLAC National Accelerator Laboratory
Insights on Innovation in Energy, Humanitarian Aid Highlight UVA Darden's Net Impact Week
University of Virginia Darden School of Business
Ivy League Graduate, Writer and Activist with Dyslexia Visits CSUCI to Reframe the Concept of Learning Disabilities
California State University, Channel Islands
Photographer Adam Nadel Selected as Fermilab's New Artist-in-Residence for 2018
Fermi National Accelerator Laboratory (Fermilab)
Showing results
0-4 Of 2215