Newswise — Replacing a beloved tool is never easy. Erik Johnson had worked with the National Synchrotron Light Source (NSLS) for nearly 15 years when he and his colleagues began thinking about its replacement. But this switch wasn’t a matter of walking down to the hardware store.
The NSLS, a Department of Energy (DOE) Office of Science (SC) user facility at Brookhaven National Laboratory (BNL), opened in 1982. Over 30 years, scientists — three of whom won Nobel prizes for their work — used its intense beams of light over the course of more than 55,000 visits to study atomic structures and chemical processes. Johnson came to the NSLS in 1985 as a post-doctoral student. By 2000, Johnson and other leaders in the field realized the NSLS would soon be past its glory days.
They began dreaming up its successor: the NSLS-II. After five years of planning and research, SC approved the project to move forward.
“There was elation in the hallways,” said Johnson.
This was just the beginning. Ahead of Johnson and his co-workers stood a project that would take a decade and almost a billion dollars to build.
SC’s scientists and managers are familiar with this type of challenge. For decades, DOE and its predecessors have been developing federal user facilities for scientists to probe the building blocks of the universe and of life.
Constructing these user facilities requires immense planning and coordination. Good project management keeps projects on-time and on-budget. The $912-million NSLS-II project put SC’s project management skills to the test.
Not a Simple Process
Institutions can’t take project management for granted.
DOE construction projects run through five stages. First, SC confirms the need for the project under DOE’s mission. Then, the project team evaluates the alternatives and develops the basic design as well as an initial cost estimate for the approved alternative. Next on the docket is establishing the baseline — when managers set the project’s cost, schedule, and scope, instituting the standards they will measure the success of the project against. Then, DOE provides the go-ahead to start construction. The last step is certifying the project as complete.
A Big Machine Comes with Big Challenges
With the NSLS reaching its technical limits, scientists needed the next big tool to study incredibly small materials under real-world conditions. To look closer than ever at subjects ranging from batteries’ chemical reactions to viruses’ structures, researchers required an X-ray beam 10,000 times brighter than the original NSLS.
To make the investment worthwhile, the team needed to design NSLS-II to be so advanced that it could stay at the forefront of science for more than three decades. It took more than 20 different scientific workshops to hammer out the requirements. Participants included both scientists who would use the facility and engineers who would design it.
“I had gone from 15, 20 years where I knew everybody by first name and what they did to being surrounded by all new people,” said Johnson. “Which was pretty neat.”
To satisfy as many stakeholders as possible, the team designed the facility to immediately run at full capacity. They then scaled the plans down to what they could build initially. If they finished early or had funding left over, they could add pieces back. To make it possible to adapt the most advanced technologies, the team planned to add more research equipment and capabilities in phases after they finished construction of the main facility.
“[The process] was 90 percent preplanning and 10 percent execution,” said Robert Caradonna, the DOE Brookhaven Site Office deputy federal project director for the NSLS-II project.
All of that planning clarified the challenges that awaited them.
Creating powerful X-rays requires massive machines that accelerate electrons to high energies. These machines use specialized magnets to control the electrons that produce the X-rays. Equipment at experimental stations then harness these X-rays. Most facilities have rings that store the electrons; NSLS-II’s ring needed to be a half-mile around.
The building needed to meet several other specific requirements. Because temperature changes can influence the magnets’ size and position, the inside of the storage ring could never waver from a balmy 78 F. To ensure the beams would never tremble, the team needed to minimize the effect of vibrations from trucks on the nearby Long Island Expressway and waves from the Atlantic Ocean.
The project’s sheer complexity was perhaps the biggest challenge of all. The original schedule projected 5,000 separate activities. At the height of the project, the list expanded to 11,000. At some points, the team was managing a million dollars of work each day.
“We spent so much time with our heads down pushing on this thing that it didn’t really get to be overwhelming,” said Johnson. “We were too damn busy for it to get overwhelming.”
Prepared for Disaster
In project management, being a pessimist can pay off. A huge part of ensuring the project ran smoothly was anticipating and managing risks.
The Office of Science purposely plans for the worst case scenarios. Over the course of the project, the team ran more than 400 separate risk assessments. They also built numerous computer models that pinpointed exactly where every single piece of equipment needed to go.
One of the biggest areas of uncertainty was manufacturing the massive magnets. Buying 900 magnets anywhere is difficult. Buying 900 extremely high-powered, cutting-edge magnets from one supplier wasn’t going to happen.
“It would be nice if we could give it to one guy and he could produce all the magnets,” said Caradonna, but, “we knew that was going to be impossible.”
There are only a handful of places in the world that could produce the magnets and other specialized pieces of equipment. The team developed seven contracts with five different suppliers, several of which were in other countries. The different languages, management cultures, geographic locations, and even measurement units caused conflicts. Some vendors were unresponsive.
To solve these problems, members of the NSLS-II team flew to the suppliers and consulted with them on site. The hands-on assistance improved communications and quality control. It even allowed NSLS-II staff members to get experience with the components before they arrived to BNL. Despite some early delays, the magnet manufacturing didn’t hold up the process as a whole.
The team was able to compensate for these delays because of savvy scheduling. In many projects like this, a delay in one step can cascade down to others, creating multiple delays and scheduling conflicts. In contrast, the NSLS-II team designed the project so they could change the order of the steps. For example, they built five identical pieces of the accelerator ring that fit together like LEGO®s. If the magnets weren’t finished for one piece of the ring, they could still finish the others in time.
This approach also came in handy when the project received $150 million in funding earlier than planned through a special bill to help the country recover from the economic crash in 2008. With this funding and a favorable construction market, they negotiated a lower price with the construction company and finished the laboratory buildings nearly a year ahead of schedule.
“You never know when fortune is going to smile on you, and you never know when you’re going to do something sooner rather than later,” said Johnson. “It’s all about being ready.”
Here a Review, There a Review
Preparation will only get you so far. SC’s regular and strict project reviews by independent experts got the team the rest of the way there. Every step of the process had a review, from the scope and scientific goals to the construction: 54 in total.
Over the course of the project, the review teams gave more than 1,300 recommendations.
As the NSLS-II lessons learned document states: “Project reviews are the most important management tool to ensure the project is staying on track. If you are not required to have them, you should inflict them on yourself.”
NSLS-II is not the only project to benefit from rigorous reviews. The GAO has cited SC’s reviews as a major reason why the majority of SC’s projects are completed on-time and on-budget.
The project teams aren’t the only ones who learn from the experience. The independent experts are often in charge of similar projects at their own agencies, national laboratories, or universities.
10 Years Later
Ten years after DOE approved the idea for NSLS-II, it was finished. SC declared it complete in March 2015, three months before its target date. It opened to users that July.
Owing to sound project management practices, there was enough funding available for the NSLS-II to include $68 million in optional features beyond the basic construction plan. It had an additional beamline to provide X-rays to another experimental station, a larger building that would make it easier to expand, and extra components to increase reliability. In 2016, the team won both the Project Management Institute’s Project of the Year award and the DOE Secretary’s Award of Excellence, the highest honor that DOE awards to a project.
“I was immensely proud,” said Johnson, “but fully cognizant of all of the work that needs to be done still to fully realize the potential of this instrument.”
Since it opened, the team has launched 28 experimental stations, or beamlines, out of a total of 60 stations it can support.
“Everything from the dreaming to the final delivery is going on at the same time,” said Johnson.
Despite all of the new machinery, one day the NSLS-II will become obsolete just like its predecessor. One Friday night, Johnson went home and noted that the old NSLS sign was still up. The next Monday, it was gone, replaced by a sign for the lab’s new Computational Science Initiative. To Johnson, that change reinforced the fact that his mission is about more than equipment.
“Those people who used it, they still have those experiences,” he said. “It’s not the stuff that you build, it’s that what you build that enables other people to do what’s important.”
The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information please visit https://science.energy.gov.
Shannon Brescher Shea is a senior writer/editor in the Office of Science, [email protected].
RSS