MORE ABOUT THIS BOOK
OUT OF LOCK
On the Saturday afternoon of July 4, 2015, NASA’s New Horizons Pluto mission leader Alan Stern was in his office near the project Mission Control Center, working, when his cell phone rang. He was aware of the Independence Day holiday but was much more focused on the fact that the date was “Pluto flyby minus ten days.” New Horizons, the spacecraft mission that had been the central focus of his career for fourteen years, was now just ten days from its targeted encounter with the most distant planet ever explored.
Immersed in work that afternoon, Alan was busy preparing for the flyby. He was used to operating on little sleep during this final approach phase of the mission, but that day he’d gotten up in the middle of the night and gone into their Mission Operations Center (MOC) for the upload of the crucial, massive set of computer instructions to guide the spacecraft through its upcoming close flyby. That “command load” represented nearly a decade of work, and that morning it had been sent by radio transmission hurtling at the speed of light to reach New Horizons, then on its approach to Pluto.
Glancing at his ringing phone, Alan was surprised to see the caller was Glen Fountain, the longtime project manager of New Horizons. He felt a chill because he knew that Glen was taking time off for the holiday, at his nearby home, before the final, all-out intensity of the upcoming flyby. Why would Glen be calling now?
Alan picked up the phone. “Glen, what’s up?”
“We’ve lost contact with the spacecraft.”
Alan replied, “I’ll meet you in the MOC; see you in five minutes.”
Alan hung up his phone and sat down at his desk for a few seconds, stunned, shaking his head in disbelief. Unintentional loss of contact with Earth should never happen to any spacecraft. It had never before happened to New Horizons over the entire nine-year flight from Earth to Pluto. How could this be happening now, just ten days out from Pluto?
He grabbed his things, poked his head into a meeting down the hall where he was supposed to be heading next, and said, “We’ve lost contact with the spacecraft.” Then his colleagues looked at him, dumbfounded. “I’m headed to the MOC and don’t know when I’ll be back. It probably won’t be today.” He walked out to his car into the Maryland summer heat, and drove the half-mile across the campus of the Applied Physics Laboratory in Laurel, Maryland, where New Horizons was operated.
That drive was probably the longest few minutes in Alan’s life. He had high confidence in his team handling emergencies: they had rehearsed so many contingency scenarios, if any team could handle this, it would be New Horizons. But still, he couldn’t prevent his mind from picturing the worst.
Specifically, he couldn’t help but think of NASA’s ill-fated Mars Observer. That spacecraft, launched in 1992, also went silent, just three days before reaching Mars. All attempts to reestablish communication were unsuccessful. NASA later determined that Mars Observer had experienced a rupture in a fuel tank leading to a catastrophic loss of the spacecraft. In other words, it had blown up.
Alan thought to himself, “If we’ve lost the spacecraft, this entire fourteen-year-long project, and the work of over 2,500 people, will have failed. We won’t have learned much of anything about Pluto, and New Horizons will become a poster child for dashed dreams and failure.”
As soon as Alan reached the large, mostly windowless office building where the MOC was housed, he parked, pushed dark thoughts out of his mind, and went in to get to work. The MOC looks very much like you’d expect a spacecraft control center to look, just like in Apollo 13 or other space movies: dominated by the glow of giant projection screens along the walls, and rows of smaller computer screens at consoles.
Throughout the nine long years of travel toward the ninth planet, the radio link to New Horizons was the lifeline that allowed its team to contact and control the craft and to receive spacecraft status and data from its observations. As New Horizons kept going farther to the outer reaches of the solar system, the time delays to communicate with it increased, and the link had lengthened to what was now a nine-hour round trip for radio signals, traveling at the speed of light.
To stay in touch, New Horizons depends, as do all long-distance spacecraft, on a largely unknown and unsung marvel of planetary exploration: NASA’s Deep Space Network. This trio of giant radio-dish complexes in Goldstone, California; Madrid, Spain; and Canberra, Australia, seamlessly hands off communication duties between one another as the Earth rotates on its axis every twenty-four hours. The three stations are spread around the world so that no matter where an object is in deep space, at any time at least one of the antenna complexes can point in its direction.
But now … the DSN had lost contact with one of their most precious assets, New Horizons.
Alan scanned his badge on the way through building security and arrived in the MOC. Inside, he looked immediately for Alice Bowman, the mission’s coolheaded and enormously competent fourteen-year veteran Mission Operations Manager (hence her nickname: “MOM”). Alice led the Mission Control team that maintained communications with and controlled the spacecraft. Alice was huddled with a small group of engineers and mission operations experts in front of a computer screen displaying the ominous message “OUT OF LOCK.”
Their calm attitude was reassuring, but it struck Alan that they seemed pretty relaxed, considering the stakes at hand. In fact, as he probed them with questions he learned they were already developing a working hypothesis about what might have happened.
At the time of the signal loss, they knew the spacecraft was programmed to be doing several things at once, a stressful condition on its main computer. Perhaps, they surmised, that computer became overloaded. In simulations, this very same suite of tasks had not been a problem for the identical computer on the spacecraft mission simulator at the MOC. But perhaps something on board the spacecraft was not exactly the same as in the simulations.
If the onboard computer had become overwhelmed with tasks, they surmised, it could have decided to reboot itself. Alternately, it may have sensed a problem and turned itself off, automatically switching authority to its backup main computer aboard New Horizons.
Either one of these alternatives would be good news, meaning that the spacecraft was still alive, and that the problem was fixable. In either scenario New Horizons would have already woken up and radioed home a signal informing them of its status. So if either of the two scenarios was correct, they should hear from their “bird” in about an hour to an hour and a half, once the spacecraft had automatically completed its initial recovery steps. Alice and her team seemed confident that one of these computer problems was the explanation, and after so many years of flying New Horizons, Alan was inclined to give them the benefit of the doubt. Yet, if they did not hear anything—if that hour and a half went by without a signal—it would mean they were without a good hypothesis for what had occurred, and quite possibly without their spacecraft for good.
As more mission staff started to arrive to help address the unfolding emergency, Alan set up shop in the Situation Room, a fishbowl glass conference room that looked out on Bowman’s New Horizons mission control room. Glen Fountain also arrived. Soon it became clear that the recovery from this would be involved, and that the team members might be settling in for a long haul—several days of overnighters to resolve the problem and get the impending flyby back on track.
If this were an orbiter, or a rover safely on an alien surface, the team could take its time to analyze the problem, make recommendations, try different courses of action. But New Horizons was a flyby mission. The spacecraft was rushing toward Pluto at over 750,000 miles per day—more than 31,000 miles per hour. Back to working order or not, it would fly by the planet on July 14, never to return. There was no stopping New Horizons as they sorted the problem out. There was only one shot at getting the goods at Pluto—New Horizons had no backup, no second chance, no way to delay its date with Pluto.
There is a phrase from World War I describing warfare as “months of boredom punctuated by moments of terror.” The same applies to long spacecraft missions. And it was a long and frankly terrifying hour as they awaited the hoped-for signal to return from New Horizons.
Then, relief: At 3:11 P.M., 1 hour and 16 minutes after the spacecraft signal had been lost, signals returned and a new message appeared on mission control computer screens: “LOCKED.”
Alan took a deep breath. The hypothesis that the engineers had formulated must have been correct. The spacecraft was talking to them again. They were back in the game!
Back in the game, yes, but they were still not out of the woods. It was going to take an enormous amount of work to get the spacecraft back on schedule for the flyby. First they had to get it out of “safe mode”—the state the spacecraft goes into when it senses a problem, where every noncritical system is shut off. But there was much more to do to restore the flyby than just that. All the computer files that had been meticulously uploaded since December to support the coming exploration of Pluto would have to be reloaded to the spacecraft before the flyby operations could begin. This would be weeks of work under normal circumstances; but they didn’t have weeks, they had ten days until New Horizons reached Pluto and only three days until the start of the critical data taking for closest approach, when all of the most valuable scientific observations would be made.
Bowman and her team got to work right away, and the task proved daunting. After they got the spacecraft out of safe mode, they would need to command it to switch from the backup onto the main computer—something they had never had to do before, and then they had to reconstruct and retransmit all of the files that were to orchestrate operations during the flyby. And all of this had to be tested on the mission simulator before anything could be sent to the spacecraft to make sure it would work. And it had to be perfect: if they missed even just one crucial file or used the wrong version of it, much of the flyby they had spent so many years planning could be lost.
The clock was ticking. The first science observations of the close flyby—the most crucial observations at the heart of the mission—would be made just 6.4 days out from Pluto, on Tuesday. That 6.4 days was set by the length of a day on Pluto, one full rotation on its axis, meaning Tuesday was the last time they would ever see large parts of the planet before the flyby. If things were not back on track by then, there would be whole areas of the planet New Horizons would miss mapping—forever.
Could they get the spacecraft back on the timeline by then? Alice and her team created a plan and thought they just might be able to pull it off—assuming no new problems were encountered or were generated by mistakes they might make during the marathon, sleepless recovery effort they were embarking on.
Would it work? Or would they fail? As Alan said that afternoon, if you were on the mission team and weren’t religious before this happened, you were probably becoming religious at this point. Time would tell, and so will we, but first let’s tell the story of New Horizons, and how it came to reach this point.
Copyright © 2018 by Alan Stern and David Grinspoon