NASA Johnson Space Center Oral History Project
Edited Oral History Transcript

Paul S. Hill
Interviewed by Rebecca Wright
Houston, TX – 26 March 2015

Wright: Today is March 26, 2015. This oral history session is being conducted with Paul Hill in Houston, Texas, as part of the JSC Oral History Project, and for JSC’s Knowledge Management Office. Interviewer is Rebecca Wright, assisted by Rebecca Hackler. Thank you for coming back in today. I appreciate you coming two times in one week. Where we left off on Tuesday is where we’d like to pick up today. We’d like for you to start by sharing with us about STS-107, and what your role was, if any, for that mission, or what you were doing at the time that you heard of the [Space Shuttle] Columbia accident, and then of course your major role in bringing the nation back to flight.

Hill: I had no official role on 107, and the morning of the accident, I was sitting in a gym in League City [Texas], next to Cori [Corinna] Hobaugh, who is Charlie [Charles O.] Hobaugh’s wife. Charlie was the CapCom during entry. We were watching our youngest kids play YMCA or some kind of community league basketball. My wife had taken our oldest daughter up to Channelview for a marching band competition. The only reason that’s important is I’m sitting here watching this game, I look up at the door, and in walks my wife with this panicked look on her face. It was about 10 after 9:00 in the morning. I got up and walked over to see what was going on, and she hands me my badge and my pager. This was back in the days before everybody had smartphones. She said, “Something bad just happened to Shuttle.”

I went and hopped in my truck and drove to JSC on a very foggy day that I’ll always remember. In fact, I was driving through the gates, and as I was driving through, it was the first time I thought what the hell am I going there for? What am I going to do? I’m wearing shorts and a T-shirt and hadn’t shaved in two days. I think it was a Saturday morning.

I go all the way into the [Mission] Control Center, go upstairs into one of the private areas the flight directors have for conferences, and call down to the Deputy Chief of the Flight Director Office, Phil [Philip L.] Engelauf, who was in the Control Room at the time. Again, I didn’t even know what the heck I was there for, or what I was going to do for him. As it turned out, I didn’t really do anything. Phil came up, and we talked for a little while. I in fact helped him not at all, other than give him a chance to talk to somebody not mission-related.

But then within a couple of days, I was asked to come over to the Control Center, and there was a roomful of people in a room about this size, a normal office size room. They had a couple of tables overflowing with paper, literally piles of paper piled up and falling off. They had maps that had little Post-it stickers all over them. These folks had been the first ones to start answering phone calls from the public after the accident, where people called in and said, “Hey, I saw something bad happen, I think I can tell you what caused the accident.” These folks had started taking these reports and writing them down, and trying to put little things on maps to have an idea where this person is that called.

John [P.] Shannon, who was a flight director at the time, had been over in the Control Room andRoom and saw these people, called me and said, “Hey, come over here, see what these guys are doing. There might be something important there, and see if you can get them organized and figure out if there’s something useful.” Because remember, it was weeks before we really understood what caused the accident in the first place. There were lots of theories about the foam hitting the bottom of the wing, but we didn’t have good enough data to really know for sure what exactly happened. We were definitely grasping at straws, and this was one of those things.

After, I’d say probably a few hours, if not the first day of going through these reports, we settled into trying to find actual eyewitness accounts in which somebody that said they saw something. There were a lot of calls that you might say we disregarded as kind of kooky. “Hey, I understand when the birds talk, and the birds actually told me before the entry that this was going to happen, and I feel bad now that I didn’t call you and let you know.” That was an actual report that we got.

We tried sorting them out. The ones that are like that are over here in this pile, and we’re not going to go back and look at those again. Other ones that sound more solid went into another pile—like the people in California that said, “I looked up and I saw a white dot come off the Shuttle,” or, “I was looking through my telescope and I saw something.” After a few days, or not even a few days, probably within the first day or so, we got a report that said, “Not only did I have a telescope, but I had a camera connected to my telescope. I have video of this. Would that help you?” I quickly changed the priority and said, “Now, we go through every single report, we don’t just want eyewitness accounts. We want anybody that says they have that kind of video or photography. We want to get all of that.” Within a day or two, we had something like 30 different reports from people that said, “I have video,” or, “I have high-resolution pictures taken through a telescope.”

I then called in the cavalry. We brought in maybe half a dozen flight directors, another dozen or so flight controllers, mostly people from our flight dynamics area, the trajectory guys. We had ourselves a process, a system where as these reports would come through, we would sort them out. Every one of them that sounded like they might have video or pictures, we divided those up, and everybody started calling those people, and we started arranging to have those people ship us all of their equipment—well, originally it was just their videos to us. Again, this was before the days like today when you could just post an electronic image or an electronic video online and get it – that would have made it so much simpler. These people were FedExing stuff, overnight shipping, things like that.

As we started getting them in, I handed them over to my trajectory guys. We pulled in some imagery experts from Space and Life Sciences. They started looking at the pictures and looking at the video. The challenge that I gave them, not really knowing what we were going to find, was to look for “something useful.” Look for indications where we actually see these white dots coming off of the Shuttle. Of course the hard part was the Shuttle in the picture was just this white dot. It wasn’t like you saw a picture of the Orbiter, you saw this big white dot. But sure enough, in some of the pictures, you could see another little white thing coming off, some part of the Orbiter coming off, way before we had any indication that there was something wrong with the Shuttle, as the Shuttle was just crossing the Pacific coast over California.

We focused everybody on find all of those. As we found them, the thing I asked our trajectory guys to do is see if you can identify what that is. Try to do some trajectory analysis on it, as a minimum, to determine how big is this thing. Did we have sections of wing coming off? Did we have a piece of landing gear come off, or were these things little tiny pieces of tiles? As I asked our folks to do that, the response I got back was, “Well, it’s not possible to derive that kind of information from this kind of video. We don’t really know enough about the relative motion, how fast Shuttle was going. Even if we did, it’s not possible to make estimates on where these things are going, or the trajectories.”

Ultimately, what we were hoping to do is track these things all the way down to the ground, and send people out there to go find this thing, because whatever it is will tell us where the damage started. Because an individual tile has a serial number on it. If we had hold of an entire tile, it would tell us this came from the wing, or this came from the underpart of the spacecraft, or it came from the nose area.

I had three or four different teams, each of them focused on specific goals—there was one that was focused on just imagery analysis. There was another one that was then focused on doing the trajectory analysis, to determine where is that thing going and give us a footprint on the ground that we would send people out to search. Then another group of people whose focus was determining how big each object was. Then a whole separate team was engaged with the FAA [Federal Aviation Administration] looking at radar data. Once we had what we thought was an object’s trajectory, let’s go back and look at every radar that was looking in that part of the sky and see if someone picked that thing up on radar that might have tracked it all the way to the ground, rather than just using the analytical trajectory. This all happened, again, within a couple of days.

As we started getting more pictures, it became more and more clear we might be able to do this. Each one of the folks that was my main guy in each part of those teams, each one of them said that the part that they were expected to do was not possible.

By the following Saturday, and why this sticks with me I don’t know, but exactly a week after the accident, each one of those folks came in and showed me how they had figured out how they could solve their problem. Actually, the first breakthrough was [J.] Chris Edelen (a flight controller who went on to become a flight director) walks into this room that I was sitting in, He was just beaming, and he sticks a videotape in the VCR and hits play. I see the, now, “usual” white dot. By then I’d seen lots of these videos, so I can see we’re looking at Shuttle, I see the little white dot coming off. He’s still beaming. I said, “So, I don’t get it, what are you showing me?”

He says, “See that really bright spot in the image? That’s Venus. Shuttle flies right in front of it. We know exactly what time it is, because the guy that took that video had a GPS [Global Positioning Device] on him, and we also know exactly where he was standing. We know where Shuttle was in the sky, we know where that guy was standing, and we know where Venus was. We know exactly, to the millisecond, what time Shuttle went in front of Venus, between Venus and this guy. Because of that, we can now do trajectory analysis for that thing that just came off, and where it’s going.” Blew me away.

Wright: Amazing.

Hill: The only reason we even figured it out and started going down this path was I just happened to have the right trajectory guys and the right imagery guys from Space and Life Sciences sitting together looking at the video. As they were talking amongst themselves, they finally hit on, “Oh my gosh, look what we did.” From that, we then went back and looked at all the videos. As it turns out, we had a set of videos from the California coast all the way to the Texas-New Mexico border with complete continuity. We could correlate the time exactly in each video because of that first one with the Venus crossing. In the next video, we could see where these two videos overlap, so this is the exact time at this part of this next video. And the next, and the next. I had a whole team of folks, and all they did for a couple of weeks was figure out how to put all of those things together. Once they did it, every single one of those videos, every dot that came off, we could then do really tight trajectory analysis, and build those footprints to show where each object would be lying on the ground and then we could ask: Where is that line in the sky it’s falling through so we can go look at radar data?

After that first discovery, each these guys all came back, and said, “Holy cow, well, if he can give me that, then I can do that with my part.” These weren’t all my great ideas, it was from these guys that were on this team working for me. The very next day, I woke up, four o’clock in the morning, I sat up in bed and I thought, “Oh my God, this is it, this is how we’re going to do it.”

My wife thought I was having a heart attack or something. I get up and get dressed and drive into JSC.

The whole reason this work was important was every single day I was expected to brief the folks that were running the investigation on our findings. Of course, what they were expecting us to find was something in the video where we can trace this thing down to the ground, so we could then go get it. There was no real hope of that, because Shuttle was still 200 miles in the sky in these videos.

That morning I typed a four- or five-page presentation that summarized what we’re doing, what the data is that we have in hand, how we’re going to analyze it, and the end product we are going to create—we’re going to draw a map that shows the ground impact footprints for every single one of these things that came off before breakup.

About six, seven o’clock in the morning, my team starts straggling in. I think we might have had a seven o’clock tag-up meeting, and I had to brief the investigation team at eight o’clock in the morning. These guys all walk in, and I’m sitting there like this crazy guy, all hopped up on probably a half a pot of coffee, and said, “Everybody sit down and don’t say a word. I’m going to run you through this. You guys all tell me if this makes sense, and tell me if each one of you really can do your part that I’m about to tell the whole world that we can do.” I ran through it, and as I would start to describe some part of the process someone wasn’t a part of, they would shake their heads, saying, “Holy cow, that can’t be done.” Then the guys who actually were doing that piece said, “Yeah, yeah, that’s exactly right. We can do that.” We got to the end of the full explanation, and the whole team is looking at each other saying, “Oh my God, can we really do this?”

So I briefed the investigation team, and off we were running. Within three weeks, we had the map all the way from California to Texas, where we could show footprints on the ground where every one of those things landed. We had turned on tests with the Air Force Research Lab at Wright-Patterson Air Force Base [Ohio], because as we started trying to look for radar data, we found quickly that the radar experts needed to know exactly what the material was. They didn’t know what radar reflections looked like off of Shuttle, and especially off of the different components and materials from Shuttle. We reached out to JSC Engineering, grabbed samples of all of the materials on the outside of Shuttle, and sent a fellow from Engineering named Steve [Steven L.] Rickman up to Wright-Patterson Air Force Base with a suitcase full of Shuttle parts. Steve coordinated AFRL’s testing in the radar chamber. The team used that test data to work with the FAA analyzing FAA and USAF radar data in hopes of refining the ground footprints.

In parallel with that, the guys that were looking at radar data from the breakup itself, we were able to use everything we were learning and refine the analysis that they were doing. We got better and better, both pre-breakup and post-breakup. By the end of this effort—and this effort probably went on for about three months—we had footprints that exactly mimicked the area that they found all of the hardware in East Texas, the significance of that being if we had another accident, then in real time, we would be able to predict where the spacecraft debris is going to be on the ground, which didn’t really help us a lot in 2003. It made us significantly smarter about when things come off a spacecraft at 200 miles, where is it going to go, how do you track it, what’s the likelihood you can go and find it, and things like that.

On the day of the accident, we didn’t know how to do that. On the day of the accident, it took us a couple of days to have any kind of footprint, and I’ll bet about once a week for the next six weeks, we got significantly smarter about the math involved in developing those types of footprints. By the end of three months, we had the process down where we could do it exactly. It almost made us look like Nostradamus, it was so eerily accurate. The real significance is in every flight after that, we ran that model during entry, so if some bad thing happened next time during entry, the same flight controllers on console that actually figured this process out after the accident would have known instantaneously where we need to send the search and recovery teams. It’s incredible.

As it turned out, the value of the pre-breakup footprints out west ended up not being as great as it might have been. For one thing, the footprints were huge. Even the smallest ones were a mile wide by three miles long. Mathematically it was phenomenal that we were capable of doing narrowing it down at all. But imagine sending 100 people out to the badlands of Utah to look for something that might be the size of this cup, and it’s sitting somewhere in this three-square-mile area. Almost an impossible job. As great as the math was, using it to direct a search was almost an impossibility, because we weren’t looking for a wing, we were looking for small things, like pieces of a tile. But we were right on the cusp of sending people out to several of those sites as part of the investigation, because we didn’t have any other choice. When they found the MADS [Modular Auxiliary Data System] recorder on the ground, were able to download the data, it told us pretty clearly where the heat came from and what had happened to the Orbiter. So now there wasn’t really the need to continue trying to grasp at this straw using these large, ground footprints.

Phenomenal experience. It again reinforced that notion to me that the impossible challenges that scare us off are only impossible if we keep trying to solve them the same way we’ve always failed. I had yet another case where a bunch of smart guys working on my team figured out how to solve things that each one of them thought was impossible, and they did it in less than a week. Then we made it look like it was easy. Yes, incredible stuff.

Wright: It’s always the end of that story with space business, isn’t it? It always looks easy to someone who wasn’t involved in the inner part of it.

Hill: Yes, once it’s not impossible anymore. Yes.

Wright: At what point did you start to move into the new responsibilities of helping with the STS-114, and/or the whole mission of Return to Flight?

Hill: A month into the accident investigation, at the end of February, I was assigned to be the lead flight director for STS-114, which was the Return to Flight mission. At the same time, I was assigned to lead a team that we called the Vehicle Inspection and Repair Team, and they overlap, but they really were separate assignments. The Vehicle Inspection and Repair Team was all about on our next Shuttle launch, assess whether or not it’s possible; to detect damage on the outside of the Shuttle; to get astronauts outside to all areas of the outside part of the spacecraft; and then to repair damage. If we get struck going uphill again, we want to be able to get an astronaut out there to put hands on it and fix it. So assess whether or not that’s possible. I was assigned to do that probably within a couple of weeks of the accident. So I was juggling that and this accident investigation team that I was already leading, and now Vehicle Inspection and Repair, both starting in the middle of February.

On the vehicle inspection and repair side, we started with simple things like what does damage look like on the outside of a Shuttle? Especially in the flight control world, we didn’t have a lot of experience with that. Our job was to bring spacecraft back unhurt. Whatever minor damage we tended to land with, a bunch of guys at KSC [NASA Kennedy Space Center, Florida], repair it and get the Shuttle ready to go again. We always were handed almost pristine Space Shuttles. We had to go off and look at a long series of things. What does damage look like on the leading edge, what does it look like on the nose, what does it look like on the blankets, what does it look like on tile? How do we take pictures of it? Can you see that damage in pictures from the ground? Can you see that damage in pictures from the [International] Space Station? Can you see that kind of damage with the camera on the end of the Shuttle arm? There were all kinds of other questions, like how small of a damage would be critical, such that if we don’t see and repair it, it could still kill the Orbiter and kill the crew that’s relying on us? We have to be able to see that. Just being able to see something the size of a breadbox isn’t good enough, unless anything smaller than that is okay.

As it turns out, some areas, it was okay to land with damage that was three inches wide. In other areas, tolerable damage was as small as a quarter inch to one inch, at least that’s what we thought in February of 2003. By May of 2003, the areas that we previously thought could tolerate a quarter-inch hole all the way through, were determined to be capable of tolerating only a tenth of an inch, and if there’s any cracking around it, all bets are off. We need to be able to see things as small as a tenth of an inch, and we even need to be able to see cracks in the leading edge and in the nose.

We were inventorying all the cameras we had to determine if they could see that small of an artifact, on these materials, and in the light we expect to have in orbit. Sort of like with the radar material in the accident investigation, we had to fire up a bunch of tests. We pulled out every camera in inventory and did all kinds of tests to make sure we understood exactly how good the images it could see of the materials on the outside of the Shuttle, in the expected lighting conditions that we would see in orbit, and with the Shuttle robot arm, because the conventional wisdom for most people was that the arm reach everywhere and see everything. As it turns out, it could see almost everything, but it couldn’t see very much at a fine enough level of detail to tell us that the Shuttle did not have critical damage. You could see it, but you might be looking right at damage that would be too severe and not recognize it. It took us a couple of months of testing cameras before we knew that.

We had maps of how small or large a damage we can tolerate on all parts of the Shuttle. We had other maps for camera resolution any camera we had in inventory, on Shuttle, Space Station, or on the ground. When we combined them, they revealed the parts of the Shuttle we couldn’t inspect well enough to make sure that we see entry-critical damage.

As we started going after this inspection thing, we had a different group of people who were looking at how get an astronaut to any part of the Shuttle. The only methods we had to get astronauts to the bottom part of the spacecraft were scary. The least scary involved taking a bagful of stuff connected to a tether, an astronaut in the payload bay and having him throw this thing over the wing. It would fly around the Shuttle, literally, like a tether ball, and the astronaut would catch it on the other side. They would shinny their way around. We had that in case we ever had to manually close the ET [external tank] doors, which we never had to do. We knew that wasn’t going to be good enough, so we started looking at different ways to get astronauts underneath, whether we fly them down with a backpack, or different methods of putting them on the end of a boom on the robot arm.

We evaluated probably half a dozen or more different options. Some of them were as simple as just a boom on the end of the arm, some of them were really complicated—Big deploying mechanisms, big balloons that you would deploy, and shinny up, the backpack idea. We ended up settling on the boom, the OBSS [Orbiter Boom Sensor System]. We also then settled on putting a couple of different cameras and lasers on the end of the OBSS. Thankfully all cameras and lasers that had already flown on Shuttle, we knew could tolerate flying in space, which simplified the development. The cameras and lasers on the end of the boom, we could get them close enough to see everything at high enough resolution that we needed to see.

We had vehicle inspection figured out. We had a method to get astronauts out there. We were also chasing after now how do we repair damage in orbit. This was all February to June 2003—we were going really fast. Several engineering teams all operating in parallel, going into different labs, testing all this stuff. Everybody was dubious about the repair part, because it had been studied back in the late ’70s, early ’80s. Shuttle Program spent, at that time, I’m going to say it was $20 million, which in 1980 was a lot of money. At the end of that study, they concluded you can’t repair tile, it’s not possible, because when it breaks, it’s this powdery stuff, kind of like Sheetrock, and you can’t adhere anything to it, it just peels right off. The expectation from the Shuttle Program was we would come back and have some inspection method. We might figure out how to get an astronaut close enough to make a repair, but we would end up coming back and confirming that we can’t repair tile. If we damaged another Shuttle severely, we’re going to have to leave the astronauts on Space Station and go rescue them another day. That was their expectation. They didn’t come right out and say that publicly, but in private conversation it was accepted that it wasn’t possible, and that my team was supposed to exhaust all options, and if that was the answer, we were to come back and say so.

The expectation was that we’d have some hard conversations and decide we were still willing to fly, even though we couldn’t repair damage, and then we’d launch again in September - October 2003. Seriously, September - October of 2003, that was the goal. Those were the marching orders I was given, “Work your ass off. Get these answers fast, because we’re going to turn it around, and we’ve got to be ready to go fly. Don’t pull together some five-year study or research and development project, that doesn’t help us. We’ve got to fly now. We’re not finished building Space Station.”

Lo and behold, we did some tests, and we looked at the reasons why the study back in the late ’70s failed, and some Boeing and Lockheed materials scientists, working separately—both of which were working with some JSC Engineering materials scientists—had some early luck by looking at exactly what failed before, and they tried a few things and tweaked some of the formulas that had been tested before. By God, it started to work. We didn’t really even understand why at first, but the fact that it was working at all was exciting. Right now much of our community was really not happy. Because we couldn’t say why it was working, they didn’t want to accept it.

I’m the more pragmatic engineer. The fact that we don’t understand all the nuances of the tool doesn’t necessarily mean we can’t use the tool. We ought to keep trying to figure it out, because the fact that we don’t understand it may mean in some conditions we expect it to work, it won’t work. It’s better for us to know, but the fact that we had some early success got me very excited. It’s like having one of my experts in some other study come back and say, “I think I’ve figured out this one thing that’s no longer impossible.” Then, my whole approach was, “Okay, what part of the tests are failing. What things are we not happy about, what part of this material is the community skeptical of?”

The tile repair material was kind of like a caulk. You mixed a couple of chemicals together, in room temperature they would turn into rubber, and if there was a hole in the tile, it would fill the hole and insulate. Beautiful. It worked great in a lab environment.

There much skepticism that it the repair material would cure at all in a vacuum. I immediately had people on the team thinking, “Oh great, now we’re sunk, everybody thinks it’s not going to work in a vacuum.”

I said, “Guys, don’t we have a vacuum chamber? Let’s get it in the chamber and see.”

They ran a couple of tests, and within a couple of days, damned if this stuff was not curing in a vacuum. We’re not exactly sure why it’s curing in a vacuum. Although the materials scientists closest to it had a much better idea, most of the rest of the community didn’t.

There were a number of other concerns like: It won’t work at orbit temperatures. Even if it does, there’s going to be air bubbles in it, and with the air bubbles in it, it won’t perform right during entry, because it will burn right through all of the bubbles. Again, every time we heard one of those, most of the community’s initial expectation was, “Damn it, we’re sunk. We can’t get there.”

I said, “Wait, not only do we have vacuum chambers, we have thermal vacuum chambers. Let’s get this thing in a thermal vac chamber, and let’s make it cold.”

“We’re also worried about it when it’s hot.”

I said, “Okay, let’s get it hot, and let’s do it then.”

Day after day, reports would come in and say, “You know what? It’s still working. We’re figuring it out.” Air bubbles. Let’s take some, let’s cure it at a cold temperature in vacuum, let’s stick it in the arc jet and let’s see what happens. Does it burn all the way down, or not? It didn’t.

We worked all these things off, day after day after day, and by July 2003, here we are, three to five months after the accident, damned if we didn’t find a solution to every single part of that problem that had been declared impossible, except for RCC [Reinforced Carbon–Carbon] repair.

RCC is the material on the wing leading edge and the nose cap. There were some promising materials, but solving that from a temperature and a material perspective was so significantly difficult, it actually took another two years before there was a reliable repair method for some relatively small RCC damage. If it was anything big, all bets would have been off.

Although it took another couple of years to develop an RCC repair method, we already had a much better understanding of how much of damage could we tolerate and still land. On the tile side, we could absolutely detect critical damage, and we could absolutely get an astronaut to any location on the outside of the spacecraft to make a repair.

All of that stuff was wrapped up by July, and I presented it end-to-end to the Shuttle Program. The Program Manager accepted took each one of those recommendations. Then as the Program took on manufacturing the tools, the boom, and repair materials, I shifted gears from studying all of this to figuring out how to do it in flight. It was a little like, “Okay, smart guy. You said this was all going to work. You’re the flight director that’s going to have to lead the team to figure out how to use all that stuff, and then demonstrate it on the first flight. Good luck.”

So I spent the next two years doing just that, leading the ops team, turning all of those things into real operational tools, with the procedures and everything behind them, that we could then go off and execute. Technically, that part was easier than the original engineering effort. The political fallout went on for the next couple of years, certainly all the way through STS-114. Some of it lingered even after STS-114, and there’s bad feelings to this day in part of the community from some of that effort.

For the next two years, we continued to fight and refight every one of the uncertainties. For example, we would continue to come back and reargue whether the tile repair material would cure in vacuum, even though we had test data after test data from vacuum that showed that it was working. Because it didn’t fit with the going in opinions of some of the community (who weren’t part of the team), it didn’t fit with their understanding, and they could not accept the individual pieces of the things that worked. We went all the way up to the time we flew with that being something that we were fighting behind the scenes continuously. It was bad and it became very personal.

I had folks in the community, in particular from the Astronaut Office, that were e-mailing and calling the press and the CAIB directly, and using my name. I wasn’t the only one, there was a couple of folks from the Program Office too. But using my name, saying that I was, “in cahoots with the Program Manager to gloss over all of this stuff and make it look like it would work, even though everyone knows that it won’t, because all we care about is flying again, and don’t care if we kill another astronaut.” Which just broke my heart, and it infuriated me. In fact, it made me sadder than it made me mad. I thought, “How could these people actually think that, as much as I or any of the rest of us were pouring our heart and souls into doing this, that all I really care about is flying again, and I don’t care if any of this stuff works?”

I’ll tell you, from February of 2003 till we flew in August 2005, I worked probably 70, 80 hours a week, had virtually no family life at all. I was completely dedicated to this and making sure we were doing it the right way. Boy, we fought that stuff all the way to the bitter end. Even after we demonstrated all the equipment, all the procedures, even the tile repair materials on STS-114, there were folks in the community afterwards who could not let go of those animosities, and felt like some part of the community, including me, if not especially me, had cooked all of this up just to fool everybody. Crazy, huh?

Now, what is the other thing that stuck with me? Same recurring thing you guys have already heard from me, and that is this stuff was all supposed to be impossible. Talk about the ultimate consequences. First, we’re never going to fly again, but more importantly, think of what we owed the STS-107 crew. Go understand these things that we gave up on before, understand them so that if we ever put another crew in the position that they were in, 1) we’ll know it, and 2) we can do something about it. By gosh, we did. All of those things that we went after were supposed to be impossible. Within five months, most of the impossible stuff we had already solved. It turned out it wasn’t so impossible. That was huge.

When it was over, I was expected to leave NASA. In fact, the Center Director, Beak [Jefferson D.] Howell, pulled me aside after the flight, told me what I great job I had done, tremendous leader. He said things to the effect of greatest leader in NASA. He probably didn’t say greatest leader in NASA, but he said things that made me feel good to hear from the Center Director about my leadership. He said, “But you left a lot of bodies in the wake. I’d say your NASA career is probably over, you need to go find something else to do.” That was after the flight that was completely successful, and all the things that we’d been fighting about, it turns out my team had been right.

By then, that two and a half years had been so hard, I didn’t care. I was going to leave. I thought, man, if this is what I get from this community after putting out this kind of effort to do what I thought was the right thing, I do need to go do something else. A friend of mine, Steve [Stephen C.] Doering, was the manager of the EVA [Extravehicular Activity] Office at the time. Steve says, “Hey, I need a deputy for a year, because my deputy is off on this rotational assignment. Come fill in for Glenn [C.] Lutz as my deputy for a year while you figure out where you’re going to go. Once you’ve figured it out, Glenn will come back and then you can leave NASA and go do whatever you’re going to do.”

I said, “Sure, I’ll go do that.” It was good with me because it separated me from the community that I’d been off fighting with, so I didn’t have to keep fighting those fights.

After a few months, Steve, as a good boss, says, “Hey, there’s all this professional development stuff you haven’t done. I’m going to start assigning you to go do these professional development things.” They were things that, for the most part, MOD [Mission Operations Directorate] management, and certainly the Flight Director Office, generally look down our nose at. Our attitude is we do the hardest stuff there is, we groom our own. We don’t need any help from anybody else. By the way, any of that other stuff that we don’t know, promote us to some other job, and after a couple of weeks I’ll know that too, because the hardest stuff I’m ever going to learn is the stuff I had to know to be a flight director. I’m not saying that attitude was right, but that was certainly the prevailing attitude.

As Doering is sending me off to do this stuff, I didn’t really want to do it, but I figured, sure, I’ll go do it. I went to Wallops [Flight Facility, Wallops Island, Virginia] to MEP [Management Education Program], and I’ll never forget. I came back from MEP—changed attitude. Actually, much of the animosity, the bodies in the wake part of the Return to Flight, I had a much clearer understanding of the human effects of things that led up to some of that. Much greater awareness of my own blind spots, as well as the blind spots and communication challenges with other parts of the community, that rather than me just always wondering, “Gee, I wonder how those guys ever could have reacted that way,” I had a much better understanding, and a much better understanding of how I would lead as a manager in some future role, while being more aware of those things, and maybe being able to control them more. Not that it was all my fault, but just being aware of it would have made it easier for me to negotiate my way around it. Some of it might have been damped out.

The only reason I tell you all that is six months into this, [G.] Allen Flynt, who took over as the Director of MOD calls me up, and he says, “Hey, I need somebody to run Shuttle Operations for us. Bob [Robert E.] Castle is going to go off to the Program Office. I want you to come back to MOD,” which I did not want to do. This is six months after the [STS-]114 where the Center Director told me I was persona non grata, I need to go do something else. This would have brought me right back to the table with the Shuttle Program, and the different constituencies of the Shuttle community that I had had such a hard time with, and that had such great animosities towards me.

I told Allen, “No, I think if I did that, I’d be a lightning rod, and I don’t want to cause bad feelings with the program, and bad feelings for MOD. I don’t know how I could help but do that.”

He goes, “Well, okay, I guess that’s okay. You once told me that you felt like you owed MOD a lot for the opportunities that you had, but if you feel differently about it now.”

The next day, I showed up on the eighth floor as the Manager for Shuttle Operations for MOD. I expected it to be bad and ugly, because of my Return to Flight experience. Within I bet a month, probably less, Allen had folks from the Shuttle Program and other parts of the community coming to see him saying, “Oh my God, what happened to Paul Hill? He’s like a different guy. He used to be such an asshole.” Actually, some of them would come tell me that. “Oh, you’re such a good guy now. You were such an asshole.”

“Oh, thank you. Thank you.” A great part of that was some of this awareness I got in going up to MEP. (Which, by the way, is why it ticked me off so much when NASA eliminated MEP. I thought, are you kidding me. I’m the poster boy of strong leaders who did great things for the Agency that they would have just cast aside, that I then ended up coming back and being a very highly regarded leader in the executive ranks, and it was only really possible because of MEP.)

I ran Shuttle operations for MOD for a year and a half, became the Deputy Director of MOD, and then six months after I was Deputy, I became Director of MOD. The irony wasn’t lost on me that it was just over two years after I was told my NASA career was over, now here I was being selected by the next Center Director to be the Director of MOD. I’ll tell you another thing, the one thing that stuck with me in all of that, it was never lost on me that as the leader, whether it was the flight director, or an XA [EVA Office], or a Shuttle Ops manager, any role that I had, my job was to catalyze the discussion for the team, help us identify what’s getting in our way that actually is resolvable, and figure out how to either solve it or go around it to get to what it is we actually need to do.

The folks that are on the team, certainly as a group, are all smarter than I am. Many of them, if not most of them, are probably individually smarter than me. If I can harness them as a team, if I can focus them as a team and help find the things that they’re accepting are impossible that maybe aren’t, they will figure out how to solve what’s supposed to be impossible. If I can help them understand, they will solve it. Whether were doing rocket science or managing the directorate as an executive, that’s always been my approach. It’s never failed me.

Wright: The direction of the Shuttle Program took a turn while you were working with your team to Return to Flight, and that was the Vision for Space Exploration when there was an actual deadline set for when the retirement of the Shuttle Program would be. Give us how that impacted what you were doing, and also in your new roles of your participation, in the closeout of the 30-year program.

Hill: Our first focus in MOD was now that everybody knows there’s a sunset clause, this is when Shuttle is going to end, how do we get there and not lower the bar? That’s generally the thing that scares MOD. Whenever you talk about changing anything, “Oh my gosh, we’re going to lower the bar, we’re no longer going to be MOD.” It was a very valid concern, because 83 percent of the MOD workforce was contractor, which means 83 percent of our flight controllers, 83 percent of the people training astronauts, developing the plans for the mission, maintaining the computers, were contractors, not NASA [federal government employees]. Which means the ones that were working Shuttle all knew at that moment that half of them didn’t have a job at the end of the Shuttle Program.

At first, it wasn’t worse than that, because Constellation [Program] was going to essentially replace Shuttle. But we had committed to flying Constellation missions with half the people it took us to fly Shuttle. We didn’t just make that up, we had ways we knew we could do the job differently for Constellation, and it would take fewer people. The idea for the fewer people, of course, is most of the MOD cost is labor, it’s people. If we’re trying to save the program money, we have to figure out how do we do the job with fewer people. How do we do it with fewer people and not lower the bar? We still want to deliver the same mission reliability, mission success, safety for the astronauts, whether we use 1,000 people or 500 people.

The challenge became worse when Constellation was canceled, because all of our contractors who were working Shuttle knew they were out of a job. We said it just like that to them, from the top of MOD, we never tried to sugarcoat it and say, “Well, something else is going to come along, you guys are going to be good. Just put your heads down.” I understood, my people had families to feed. They had kids to put through college. I can’t talk to them like that, and besides, if I talk to them and say something that they and I both know isn’t true, I lose credibility. I need these people to believe everything I say to them, every time I say it to them, because I also need to know they’re going to tell me even the harsh truths, every time I ask them a question. I could only count on them doing that if they count on me doing it for them.

At my first MOD all-hands after Constellation was canceled, the very first thing I said to them, “Guys, I can’t tell you where this is going. I can tell you how bad this can be for all of us, and for some of us individually, as well as an organization. Half of the directorate is at risk here if all we’re left with is Space Station. Here’s the thing we have to focus on in the next 6 months, the next 12 months, so that we are not letting down the astronauts that are trusting us, that are flying next month and 3 months from now. That’s all that has to matter right now.”

Our job has to be keep our eye out for signs that we are lowering the bar, signs that we’re now just going along and getting along. At the working level, if anybody thinks that yes, the management just made this decision and we all know it’s stupid and it’s going to get the astronauts killed, but that’s just where we are now. If people start hearing those things, I need to hear that right away, because we are never going to go along and get along. If we get to a place where we’re getting close enough to that last Shuttle flight that people are now leaving, and we’re starting to do dumb things, or we can’t do the right technical rigor on the work that we’re doing, that we now can’t step up at FRR [Flight Readiness Review] and tell the program, “Yes, we’re safe, we are still MOD and we’ve still done due diligence,” I will personally tell the program in public, “We are no-go and can no longer fly Shuttle.” We actually talked that way at our very first all-hands.

Our focus throughout, even before Constellation was canceled, when we knew Shuttle was going to close, our focus was on reassuring the people down the line that that’s our attitude. We will back them up on anything if they think that we are now choosing to not do something that is absolutely required to keep the astronauts safe. There’s nothing we won’t go to the mat and defend them on on that subject. Which is different than no, we have to keep everybody. We can’t just trim down the workforce by 10 percent as we’re getting close to that last flight. I hate the personal impact that it would have on my people, but that doesn’t necessarily kill astronauts, and our first responsibility is protecting the astronauts, then protecting the spacecraft, and then getting the mission done. At the risk of sounding like I wasn’t loyal to my folks, but you have to be able to treat them separately intellectually.

Our folks got that right away, from that very first MOD all-hands when we talked about Constellation being canceled, and that was truly a chaotic and scary and dark time for us. Oh my gosh, half of our customer base just went away, half of our workforce is just going to be gone, and with it, expertise that doesn’t reside anywhere. There’s things that we are going to now unlearn as an agency when we let go all these Shuttle people with ascent and entry experience.

Throughout that whole timeframe, the MOD workforce absolutely trusted that we would do the right thing, and would not sell out on the things that were most important to protecting the astronauts. They also understood that many of them were going to be getting laid off as we got closer and closer, and they did not let that affect their attitudes towards doing the right thing, it did not affect their trust level for us.

When it got around that I had been that candid with MOD, much of the management at JSC, including a lot of my peers, thought I was crazy. I thought, I don’t know how you’d not do that. These people, our workforce at JSC, if not across the Agency, is significantly smarter than some managers sometimes give them credit for.

I think sometimes as managers we forget “they” is “us.” Where do you think we came from? These guys are smart, that’s why we have them here, and they’re doing incredible stuff, which means if we don’t just come right out and tell it like it is to them, they’re going to start filling in the blanks on their own. They’re either going to come up with a darker, more nefarious answer to something, or they’re just going to lose trust for us. They’re going to think, okay, Hill knows, and he’s just not telling us. Or, he’s just trying to play us, we can’t trust this guy to be straight with us. We owe them better than that. Hell, we owe them to talk to them like adults. Hey guys, when this ends, actually in our best outlook, half of our Shuttle workforce would have been gone at the end of Shuttle, even if we had progressed into Constellation, which is still 25 percent of the MOD workforce, still an enormous management challenge.

Most of them—actually all of them knew that. I didn’t really buy anything by not telling them that right up front, and it bought me all kinds of trust from the workforce, which to tell you the truth I didn’t do in order to try to goad them into trusting me. I did it because I thought it was the right thing to do. What I really didn’t want to do was risk having them stop trusting me, and to start wondering, all right, what story is he going to try to spin us on now.

They did tremendous—it’s funny from the time Shuttle retirement was announced until the end of the Program, on STS-135, MOD’s biggest fear, like a lot of the Program, was people are going to start bailing out and eventually we’re not going to have enough people to do the job. It never happened. Our Shuttle folks worked right up to the last minute, every single one of whom had already gotten their pink slip, they all knew they were out of a job the day after landing. Our attrition never went up on the Shuttle side. If you were to look at our data, just the personnel records, and just mining it impersonally, you would assume that we had hidden all of that from our Shuttle people and then surprised them with the layoffs. Our attrition—in fact, not only did it not go up, it went down a little bit as folks hung in there, and wanted to be part of making it work and doing it right all the way to the end.

Really inspiring. As an executive who knew that when it was over, I was still there and still had a job, and here I am being proud of my folks that are doing such great work and being loyal MOD family members, right up until the bitter end. All I can do is offer them a handshake and a hug. It was quite inspiring. Also breaks the heart.

Wright: In the midst of all this, you had a Station to complete.

Hill: And a Station to keep flying. While the MOD management team is, like everybody else, in a certain amount of chaos trying to figure out what’s our best strategy, how do we do this, we also have this Space Station that is still up in orbit 24/7 that we can’t take our eye off that while we all sit and bemoan the fact that Shuttle is coming to an end, or the Constellation is being canceled. We have to keep showing up every single day, still, on Space Station. We talked that way in MOD management ranks from the beginning. I pulled the directorate staff and the division chiefs all together, first when Headquarters announced we were going to retire the Shuttle, and especially when Constellation was canceled.

I got us all in a room and said, “It’s just us. What are we worried about?” There was a lot of teeth gnashing, and as I tell people all the time, and it’s not facetious, I would say that there was some emotional discussion for maybe 20 minutes, and some discussion about, “We’ve got all these smart guys, we just need to go get any old work that we can, it doesn’t matter, because our guys can do anything, so we just need to go—we’ll go to oil and gas and see if they need us to do some analysis for them on oil and gas crap. That isn’t the work that would preserve critical expertise in MOD, but it’s work that we can get for our technical guys.” The idea being the more odd jobs like that we can find, it means that’s one less guy that’s going to be laid off. Like I said, we might have spent 20 minutes talking that way in MOD.

After that 20 minutes, I brought the discussion back to what we were at most risk of failing with Shuttle retiring and Constellation being canceled. In what technical areas might we actually lose all expertise? We’ll have so much attrition, we can’t do some facet of spaceflight anymore. When and if the Agency decides we’re then going to go back to the Moon, or go to Mars, or do some other thing outside of Earth orbit, which things are we most at risk of not being able to do? What work do we have to capture to prevent that from happening? Which is different than how do we go get work to keep one more person, or ten more people, or a hundred more people on the roster. My principal job was on the capability, it wasn’t having 10 more people or a hundred more people. In fact, for some time, we had been working to have a hundred less people, because a hundred people is $20 million that we just gave back to the program every year. If I could do the job with a hundred less people, I owed it to the program to figure out how to do that.

We had that discussion on what kind of work should we go get the first week of February 2010, immediately after Constellation had been canceled. Very quickly, what we realized is if anybody’s considering other space operations besides Station, we need to go see them, and we need to figure out how do we get those people to hire us to do their job, whether it’s another NASA mission, or a military mission, or a commercial mission, whatever it is. Who is it that’s going to still be flying in space, still going to be launching stuff? How do we approach them? And let’s go get the job. What made that more complicated is we were told from [NASA] Headquarters [Washington, DC] we weren’t allowed to seek business from the potential Commercial Crew [Program] companies.

You might speculate that there was an intent to put organizations like MOD out of the business, because there were folks at Headquarters that had decided the commercial industries could do everything not only cheaper but better than some crummy old NASA organization. There was no way a big organization like MOD could ever be cost-effective, and industry would always be able to undercut us at cost. For that reason, they didn’t want us going off and trying to get business from any of those guys. To be fair, part of the fear at Headquarters was that the parts of the Shuttle and Constellation programs that now just lost their customer, instead of closing down obsolete and expensive facilities that no longer served us very well, would find some sugar daddy in industry to use us. Now, all of that old stuff that some of the folks at Headquarters had hoped we were going to close and get out of so NASA didn’t have to continue paying pay for, would instead be it alive to support these commercial guys, and NASA’s bill stays really high. Thus, we would make commercial more expensive because some company “screwed up and went to some government organization that clearly isn’t going to do it as well or as cheaply as they could have done it.”

For me, it was like the Return to Flight all over again, because the folks that had those attitudes, I could never get them to engage in open formal debate. I understood most of their fears and considered them valid. I just wanted an opportunity to address each one of those fears individually. I thought the right strategy would have been for every NASA organization that thinks that they can provide service to, say, Commercial Crew, should have to show how we mitigate that cost concern from the Agency level if our organization is hired by a private company to do their work. How do we keep the Agency from incurring a bill for obsolete infrastructure? How do we keep that from driving up the cost of that Commercial Crew company? I thought MOD had good answers for all that, but the Agency did not want to hear it at all. I was told again not to pursue it. Like a good MOD guy, or a malcontent like me, I then scheduled some meetings offline with every credible industry player that might actually go after launch and entry work commercially. I started having these discussions by March, less than a month after Constellation was canceled.

My question to all of them was, “Could you tell me why you would not entertain MOD planning your missions, training your astronauts, and flying your spacecraft for you? I’m not saying because the government is going to shove us down your throats. Just tell me what your perception is and why you wouldn’t want to entertain this option.” I expected to hear some of the same negative views back from them that I got from my own NASA Headquarters. “You guys are too expensive, a bunch of old white guys with flattops and white shirts and pocket protectors. You do everything the way you did back in the ’60s, and you’ve wrapped yourselves in the golden years of Apollo.” They would say this to somebody like me, and I’m thinking, “Golden years of Apollo? Hey, I’m the guy that invented how to build and operate Space Station. Screw Apollo.”

What I got from the industry guys was much different than my expectations. What I got from them was, “Oh my God, if we could afford you, of course we would have MOD handle our operations. Nobody is going to ever be as good as MOD. You guys are the best. You guys are perfect in everything you do. But we can never afford you, you’re this marching army.”

I said, “Really? How about I come talk to you about 1) what our history has been in the last 10 years, and whether or not we really are a marching army, and what we’ve done to reduce costs, and 2) how about you give me an idea of how many flights per year, and about what your spacecraft looks like? Give me an opportunity to tell you how much it would cost for us to do the job for you, the same technical job we’ve done for every program we’ve served. Not necessarily doing it the same way we’ve always done it, but holding the bar at the same level. You’d, no kidding, have MOD protecting your operation.”

They said, “Well, that sounds interesting. We’d be willing to talk about it, anyway.”

I had a group meeting in April 2010 in Colorado Springs. It was me, Scott [Q.] Hartwig, who was a senior manager with USA [United Space Alliance] at the time, (before he was CEO), and then half a dozen representatives from the major, credible companies that were either already going to compete for or were considering competing for Commercial Crew. I gave them some of the feedback from those telecons. I said, “Look, let me give you guys a different idea about how much it would cost, how MOD would do the job.” Within a few months, I had Boeing and Sierra Nevada both approach us and ask us to give them formal bids, sort of like a competitive bid process. ATK came to see me privately and had started some of those conversations before they ended up dropping out of the competition.

We pulled together informal proposals that for Boeing and SNC and said, “Look, if you’re going to fly at this flight rate, and this is about what your spacecraft looks like, this is how much we would cost to do your mission planning, your astronaut training, and your flight control. This is how we will get you into the Control Center and do the job for you. I’m not going to rent space to you in the Control Center. MOD will provide the service to you in our Mission Control Center, and this is what it would cost.”

Boeing and Sierra Nevada both went forward in their proposals to NASA with MOD as their operator, essentially the full MOD role for them to provide Commercial Crew services. Sierra Nevada, as you probably know, wasn’t selected in the most recent round of the competition. Boeing was. Boeing announced publicly that MOD at JSC is their operations arm for CST [Crew Space Transportation]-100 operations. How about that?

How did we get there? Second in importance only to the quality job industry knew MOD would perform was the cost. MOD had made so much progress in the previous three or four years in process improvement that we made Boeing and SNC offers they couldn’t match on their own.

To MOD, we took the chance against the direction from Headquarters because if we didn’t capture business like that, there was expertise that we knew that we were going to lose. Some number of years in the future when the country changes the political winds for whatever reason and decides, “Well, instead of putting you guys out of business, we want you to go do this new thing,” we had to still be able to do it. I felt duty-bound to pursue the commercial work, if I could do anything to affect keeping those things alive in MOD, because I knew MOD culture. I knew what it meant when we engaged and held the bar high. These industry guys, some of whom had never flown in space at all, they may be really good, they maybe have really smart guys, and 10 years from now, 20 years from now, they might have a great track record. How would I let go of that responsibility and assume that they can just do it? I don’t care that there’s guys at Headquarters who are willing to take that chance.

In fact, I had a Headquarters guy tell me at the end of 2009 just that when he and I had a private conversation. Actually, it wasn’t so private. We were standing in a crowded auditorium lobby, and it became less private because I started yelling at him.

This was before I knew Constellation was going to be canceled, so for this argument, it was just for this commercial crew strategy by itself. I said, “If we’re not allowed to at least pursue these guys, then I’m concerned about how good a job they’re really going to do, I’m concerned about us losing some things on our side.”

This Headquarters guy’s response was “Well, if we try it and it doesn’t work out, no big deal. It didn’t exist before the ’60s anyway. You guys invented it then. If we try this and we screw it all up, then we’ll just stand MOD back up and you guys can reinvent it.” That’s when I started yelling at the guy.

I felt like my duty as the executive responsible for these things is to keep fighting that fight until somebody that I work for up the chain says, “That’s it, we’re done, we made the decision and it’s over.” It’s my job to keep trying to make them aware we’re giving up more than just a few people doing a job, there’s something more important. This is a national asset. Until we know somebody else can do it and do it the way we expect them to do it, to protect our astronauts, I’m not going to give up the good fight, which included I’m going to go off the ranch and I’m going to go chase this business if I can get it. Fortunately, once it became public that I was doing it (and that these companies had wanted to sign us up), we had also been successful in some other behind-the-scenes missionary work with some folks at Headquarters. While those folks at NASA Headquarters may not have been helping us, they were at least listening, if not sympathetic to the MOD point of view, and gave us an opportunity to make the other side of the argument.

At pretty much the eleventh hour, we got the right support and the right decisions from the key folks at Headquarters that allowed us to sign the Boeing and SNC agreements. I’ll tell you, there was a couple-year period while we were pursuing all of this that I expected any week to be given a call to cease and desist, and not only that, to pack up my crap in my office and get out of NASA’s office. The fact that we were allowed to sign, I still couldn’t believe that they were allowing it to happen. It was one of my prouder achievements.

Wright: I can understand that. That is quite an achievement, working on all that together. Going back to one of the statements you mentioned about how important it was to keep Station flying and flying safely, as your role, you were now in the position to select leaders that were going to come up through MOD, knowing that you didn’t know what the future would be, as far as where they were going to be, years from then, and how MOD was changing. Share with us what you were looking for, as you were pulling new flight directors in to run Station, and hopefully now to take on this new venture.

Hill: I’ll tell you, my focus wasn’t flight directors. John [A.] McCullough, who became the Chief of the Flight Director Office early on when I became Director, and then his deputy who is now the Chief, Norm [Norman D.] Knight, they had a great focus on the folks they were bringing in as flight directors. My focus was a little bit different. Not that I didn’t still have great value in that, but I knew they had their arms around it. In those areas, we were still doing everything right.

In the 2006-2007 timeframe, we started talking at the directorate level about leadership values, and how we managed as a directorate in the management ranks. What became more and more clear, and what became more and more accepted by the folks who worked for me (division chiefs and the directorate staff), was the awareness that MOD didn’t manage the same way we ran Mission Control. When I say we didn’t manage the same way, in some ways that’s a good thing. Running the organization isn’t the same as leading a team of flight controllers, or leading a technical investigation.

There are different things you have to emphasize in a management role. What we had lost, though, was some of our values. This notion that every decision we make affects our ability to protect the crew, protect the spacecraft, get the mission done. The fact that we needed the whole team to be fully transparent, that from one division to the next division, there had to be nothing hidden. That if we’re having a problem in some work, that problem is important to the whole directorate, even if it only exists in your division, and it only directly affects your division. If you guys do something that causes the performance in your division to decline, that now impacts MOD’s ability to protect the crew across the board. Even though that problem resides in your division, and your guys are working it, the risk affects all of us. All of us need to be aware. You need to be listening to your peers. Some of your peers might have experience solving that same problem. You need to be open to hearing that from them.

In 2006, that was not normal behavior in MOD management, and it hadn’t been for a long time, for over 10 years. Allen Flynt had started making inroads into that. He was Director from 2004 to 2007, and I replaced him when he left NASA. He had started making progress in those three years. Most of the progress Allen was able to make though was to identify which of the old guard, guys that had been entrenched senior managers when Allen came in and who were not going to change their attitude. Attitudes like, “I don’t need to share things with the other division chiefs, I don’t need to share with my peers, I don’t need to share with the damned directorate. I’m going to manage my own problems, you guys leave me alone. By the way, if I share some of my problems with my peers, they’re going to use it to get a leg up on me with the boss. If I share it with the boss, all it’s going to do is 1) piss the boss off because I’m not fixing my own problem or 2) now the boss is going to send somebody from above to help me. All I need is some idiot that’s not from my division to come help me. Keep everything internally, and I have my own financial practices, my own personnel practices, everything is down individual fiefdom of each one of the divisions.”

Allen had done a really good job helping those guys figure out that they probably needed to retire with those attitudes, and he replaced them with guys who might be able to change some of that. Most of the behaviors hadn’t changed yet, by the time I replaced Allen. He left me with a team full of folks who were reachable. We just had not yet had success reaching all of them. From 2007 to about 2009, that changed. We had many, many group discussions, and that was the emphasis—the group, the leadership team—it wasn’t me with individual division chiefs all of the time. As often as possible it was all the division chiefs and all of the eighth floor staff all together as a single unified leadership team, in that regard the same way we did business in the Control Center. Every flight controller knows what’s going on with every other system, every other flight controller. Every one of them understands what the flight director is doing at the back of the room, and how that guy or that woman is making the decision that’s either going to protect these astronauts or not. Every one of them is aware, every one of them has input. Even if it’s not on their system, if they know we’re doing the wrong thing, they can push the mike and are expected to push the button and speak up.

From 2007 through 2009, I was able to evolve the management team, the top-level leadership team at MOD, to doing business exactly that way. It wasn’t with me being their flight director, because that would not have worked. It was in leading a discussion such that we could articulate, “Here, no kidding, are our most important leadership values. These are the things that are most important to us as a directorate. In these areas, we might have offices that have been in this directorate for 40 years now.” Like we had our own IT [Information Technology] shop doing office IT stuff, because it wasn’t much before I came to NASA that there was no such thing as office IT, or it sure didn’t look like the way it looks now. But when that industry took off, MOD never got out of that business and kept hobby shopping our own solutions. We had lots of things like that that we were still doing, because we always had.

As we started pulling everybody together and getting them unified, getting all of them talking completely transparently with their peers, with the eighth floor, and having the opportunity to say, “You’re not doing it right in your division,” and it not being a food fight where their peer would say, “Well, tell me why. What is it you guys think that you have solved differently than what we were doing to solve that same problem? Because we don’t need to reinvent the wheel if you guys have already done it, or if you’ve done it better and we haven’t.” You never would have heard that before 2007, at the directorate level in MOD. At least not for more than a decade, you would not have heard that.

By 2009, again, before Constellation was canceled, the whole team listened to their peers and were willing to take criticism from their peers, because they knew it’s coming from the right place. They’re not trying to get a leg up on each other. And, every one of them was aware of any decisions I made that affected risk we’re accepting, recommendations we were making to the program. Things like the discussion to not go after little bits of work when Constellation was canceled, I didn’t do that on my own and just tell them all. We had that as a group discussion. They all got to hear my, innermost private thoughts, and why something was the right decision, and each one of them was able to vent their spleen and say, “No, you’re wrong. Here’s why.” We talked it out. We didn’t necessarily vote, and certainly everything we did wasn’t unanimous, although I can’t tell you a single, space-related, important decision we made that wasn’t fully unanimous, not one.

There were some strategic risks and some financial things that we did that were not unanimous. Some division really, really, really wanted to get an investment for some new software package, or something like that, but in the end we decided it just didn’t have enough merit, that we didn’t want to commit the resources to it. We had higher priority things we wanted to spend the money on. There were occasionally decisions we made like that, and the folks that wanted it understood the rationale and let it go. But they didn’t necessarily agree. But, even those occasions were in the minority.

I would say 95 percent of the time, even those discussions, the “losers’ in those discussions in the end did the math themselves, and after the full discussion also agreed, “Yes, this is the right priority discussion for where we are. My folks really want this, but I understand this.”

As important to me, as recently as 2014, at the end of my time as Director of MOD, there was not a single discussion we had that every single one of the division chiefs or the eighth floor staff was not fully comfortable stopping me in front of everybody and saying, “No, I think you’re wrong, and here’s why.” Never would have happened before the 2007ish timeframe. Just wouldn’t happen.

The single thing I am most proud of in my whole NASA career was that evolution and effecting that change in the MOD leadership culture. For the first time in pretty much my entire time at MOD, here is a leadership team now that has the same trust level all the way around the table, say we’re sitting in the conference room, same trust level we demand of our folks in the Control Room. Same level of forthcoming interaction. The way they would quote me if any of them was in the room today is, “All cards are on the table face up on every single discussion we have.”

I wouldn’t move money without every one of you knowing about it. I wouldn’t help somebody fix their problem without everybody else hearing it first, so that if there’s something I was not aware of, I didn’t end up helping solve the wrong problem, when there was another division with a bigger problem we should spend that silver bullet to solve.

There’s not a single discussion we had, no matter how critical, that they couldn’t stop me and tell me I was wrong on. In fact, by the end, many of them would tell me in no uncertain terms in front of witnesses, which I took as a healthy sign every time. Their job isn’t to listen to me and do everything I tell them to do. It is to engage in the discussion, give all of us the benefit of your judgment, make sure that we are doing the right thing, because our job is protect the crew, protect the spacecraft, and get the mission done. It’s not to make me look good, it’s not to recognize that I’m the boss and just do what you’re told until you get to be the boss. I say all that to say what I was looking for in leaders all that time – people that I thought had the potential for understanding and being able to do that.

When I promoted folks to Division Chief, I was promoting folks that I knew would be capable of leading their division like that, fully transparent with their peers, fully transparent with the eighth floor, and that they would be able to evolve to the ability of contradicting me in front of their peers. Taking the same risks we expect of our flight controllers in the Control Room in the management ranks, where if they make a mistake, they didn’t just kill everybody. It can’t be all that scary. Jump in. Do the right thing here.

The evolution really came about because of the people Allen and I picked to be direct reports as we had attrition. The focus wasn’t, “Who are my smartest rocket scientists, who are my scariest, tough, fire-breathing flight directors?” Instead it was, “Which of these leaders can make this evolution from strong technical leader to this type of inclusive leader?” When I say inclusive, I mean it differently than most people mean inclusive. I mean fully transparent, fully engaged, no unmentionables, nothing you can’t talk about.

Further, when I was selecting a direct report, I wouldn’t do it without pulling in all of my current direct reports after the interviews were over, and I would think out loud with them in the room and say, “Guys, here’s how it looks. I’ve done these interviews, here’s what I’ve seen in these people. For those of you who have worked with these three candidates for this Division Chief job, what is it some of you have seen that’s different than what you’ve just heard me say? Would any of you do the math differently? Or do you also see them in this pecking order? Do all of you agree that this is the guy we want to sit around this table with us?”

After doing that for a while, when it was time to pick deputies to those division chiefs, we would have a similar discussion. I would remind them that if you’re going to make somebody a deputy, that person is now in the pecking order to sit at this table with us in their next promotion. You need to be picking people that you think show the potential and the aptitude to make the same evolution that you have each made as leaders to sit around this table. When you have them as deputies, your job is to look for signs that maybe they’re falling short of that, and help figure out how do we get them around it.

In fact, I got the division chiefs to help me from time to time move them around, kind of a musical chairs thing. Actually, some of the MOD managers in the middle would complain about that. “You guys are just moving us around, it’s just musical chairs,” which is true. We did it very deliberately and very strategically. I would sit down, maybe about annually, with all of the division chiefs as a group, and we’d say, “Guys, we have these three deputy division chiefs who have never worked outside of the divisions they are now in as deputies. Don’t you think we need to move them somewhere to open their horizons or change their perspectives some?” Sort of like the experience I had going to XA to work for my friend Steve Doering. That way, we’d break them out of just being in charge of the thing they grew up being a god in. They’ve got to be a leader for these other reasons and rely on their people to be their experts. We would take two or three of them and we would rotate them, but very surgically, very intentionally. Every single division chief knew this is why we’re looking at these three, and this is why we’re moving them.

We would also have discussions about all the deputy division chiefs, who we thought were most ready to be one of us sitting around the big table. We would put them all in a pecking order, and we would talk about it as a group. Often, the division chief whose deputy was the one that we were talking about would accept the assessment. “Yes, my guy is number three on the list, my guy is not number one on the list.” If we’re going to promote some next person, the attitude became less about promoting them to run that specific Division, and more about promoting that leader to sit around this table with this leadership team. It might be in your Division. It might just as well be in a different division, we don’t know. It would depend on what the Directorate needs at the time. What’s most important is we’re promoting them to the MOD BOD, which is the MOD Board of Directors. If you report to the Director of MOD, you are now part of the Board of Directors. I don’t care what division you’re in, your primary responsibility is that. The division you’re in is just your day job, but your primary function is the MOD Board of Directors. If we promote somebody to deputy, every one of us has to think that that person has the potential for being on the Board.

Branch chiefs, what are we looking for in branch chiefs? Who are those guys that we think we can start nudging down that path that eventually this group is going to think, I’m willing to make this woman a deputy division chief, because that woman is going to be on the MOD BOD. What are we looking for in branch chiefs? Where are they falling short? Where have they not had experiences to get them ready to do that?

I started with my direct reports, eventually got those direct reports to use the same attitude with the deputies, and we would talk to them as a group. By 2014, we had the same discussions about branch chiefs. As it was time to select branch chiefs, division chiefs would all be talking amongst themselves about some branch chief need. We’d talk about the group leads (first line managers who respond to branch chiefs) and where they are in readiness. What are we seeing this person is falling short in? The things without exception that they talked about and were critical of were things that would make that person less effective on the MOD Board of Directors. As we rank-ordered them, with me typically just listening to the discussion, the things that would move a candidate up or down on the list were things that made them better or worse suited to be on the MOD Board of Directors.

Frequently, for the areas where a person is not very strong for the MOD BOD, we’d discuss what we needed to do that might fix a concern, that might sharpen them up. Or, is this a blind spot this person is never going to be able to do anything about? We had that discussion from time to time, and you would hate to be that person, because as soon as we start saying, “Yes, this person has got this weakness they’ll never be able to overcome that is a critical ability for the MOD BOD,” you likely are never going to be a deputy division chief in MOD, because of that, unless in working with your division chief and occasionally an executive coach, you can figure out your own way to get around it, kind of like my experience when I went off to MEP.

Wright: Was there a thread, a recurring weakness that you saw in people that you felt seemed to show up more? Was communication an issue, or was it more technical, or was it more financial? Was there a weakness that seemed to occur more often than others when you started to look at that as a whole?

Hill: I’ll tell you this—yes, there were. Some of it was across the board. Almost all of us came with the same challenges, same things that we needed to change, or very similar things that you needed to change. Like you have to stop being the fire-breathing technical expert in trajectory, or life support, or in my case, you can’t just be the flight director that damns the torpedoes and jams the right answer down the community’s throat, even though they’re not keeping up with you, or they’re mad about it. You can’t keep doing that, because we’ll alienate our program customers, or we’re going to alienate this other organization that we work with, or whatever that is. Things like that are pretty common.

In MOD, most of our A-Team, we would call them the A-Team at the working level because they were really good at whatever it is that they did. They were smart technical guys, or they were smart money guys, or whatever thing their job was, they were good at it, and they were thought highly of because of that. Most people’s tendency as you get promoted in organizations or in work like that into management positions, you keep managing like you’re that guy. In fact, if you’re a really smart life support guy, and you become the group lead or the branch chief over the life support people, you become the god of life support. You’re reviewing all your people’s work on life support stuff as if you’re still the life support guy, still working Mission Control. The challenge is to get people that have grown up being rewarded for being the god of whatever it is they do, to let go of that, to now be this leader that’s focusing on these other things? That’s focusing on creating other opportunities for the gods coming up behind them to also ascend to godlikeness? Now I have the risk of overusing the metaphor.

I’d say most of us—it’s probably not even an Agency thing, it’s the nature of this kind of organization—typically fail at making that evolution to deliberately grooming our people out of continuing to be the expert that they always were, and now accept these other leadership responsibilities.

Wright: Speaking of leaders, how did your management accept the fact of this ripple effect that you were beginning, and a new evolution in MOD?

Hill: Originally with some consternation, because originally I did it with a few division chiefs. It was for the same reason. I had some division chiefs that had grown up in the same division their entire lives. I felt very strongly that it limited their ability to look at some things objectively, especially things that changed some of the work they did down in their division. I was very concerned that there were certain changes we needed to make that were good things to do, whether it was technical or financial, that certain divisions would not be able to do because their leaders were protecting their guys. Actually, that’s one of those things that always worries me as a leader, when I hear somebody down and in say, “My job is to protect my guys.” Not really. Your job is protect the crew, protect the spacecraft, and get the mission done. Eventually, you get down to protecting your guys. If it’s doing the wrong thing, how is just defending your people on some issue that we screwed up a leader’s job?

MOD’s normal practice is to talk about each error. How did that happen? What was done? But sometimes, the management, tendency would be to stand between the working troop and the management, “Don’t let this stick to my guy.” The problem, then, is that at the end of the discussion, the root cause and corrective action are going to be kind of cloudy. We’re not really going to know how this was the mistake that was made, because we’re so concerned about protecting this guy. We’re not actually being clear on how we made this mistake, how we, MOD, made this mistake. For some reason, we’ve now let loyalty to our guy come first.

What is it I wanted from my guys? We want to defend our guys, but first let’s make sure we understand. If in the understanding it turns out we have an individual who made a mistake, we’re not going to go hang that guy out to dry. Let’s understand how that guy made the mistake, and see did we train him right? Was it just a goofy human error? Is that somebody we’ve seen make similar mistakes that we’re not paying attention to? Do we need to retrain that guy, or something else? Is it the unlikely case that we have somebody that we need to cut out? Which rarely, rarely happened. When I say rarely, I mean maybe once or twice in my whole time as an executive at MOD.

More times than not, we reach an understanding for how it happened. Maybe we got sloppy in some processes, or we set this guy up to make this mistake with misleading guidance or training. You can’t get to that illuminating, bottom line answer if you think that your primary job is to “protect your guys.” You can get there if you’re all about making sure we understand.

I started moving division chiefs in order to help break down that kind of discomfort. It worked. It worked way beyond my wildest expectations. This whole MOD BOD thing, and the full engagement, I knew what I wanted. I didn’t really think that I would fully achieve it, I just thought I would make it better. Allen made it better, and gave them to me. I thought my job was to keep making it better. I had no idea that before I left as Director, it was actually going to be, I think, as good as it could ever be. I just didn’t think I would get them there. Actually, I didn’t give them enough credit. It wasn’t just me getting them there. I set up the right conditions, and then they all just jumped on board, and it was astounding.

As I started moving division chiefs, some of them were really devastated. You’d have a guy that spent 25 years in a division, finally became division chief, just to have the new director come along and take them out of this division they’ve waited their entire career to be in charge of. It devastated some of them. Even the ones that were most devastated bowed their heads and did it, and became better leaders for it.

By the time we had done that a couple of times with division chiefs, the deputies started understanding why we were doing it. When we finally started then doing it to the deputies, there was a little less consternation and resistance. When we did it to the branch chiefs, it became more easy still. They all saw there was a strategy in this. When this all goes well, we are all becoming more effective and more prepared for that next promotion up. This will end up being good for them.

Wright: How did Center management take what you were doing?

Hill: I had positive feedback from Center management. I can’t say that either of the Center Directors I had when I was Director (Mike [Michael L.] Coats and Ellen Ochoa) fully understood my strategy and my goal. I talked with them about it a few times, but I would say the normal management attitude is that kind of rotation is a good idea. As managers, rarely have I heard much more discussion than, “Oh, yes, you moved some folks around, it will be good for them.”

I talked with Mike and Ellen both some about this being much more strategic than that. We were very surgical. When we moved somebody, there were specific developmental challenges that we thought that guy had that by making the move, we thought he’d get improve in. While I said some of that to Mike and Ellen, I’m not sure how much of that they really caught on to, only because most of the norm for the Center if not the Agency is to simply move folks around. As I said, MOD was much, much more deliberate than that.

The HR [Human Resources] people that supported us in MOD just raved about it. I got really strong feedback and comments back from HR management, which was good to hear. For me, the whole litmus [test] was sitting and watching the MOD BOD, and when I think back in the 2006 timeframe, 2007, even when we had changed a number of the players at the table and there were folks that were more willing and able to work together, and I think about still how reticent many of them were, even most of them, to engage in discussions that were not principally about their division. How many of them would seek me out in private to talk about some of their dirty laundry, and try to maybe see if they could get me to help them with something privately, only to have me say, “This is a good discussion, I get it. I think you’re probably right. We’re going to have the MOD BOD up here tomorrow, bring that up tomorrow when everybody else is here, and we’ll see if anybody else has another observation on this before we make a decision.”

In 2010 and faced with Constellation cancellation, when I saw how quickly everybody got on board to help us do the right thing, how quickly division chiefs are part of making right decisions that affect their own organization in a negative way, if you look at it from the old way of looking at it, and that very division chief is identifying yes, this is the right thing to do, even though this is costing us. I see this is right, here’s how I’m going to help do this right thing. That’s how I knew this was the right thing.

By the end, I said already there’s not any discussion that we talked about that my folks wouldn’t contradict me on if they felt the need. The hardest discussions we had in my last year as a director, I rarely spoke. I might come up with the topic we needed to discuss, the decision we needed to make. Every now and then I might ask a question, or put a thought on the table, but some of our most contentious discussions—when I say contentious, I mean that in a really good way, where everybody’s really engaged—this is important to us—I’m going to influence this decision. I sat at the head of the table just like this, watching and listening, usually smiling, because I could not believe how engaged each of them were. Even when the discussion was as tense as it could be, raised voices and everything, I could tell this is all still going the way it needs to go. These people really feel passionately about what they’re saying, but they’re not attacking each other, they’re not trying to one-up the other guy. They’re trying to get to the right answer.

At the end of it, I bet nine out of ten times in our most important decisions in my last year as director, one of the MOD BOD around the table summarized at the end, “Well, I think this is where this is all going,” and they would look down at the table, and my whole role in it would be to nod my head and say, “Yes, that was a good discussion guys, but you’re absolutely right, that’s what we ought to go do.” That was how I knew we did the right thing. Evolving that team to being able to perform like that—again, the single most important thing I’ve done in my career. It’s certainly the thing I will be the most proud of in my career.

Wright: You felt like it’s prepared them for whatever future would come their way?

Hill: I did. I’ll tell you, in spring of last year, probably March of 2014, I gathered the MOD BOD in the conference room on the eighth floor and I apologized to all of them. What I said to them is “I do not have a very good rapport with the Center Director. I have tried. I cannot figure out how to change it. There is definitely a communication problem between me and Ellen. Where I feel bad, and what I apologize to you folks about is, it’s my job to manage that relationship. I cannot for the life of me figure out how to change this. You guys can’t do it for me. You guys need me to do it. I have other important strategic relationships I have to manage, and I think we are doing well in them. This one, I cannot. I am really afraid that because of it, I’m going to end up drawing bad, uninformed decisions, well-intended or not, from the Center level that will affect us and affect our ability to do what we are responsible for doing. Decisions that may, in fact, affect the MOD BOD and this discussion, this leadership culture. That worries me. If that happens, I feel chiefly responsible, because I could not figure out how to manage the relationship.” They all sat in the room like deer in the headlights.

Sure enough, in June, just three months later, I got called up to the ninth floor and informed that the Center had decided they were going to do some reorging that involved combining MOD and FCOD [Flight Control Operations Division]. If I was interested in the leadership position, I could certainly apply. I pretty much knew then how that was going to end, and in August I was no longer on senior staff in any leadership role at all. In answer to your question, my concern from the get-go, when Allen was still in MOD, everything we talked about doing, whether it was changing those people individually or changing the discussion, bringing the right values into the leadership team and then pushing those down, all of that we did very intentionally in order to make it a lasting cultural change, not just, “Well, this boss wants us to do this, so we’re going to do it for this boss. When it’s not this boss anymore, we’ll do it the way the next boss wants.”

The intent was to make it just like our experience in Mission Control, where generation after generation reinforces all of the best parts of this. Each next generation figures out which other things did we not evolve that we could have and should have, and they will. It will even be better. The guy that replaces me is going to start with where I brought them, he’s going to take them even further. The guy that replaces that guy, his job is to do the same thing. Everything that we were doing, all the discussions we were having, were very deliberate to make that the way it would go, that it was a cultural change.

We didn’t just make that up as we went along. We had formal exercises we went through with very specific books on leading change and how to make them permanent cultural things for all of the right reasons, rather than just because I’m the new boss, and this is what I’m going to do. The thing I told my guys is the one thing that undoes all of that is the person at the head of the table has to believe in all this. I can’t make them be the MOD BOD, but I can keep them from being the MOD BOD by my engagement with them. I said, “I am very worried that when I am replaced I will not be replaced with somebody that buys into this same thing, and that we will lose what took us so many years to pull together to make happen, that we were then able to deliver shocking financial performance in addition to perfect technical performance. I am concerned we will lose that, and that as we lose it, it’ll set us back.”

We had made so much progress as a leadership team to exercise the same values that we demand in Mission Control. My greatest underlying fear, always, would be if you don’t manage that way, eventually you’re going to erode the ability to do business that way in the Control Room, in real time. That was always my fear, including in judging my own leadership of the MOD BOD.

Again, I’m a one-trick pony. What am I principally motivated by? Protect the astronauts, protect the spacecraft, get the mission done. If we can’t manage that way, if we don’t have those right values, eventually it comes back and erodes those things. We’ll see.

I definitely was replaced, and BK and I have been friends for a long time (Brian [K.] Kelly, the Director of FOD [Flight Operations Directorate] now). Much of the day-to-day and week-to-week management interaction, things that were specifically intended to preserve and further this trust and transparency around the MOD BOD, to keep pushing us down that road and making us a more effective leadership team, and to keep pushing those practices deeper and deeper into the divisions… many of those things, BK canceled almost immediately. We’ll see. Time will tell.

Wright: Yes. Time will tell. Speaking of time, I know I have pushed the limit a little bit, and you have a noon appointment. How about if we leave it as such, that we stop for today?

Hill: Sure.

Wright: All right. Thanks.

[End of interview]