NASA STS Recordation Oral History Project
Edited Oral History Transcript

William J. Roberts
Interviewed by Jennifer Ross-Nazzal
Downey, California – 25 August 2010

Ross-Nazzal: Today is August 25th, 2010. This interview is being conducted with Bill Roberts in Downey, California as part of the NASA STS Recordation Oral History Project. The interviewer is Jennifer Ross-Nazzal. We are joined with Bob [Robert] Sechrist who is videotaping this morning. Thanks again for taking time out of your day, I certainly appreciate it. I thought I’d begin by asking you to give us a brief overview of your career with [North American] Rockwell [Corporation] and [The] Boeing [Company].

Roberts: I went to San Jose State [University, San Jose, California] and graduated with an aeronautical engineering degree. I came out of college and got a job in July of 1982 here at Downey as a system engineer, working in the department that had requirements and maintenance, and also mission support operations. My director at that time was a man named Bob [Robert] Weaver. Some of my tasks were to track engineering changes required on the orbiter vehicle, based on small improvements as we were coming through the manufacturing process. At that time [Space Shuttle] Columbia [OV-102] was already delivered to [NASA] Kennedy Space Center [Florida (KSC)] and we were working on coordinating engineering changes with Kennedy Space Center.

Another part of my job was mission support activities. At the Downey facility here we had a Mission Support Room, which was an engineering support room that had live engineering data feeds on our consoles. We were assembling orbiters up at Palmdale [Rockwell location in California], and Columbia had already flown two missions and we were getting ready to fly it again. So we were working turnaround issues, live test feed from Kennedy Space Center and the integrated tests up there at Palmdale.

I worked out of that department for five years, and an opportunity was offered to me to become a vehicle project engineer on the Space Shuttle Discovery [OV-103]. I understood what the position was because in those prior five years I was working directly with the other vehicle project engineers in the project office. So I moved from my original department into the project office and became the OV-103 Discovery vehicle project engineer. I was reporting directly to the vehicle project manager, Will Leonard. The two of us were working as a team monitoring and directing and ensuring that all the engineering deliverables for Discovery—mod [modification] kits that were being built here at Downey—were built to support the schedules needed at the site, either at Palmdale or at Kennedy Space Center.

That was 1986, right after the [Space Shuttle] Challenger [STS 51-L] accident. We were in the middle of the return to flight turnaround flow. There was a significant amount of upgrades and modifications done to the vehicle, and Discovery was the return to flight vehicle after the Challenger. We had a lot of work to do during that time period. I believe it was about a year and a half down between flights, so we had to meet schedule. We had a lot of mod kits being produced here and engineering changes, processing changes. It was our job—Will Leonard and myself—to coordinate and make sure that all that integration was happening and we were supporting the needs of Kennedy Space Center and delivering our engineering deliverables and mod kits.

I worked with Will on OV-103 for about two and a half years. The replacement orbiter to Challenger [OV-099] was named [Space Shuttle] Endeavour [OV-105], and the forward and aft fuselage was being assembled here and getting ready to be integrated and built up at Palmdale. Will Leonard was offered the job as vehicle manager down at Kennedy Space Center. He took that, and then they offered me the vehicle manager job of Discovery here at Downey. In 1988 I became the vehicle manager, and worked as the vehicle manager until 2002.

After Boeing took over Rockwell and McDonnell Douglas [Corporation], we relocated all of our engineering support workforce from Downey to the former McDonnell Douglas facility in Huntington Beach [California]. While we were at Huntington Beach the first couple years in 2001 and 2002, there was a strong movement to move all of engineering support on the Space Shuttle program down to Houston [Texas, Johnson Space Center (JSC)]. I don’t know how I really want to word this, but I’d visited Houston enough to know that I really didn’t want to relocate my family permanently down there.

So I pursued other work within the company here in southern California, and I got a position over on the ground-based missile interceptor program over in the Anaheim [California] facility as a program manager. I spent about four years over there where we designed and deployed and tested a prototype ground-based missile interceptor. We had several launches out of Vandenberg [Air Force Base, California] and also out of Kwajalein [Atoll, Hawaii] where I had to manage those launches.

During that time period the Columbia [STS-107] accident occurred [February 1, 2003]. I was returning from Kwajalein and I was in Honolulu [Hawaii], and I got a call from John [P.] Mulholland, who was our program manager for Boeing down in Houston. He asked if I would like to come back to the program and to head up the return to flight design certification review process. I said, “Sure, as long as it doesn’t mean I have to move to Houston.”

He said, “No, you can stay in Huntington Beach and set up camp there. It might be a little bit of traveling back and forth.”

I said, “Fine,” so he allowed me to pick my team. That team was a combination of engineers from Huntington Beach, Houston, and Florida, from all three Boeing sites. That task was basically to understand the accident of Columbia, and take the recommendations out of the Columbia Accident Investigation Board and try to implement them into a new design certification process for return to flight of all the remaining orbiters. We integrated with the USA, United Space Alliance, engineering corps and the NASA Shuttle engineering corps. We came up with our recertification of the orbiter vehicle, not only from a hardware standpoint but also from a ground processing and in-flight inspection and repair processes.

That job lasted about two years. We briefed our design certification material to Level I NASA about a month before the flight readiness review for STS-114. The approval of the DCR [design certification review] by Level I and all other divisions within the agency allowed the go-ahead to go to the flight readiness review for STS-114, which was another return to flight for Discovery. They went ahead and that all worked out, STS-114 was successful in 2005. Our processes were proven.

About a month after that flight I got another call from my program manager, John Mulholland. He says, “Now that we’ve got the fleet flying again, I’d like you to head up the project of retiring the fleet, which is scheduled five years from now.” That is called the transition and retirement project, which I accepted and have been doing since then.

It was a difficult task to define because we needed to understand what the requirements were to retire the vehicle. As we all know now, they’re going to be put into three museums around the country. The vehicle is designed with certain propellants and other items that are determined to be hazardous. You couldn’t just park a vehicle the way it is right now in the Smithsonian [Institution National Air and Space Museum, Washington, DC], because there’s a lot of propellants and other things that are hazardous that may cause issues with visitors.

I was allowed to build a team again. We were told that the group of engineers that you have within that team, you cannot disturb flight operations. So the engineering group I had working on this project with me was basically all the former subsystem managers that were still at Huntington Beach that were not subsystem managers now, because most of your subsystem managers are at Houston or at Kennedy Space Center Boeing facilities. I had a great team there because the folks there at Huntington Beach that were prior subsystem managers probably had more experience than the current subsystem managers.

The first task at hand was to write a document which detailed all the hazards in the vehicle. That document is called the orbiter fleet safing document. It details the hazardous commodities internal to the orbiter vehicle, and also what the recommended mitigation would be to safe that hazardous commodity for long term display. It maps out where these hazardous commodities are; we have graphics and pictorial. There’s certain things on that vehicle that you cannot remove that will still be hazardous once you display it. We’ve laid out what a future display site should do to mitigate those hazardous areas of the vehicle for long term public display.

Once we established what the fleet safing criteria would be, we had to write the requirements for Kennedy Space Center to safe the vehicle. When we’re maintaining a vehicle and getting it ready to turn around in a flight environment we work towards the requirements of the orbiter maintenance and requirements specification document, the OMRSD. The intent was to write a new document similar to the OMRSD that laid out what the safing requirements would be for the USA ground ops [operations] guys so that at a future date we could put it on the SCA, Shuttle Carrier Aircraft, and ferry it off to a display site and be safe for long term display.

Utilizing the criteria established in the fleet safing document we wrote the end state subsystem safing requirements document, ESSRD. That ESSRD is what we would compare the OMRSD to, but that ESSRD is only effective during the time period of post-program. When we’re definitely not going to fly these vehicles again, we will invoke those requirements. The reason being a lot of those requirements cut hardware out of the vehicle, so you don’t want to do that if you’re going to go refly the vehicle.

The two documents, the fleet safing document and the ESSRD, three weeks ago were baselined at Level II at a PRCB [program requirements change board] as an NSTS [National Space Transportation System] document. They have their own document numbers; they’re NASA documents. So we are official, and United Space Alliance ground ops folks are writing their work authorization papers now to go off and safe these vehicles. The WADs [work authorization documents] are being written as we speak. They’re being delivered to our Boeing subsystem responsible engineers so that they can make sure that those WADs satisfy those ESSRD requirements. It’s similar to the process that we have during a turnaround flow right now.

That’s what I’ve been doing over the last five years, and we’re coming to the end of the program now. Obviously the development of these documents were the main thrust of what my team was doing early on, and we still are getting some inputs and changes to these requirements documents—as we have in the OMRSD for the life of the program. Those are called RCNs [requirements change notices] that formally change the OMRSD. We now have established a process similar to that for changes to the ESSRD. We have weekly meetings, go over these proposed changes to the ESSRD, and once a month we’ll have a board meeting and change the document formally.

That will continue on through the end of the program when we get into the implementation phase, where we actually start cutting the vehicle and safing the vehicle. Right now the schedule of that is a little bit fluid. It used to be STS-135 was the launch on need flight for the scheduled last flight, STS-134, which is February next year [2011]. But it looks like they might actually get the STS-135 formally approved, and that’s scheduled for June of next year. That would be the actual last flight of the Space Shuttle fleet. At that point in time we would start our implementation of our ESSRD, our safing requirements.

In parallel to all this we also have a lot of requests for artifacts coming off the vehicles. We do have guidelines that unless it is formally vetted through the process and unless it’s a safety issue, no hardware will be removed off the vehicle. Our goal is to deliver these vehicles as close to flight-like configuration as we can. Obviously you got to remove certain components due to safing. You’re going to get formal requests from colleges or other NASA divisions wanting OMS [orbital maneuvering system] engines or RCS [reaction control system] thrusters. They’re out there and they’re coming in.

Those will become new hard requirements to processing these vehicles when we get to the implementation phase. All of those new requirements—not only our safing requirements that we’ve already written, but these artifact or new program requirements for removals of components off the vehicle—all get integrated into our workflow during the implementation phase, and they become flow requirements. Those are all being worked by our team along with USA ground ops and also obviously the NASA folks. We’re discussing those as we progress down the road to the end of the program.

Ross-Nazzal: I thought today we would talk about Discovery, and tomorrow we could talk about return to flight and transition. Would you give us a brief overview of the history of Discovery OV-103?

Roberts: It first flew in 198[4], STS [41-D]. Right now it has 38 flights on it, more than any other vehicle. When I was managing Discovery it had the fewest in-flight anomalies. Because 103 was the return to flight vehicle after the Challenger accident, all the best resources were put into that vehicle during that turnaround. So when that vehicle flew STS-26, that return to flight mission, it was prepared as good as any orbiter vehicle could possibly be. Obviously the other vehicles following behind it went through those same upgrades and modifications, but being the first, I think the folks in the program were definitely focused on that one vehicle during that timeframe.

For years, from STS-26 until I left being the vehicle manager, when it flew it had very few problems. After STS-26 it went through its first OMDP, orbiter maintenance down period, at Kennedy Space Center. That’s when they put in the drag [para]chute system and carbon brakes. There were quite a few upgrades done, but not to the amount that were done on 103 and other vehicles when we started doing OMDPs up at Palmdale.

The next OMDP was done up at Palmdale, and that was a unique one for 103 and the program because it was also the first time since the vehicle was built that we did all these weight savings mods. We removed the payload bay doors. We pulled the internal airlock, put an external airlock in it. Basically it was the first time an orbiter was torn apart to the level it was since it was built.

That was in ’94. It was pretty significant. I’ll be honest, I was concerned. I looked at that vehicle and it was all torn apart up at Palmdale. I was not part of the program when we were building these orbiters. There were already three orbiters built by the time I joined, so I really wasn’t part of that whole manufacturing processing phase. When I saw [OV-] 103 torn apart the way it was I’m thinking, “Holy cow, how are we going to put this thing back together?” Obviously the folks that took it apart were also the folks that put it together the first time. I’m not a manufacturing guy; I’m an engineer, I look at drawings. I’m not a nuts and bolts guy out there in the bay, but I got to know them real well.

[OV-] 103 has flown the most, it’s had all the upgrades. It was also the first vehicle that came out of the initial upgrades. [OV-] 101 Enterprise and [OV-] 102 [Columbia] were built with a lot more conservatism in their design than OV-099 [Challenger] and 103 [Discovery],

[OV-]1 04 [Atlantis], [OV-] 105 [Endeavour]. When they were designing those first vehicles they designed them with unknowns, meaning they didn’t know the actual loads that the vehicle was going to see on ascent. They didn’t know the descent loads, they didn’t know the thermal loads.

As we start flying these vehicles and the vehicles were instrumented, we started getting real-time data and they realized that we overdesigned some of these things. So in the follow-on vehicles we designed them so that they would be much lighter. When you have a lighter vehicle you have more performance, when you have more performance you can bring more payload. [OV-] 103 was the first vehicle to become one of the “high performance” vehicles compared to the prior ones.

Ross-Nazzal: It was also supposed to be used by DoD [Department of Defense]?

Roberts: Yes, 103 was also the dedicated vehicle for Air Force, and it has a different TPS [thermal protection system] design because of that. It was designed to fly out of the Western Test Range out of Vandenberg Air Force Base. Reentries coming into Vandenberg Air Force Base had a higher cross range requirement, meaning as you’re descending you had to come off of your normal inclination and turn into Vandenberg at a much farther distance from your normal trajectory, which means you had to put it down steeper and you’re getting higher heat loads. So it had a different TPS design on the underbelly of the vehicle.

Ross-Nazzal: Different HRSI [High-Temperature Reusable Surface Insulation] tiles?

Roberts: Yes, certain areas had higher heat load tiles because they knew that when you have a higher cross range entry certain areas of the wings and elevons get hotter. [OV-] 103 never made it to Vandenberg—actually it did, but it was on the back of the SCA. It just stopped by there, and then went on.

Before the Challenger accident I spent a year and a half up at Vandenberg when we were preparing that launch site, and one of my jobs was to make sure the orbiter requirements were being implemented. I was going back and forth with Gabe Shopei and Jim Cox and those guys monitoring the progress on SLC6, which is the Space Launch Complex 6, where they were going to launch the Shuttles out of. OV-101 Enterprise was our FVV, facility verification vehicle. We actually brought it from the OMCF, which is their version of an OPF, an Orbiter Maintenance and Checkout Facility.

We used for the first time the orbiter transporter, which is this Italian device that they use now down at Kennedy Space Center that has 72 wheels on it, and they mate the orbiter horizontally to it. It was designed specifically for Vandenberg use, because from the OMCF to SLC6 was a 12-mile drive. Rather than towing it on its own landing gear, they decided to button it up and get it ready for vertical mate and put it on this orbiter transporter and go down the road to SLC6.

We brought [OV-] 101 out there and we cycled power, made sure all the plumbing fit where it was supposed to fit and all the electrical connectors, and it went well. The week we did the FVV tests, I was the only Rockwell employee that had pad access, and that week they had what they called media day. They allowed all the employees to take pictures out at the pad, which was a no-no at that time prior to that. I took three rolls of 36 [millimeter] film. I brought it back over here to our Mission Support Room.

Bob [Robert M.] Glaysher was our vice president of engineering then. Every morning we had the KSC time calls at 7:45. Bob would sit in those meetings and we’re looking at my pictures. Bob Glaysher says, “Can I borrow these?” and he had official Rockwell pictures made from my photos. At my house I’ve got all those pictures he had converted into 8 ½ x 11 [inch] enlargements. Those were the only pictures that Rockwell ever had in their possession of [OV-] 101 out at the launch pad. NASA was taking pictures, and the Air Force obviously. But those were the Rockwell pictures at that time, and still are.

Ross-Nazzal: Any other differences between the rest of the fleet for Discovery, other than the TPS and the weight?

Roberts: Right now they’re all configured similarly. I think out of the three remaining vehicles OV-104 is the only vehicle that does not have the [Station to Shuttle] Power Transfer System, where you transfer power from [International] Space Station over to the orbiter for long stays on Station missions. Prior to that, for the orbiter to stay up and running for two weeks, you’d have to use the cryos [cryogenic fuel] for the fuel cells, which minimized your time up there. Now that you’re taking solar energy power from the Station and plugging it into the orbiter, the orbiter can stay up there a little bit longer. I think [OV-] 104 was slated to have the mod, but after the Columbia accident they said only two vehicles really need to have that.

Ross-Nazzal: You mentioned that you became vehicle engineer for Discovery after the Challenger accident. What were things like around here at that point?

Roberts: I was working in that Mission Support Room the day of the Challenger accident. In fact because I was one of the young guys then, they put me on third [overnight] shift. That night during the countdown we all saw the ice on the launch pad, but the accident was a complete surprise to all of us. Officially I think Rockwell was a no-go during the initial poll, but it was for the ice hanging on the pad, not for any other issues.

Right after that we were all in shock, we were in shock. Especially myself being new to the business, I was just waiting for guidance by our management. And guidance came quickly. One of the requests by the customer, NASA, was to identify all the Crit 1 (Criticality 1) hardware. The FMEA [Failure Mode and Effects Analysis]/CIL [Critical Items List] was rewritten. It’s a document that identifies all the components and how they fail and what the effect would be on the whole orbiter vehicle.

At that time Bill Fleming was in charge of rewriting all that documentation. We identified all the Criticality 1 hardware that could be either modified or eliminated from the orbiter vehicle. Coming out of that whole task we proposed modifications to NASA, and those all got folded into that return to flight flow. I believe we reduced the Crit 1 hardware by 40 or 50 percent. Couldn’t eliminate it all, but we definitely reduced a lot of it.

Ross-Nazzal: Can you give a few examples of some of those hardware changes?

Roberts: There were check valves and stuff like that. A lot of plumbing issues, especially in the OMS/RCS area.

Ross-Nazzal: Nothing major like a major system had to be completely revamped?

Roberts: No, it was at a component level where we actually had design improvements that improved the system, which either eliminated the Criticality 1 for that system or improved it such that it’s much more reliable. Those vehicles coming out of the first return to flight flow came out much safer, much safer. Not absolute safe, but much safer. You can never eliminate all the risk.

The attitude during that time—yes we were all down, but once we got our new direction and started working towards a goal this place was really busy. We hired a bunch more people. The workforce went up probably by 25 percent because we needed more engineers, more manufacturing, more quality—we needed a whole workforce. It was really busy.

Then leading up to STS-26, I was the vehicle project engineer, and every Tuesday afternoon we would have a status on all the mod kits, the vehicle down at Kennedy, the design issues here, and we had to brief the president of the division. At that time it was either Rocco [A.] Petrone or Sy [Seymour] Rubenstein—standup briefing every Tuesday afternoon at 1:30. That was a real interesting time. Sometimes we’d run behind and we’d have to catch up. There was a lot of pressure, but it was exciting.

Luckily at that time in my life I was not married and I could put in 12, 15 hours a day. It’d have been a little different if it was this time in my life when I have three kids and sports and ballet. It was exciting. There was definitely a dip and then it became real exciting when we got close to STS-26. Then obviously with the successful launch of STS-26 it was real good.

Ross-Nazzal: When you became vehicle engineer did you know at that point that Discovery was going to be the return to flight vehicle?

Roberts: It was about a year and a half after the Challenger accident that STS-26 flew. I became the vehicle PE [project engineer] for [OV-] 103 just about a month and a half after that accident, so I don’t think that Discovery was named the return to flight vehicle at that time.

Ross-Nazzal: You’ve mentioned the mod kits several times. Talk about the changes that were made to the vehicle as a result of the accident and the [William P.] Rogers Commission [Report on the Challenger disaster] and how all of that played out in the different changes that were made.

Roberts: The great majority of the mods and upgrades were eliminating those Crit 1 systems, and where we couldn’t eliminate improving the reliability. The Challenger accident was not a result of the orbiter design, so the downtime that we had in the orbiter project was an opportunity to make improvements that we wouldn’t have had during normal turnaround flows. Number one goal was to make those improvements to eliminate as much risk as we could in the operating systems as we knew them at that time.

There were 226 modifications done on the vehicle, of which each mod—we call them master change records, MCRs—could have one mod kit or they could have 15 mod kits associated. A mod kit would be just a little bit of redesigned TCS [thermal control system] blanket or it could be a whole new pump. We did some hydraulics, we improved our avionics. There’s quite a few. The main thrust of it was that we definitely reduced risk by that initial analysis on our FMEA/CIL [Failure Modes and Effects Analysis, Critical Items List] database and then identified the hardware that we could either redesign and eliminate that Criticality 1 category off that component or reduce the risk substantially.

Ross-Nazzal: All the mods were done at KSC, not here in California?

Roberts: Yes, all the return to flight work was done at KSC.

Ross-Nazzal: Did you spend a lot of time down at the Cape [Canaveral, Florida]?

Roberts: I’d go down there about once a month for about a week. A lot of our manufacturing folks from Palmdale were sent down there to help out the ground ops folks. There was a lot of running around the country at that time. Some things worked out perfectly. Some things worked out challengingly, meaning once we got into putting in our kits sometimes it wasn’t exactly the way it was supposed to go so we had to do redesign.

One of the more challenging events during that time period was when we got done with our return to flight flow in the OPF [orbiter processing facility], the vehicle was configured for flight and rolled out. We went out to the pad two weeks before the scheduled launch of STS-26, and we started filling the OMS/RCS system with the hypergolics [propellants]. There was a leak in the left-hand OMS pod. It was MMH, monomethylhydrazine.

It was obvious it was leaking. It was a bad Dynatube fitting. It was not disturbed at all during the turnaround flow, but it had some seals in there that had shrunk up and become brittle during that year and a half downtime. When they started flowing the hypergolics in the prelaunch loading, there were leak detectors and they isolated the leak to this one Dynatube fitting.

After all the work to get that vehicle ready to go and we’re two weeks away, what would have had to happen is the vehicle rolled back to the VAB [Vehicle Assembly Building], unstacked, go back to the OPF. Then remove the OMS pod from the aft fuselage, take it over to the HMF [horizontal mating facility] where the bottom of the OMS pod is exposed and you can gain access to that one Dynatube fitting, and change it out.

That whole process would have taken about a month and a half to two months. Well, our guys here at Downey came up with a way to get at that Dynatube fitting while out at the pad. Basically when the vehicle is vertical and the payload bay doors are open on the pad, your 1307 bulkhead, which is the back side of the payload bay, is a floor.

You had two bulkheads, the 1307 bulkhead and also the forward bulkhead of the OMS pod. You had to cut square holes adjacent to each other, and that leaking Dynatube fitting was right there in front of them. We gained access by cutting those four holes, two in 1307 and two in the OMS pod bulkhead. George Gallagher, our manufacturing guy that put in the 17-inch disconnects in all the orbiters, lay on his stomach and went in there and bolted on a clamshell on the Dynatube fitting.

We designed and built this clamshell. It’s basically two halves and it had a zerk fitting coming out, like the way you grease your car. We put a solution called Furmanite in it, a foam silicone type material that is denser than the hypergolic gases coming in. It contained that leak within the clamshell. They did a pressure check. They refueled the system and it was good. Then buttoned it up and flew a week later. [OV-] 103 flew with that clamshell installed on that OMS pod for four or five missions. Then later they went in and cut [the fitting] out.

That whole line assembly, with a Dynatube fitting in the middle with a clamshell over the top of it, was donated to the Furmanite [Corporation] in South Carolina, the president of the company. It’s in a Plexiglas [transparent thermoplastic] display. An operation like that today would never be done. They would roll back. There was a lot of good help then and a lot of can-do attitudes, and we made it happen.

In our wood shop at Downey we actually had to build a wood mockup showing the two bulkheads with the Dynatube plumbing underneath the OMS pod bulkhead. We did all this on our engineering drawings, made everything per scale. Cut the holes and demonstrated we could put the clamshell on it with Allen wrenches and zerk fitting. We did that here at Downey, and it all worked.

Sy Rubenstein, our president of our division said, “Okay, everybody go home, pack a bag, we’re hopping in the Sabreliner [T-39 aircraft] and we’re flying to Kennedy.” We demonstrated it down there, and that’s how we sold the program to do this. From the time we found the leak until we flew with that wood mockup, it was four days. That’s how quick that thing turned around.

Ross-Nazzal: Amazing.

Roberts: Yes. That was before any of these CAD [computer-aided design] programs. We had to build wood mockups out in our wood shop.

Ross-Nazzal: It’s a great history of the return to flight. How closely was the crew following the orbiter modifications?

Roberts: Out here we really didn’t come in contact with them too much. They were off training. At the time when Challenger occurred they were not in pressure suits, they were just wearing their helmets and blue flight suits so I’m sure they had a lot of flight operation process changes. They were definitely involved when we went down to Florida and briefed the flight readiness of the vehicle. If they’re not all there they might have one or two representatives of that flight crew sitting in the mission briefing room.

Ross-Nazzal: So as flight engineer, you said you’re not manufacturing or production. What was your basic overall duty then for the return to flight effort?

Roberts: As the vehicle project engineer I oversee all of engineering and manufacturing and materiel and schedules supporting that vehicle. Like I mentioned before, the vehicle project manager and the vehicle project engineer work as a team. The manager has certainly more responsibilities than the project engineer, but the project engineer has to be as qualified to manage the vehicle as the manager is. If the manager is gone on vacation or something like that, he has to be acting when required.

The primary responsibility as the vehicle project engineer and vehicle manager is to make sure that all the requirements specific to that vehicle coming out of Kennedy Space Center are collected and funneled to one central point, the vehicle project office. Then they’re fanned out to the design community, manufacturing, materiel and logistics. We have to work with logistics getting things shipped.

There were times when we were running late on popping out a mod kit, and we’d call up the guys down in Houston out of flight operations and they’d send a T-38 [aircraft] out here to Los Alamitos [Army Airfield, California]. If it was small enough we’d put it in their wing box and they’d fly it down to Florida. Those weren’t often, but there were several times we did that. Typically we just did our overnight shipments.

The primary responsibility is to receive all the requirements coming in from Kennedy Space Center as it relates to that vehicle, either engineering or hardware or logistical support. Any and all requirements funnel through one place here at Downey, and then go make sure that those requirements are being taken care of, and track them, and complete those requirements.

Ross-Nazzal: Did you take part in the flight readiness reviews?

Roberts: Yes. Once the vehicle was ready to fly, it was our responsibility to prepare all of the formal reviews—the rollout review, the flight readiness review and the L-minus two [launch minus two days] review. Before you even get into a given flow, it was also our responsibility to prepare the launch site requirements review and then a month later the launch site flow review. Those two reviews are prior to the launch of the previous flight. You establish what the modifications are going to be done on the follow-up flow. So while we’re in the middle of doing rollout reviews and flight readiness reviews, we’re also doing the launch site flow reviews for the follow-on. It got pretty busy at that time.

In the early years we didn’t have computers. It was all slow processes. I’d be here till 2:00, 3:00 in the morning sometimes if the review was the next morning and I had to get the paper and the briefings out. We didn’t have email in those early years. We had to fax everything out. That was a neat machine, it was about the size of a table.

Then when the vehicle gets ready to launch, I’d fly down to Kennedy Space Center and brief the L-minus two, then I’d work the launch countdown. Once we’d get off the pad I would take the next flight to Houston and go work the flight out of the Mission Evaluation Room [MER], next to the Mission Control [Center], for engineering support. As the vehicle project engineer or manager in the MER, I had a headset that would be listening to all the console discussion in Mission Control and the MER, and it had a direct line to our Mission Support Room.

So any and all mission requirements that would evolve as a result of problems on orbit, similar to the turnaround flow, would funnel through one point, and then I would distribute to the Mission Support Room. We did have a NASA person that took it upon himself to work directly with the Downey folks, and his name was Joe Michelet. He would pick up the phone and call directly, and work outside the process. Joe and I are good friends.

Ross-Nazzal: Were you at the Cape for the launch of STS-26?

Roberts: No, I was not. I was actually here. The vehicle manager Will Leonard was in the MER, Mission Evaluation Room. Our program manager, Dick [Richard E.] Thomas, was down at Kennedy Space Center, but I was here in the Mission Support Room.

Ross-Nazzal: What was the first mission you were [vehicle] manager for Discovery?

Roberts: STS-29. That was three flights after the return to flight. That was the DoD payload. It was a pretty exciting time. I was obviously very—I don’t know if nervous is the right word—but apprehensive. At that time I was young for that role. In fact there were a lot of folks here that were upset that I got that job. I was only 31 years old and most of the vehicle managers were in their mid-50s.

Ross-Nazzal: Were there lessons learned that people passed along to you when you became vehicle manager?

Roberts: Oh yes. Bill Fleming was the chief project engineer for orbiter operations. Orbiter operations here was the vehicle project office. For many years he was the vehicle project manager for OV-104 Atlantis. When he got promoted, that vacated a spot for another vehicle manager, and Bud Williams got that job. Bud Williams had been the subsystem manager for fuel cells.

Bill Fleming was my mentor at that time, and he made sure I succeeded. If there were issues that I had problems with, either I would go to him or he would come to me and give me some advice, along with our program manager Dick Thomas. They definitely took a risk in putting me in that position at that time in my career, and they didn’t want me to fail. So I had a lot of help at that time, which was good help, because it made me better as I progressed down the road.

I did it for 15 years. It was a good job. That vehicle manager job is probably the best job I’ve ever had, and probably the most diversified job here at this facility or within the company, because there is no limitation as to what you’re doing. Every day is a different day, because every day brings new problems. If you like working problems and you like talking to people, that’s a good job because that’s all you’re doing every day.

Ross-Nazzal: As technical issues would pop up with some of the other orbiters, were you also looking at those issues on Discovery?

Roberts: All the orbiters are designed the same, so if there’s a problem on one orbiter, you have to look at it. Any time you have an in-flight anomaly on any vehicle, the ripple effect goes throughout the fleet. Sometimes they may have a unique design that might not be on your vehicle, a mission kit or something like that.

That was part of the scramble. When we were preparing for rollout, typically there’s already another vehicle either just launched or just flew that had its own in-flight anomalies. We would have to prepare special topic presentations that addressed all those. Not just flight problems, turnaround problems. It might be a leak in part of the hydraulics or something like during the turnaround process in [OV-] 104. If somebody bumped into a pipe and broke it that’s obvious, but if it’s a matter of applying pressure to a certain system and all of a sudden you got a leak, then you might have a design issue. So definitely be in a reactionary mode on all problems on all vehicles.

Ross-Nazzal: You mentioned that there was a vehicle project manager also at the Cape?

Roberts: We had a counterpart, a NASA vehicle manager at JSC. We, Rockwell, had our vehicle project manager here, and we also had one down at Kennedy Space Center. The way the system worked, all problems and issues associated with that vehicle would funnel through that Kennedy Space Center Rockwell vehicle project manager. He would convey that information to me if it was a design center issue, and it was my responsibility to react to all those problems. That was the mode of information transfer at that time. Before email and before computers it was either fax or phone. Typically phone was your primary mode of information transfer. If there were sketches and stuff like that, we’d use the fax.

Because we were three hours behind here, in the ’80s and ’90s I would have to get to work at 4:30 in the morning, because it’s 7:30 there [in Florida], which is okay, because living in Los Angeles [California], the earlier you get on the freeways the better. I had a pretty demanding counterpart down there at Kennedy Space Center. His name was Joe Tague. He was the vehicle manager for Columbia on STS-1, 2, 3. He was also the flow manager in charge when Columbia was received down at Kennedy Space Center. He was also the vehicle manager at Kennedy Space Center for Challenger. He took that pretty hard. In fact I think he took six weeks off before he came back to work. When [OV-] 103 was named the return to flight vehicle, they put him in that position.

Joe Tague was a very demanding, very thorough engineer manager. If I showed up to work five minutes late, he’d be on my butt. I made sure I got to work when he wanted me there because he needed to get that information to me, and I had to take action items, and we had response time. We had it all written down on paper. Whatever he had written on his action item log, I had to make sure my log was identical to his. He knew that I knew exactly what the response time should be. It was all done longhand, those were long days.

I was a single guy then and I was just gobbling it up. It didn’t matter how long I worked. Those were the days when we wore suits and ties to work, too. In fact I’ll tell you one story. It was a [OV-] 103 mission. [OV-] 103 got out to the pad, and they do this inspection on the vehicle when it first arrives. When they’re out at the pad and mated to the external tank [ET], the ET doors are open. After you launch and you have ET separation, these doors, which on the back side are all tiled, come closed. There’s an elaborate hinge mechanism system on those ET doors, and one of the lugs on that mechanism had a crack in it.

Our smart guys here came up with a way that we would demonstrate that even with that lug being cracked the door would function properly. They built a simulated lug which would simulate any kind of adverse movement during the swing of the doors. We wanted to put this simulated lug on OV-102 Columbia, which was in the OPF. Even if this crack would grow to the point where you’d actually get full displacement and the material would be lost, it would be simulated by this lug with a displacement, a cutout, from it.

We had the manufacturing guys out in the machine shop making it during the night. I was supposed to fly down and hand-carry this lug down to Florida. I was supposed to fly on the red eye [overnight flight] that night. I couldn’t get the nonstop flight because they had problems with Delta’s [Delta Air Lines, Inc.] flight out of LAX [Los Angeles International Airport]. I flew from LAX to Salt Lake City, then we ran into a snowstorm there and I got delayed in Salt Lake City. I get to Orlando at 8:30 at night or something like that, and everybody’s waiting for Roberts with this test piece.

I was in the same clothes. I went to work on a Thursday morning and I ended up in Florida on Friday afternoon in the same clothes. I get out to Kennedy Space Center. I didn’t have any badging or anything, I had to go through the little one-man gate out there. I said, “Hi, I’m Bill Roberts, I’m here to deliver this.”

“You’re Bill Roberts? Stay right there.” He shuts down his gate and gets in his car, fires up his flashers and we’re hauling down to get to the OPF. I pull up in front of the OPF, and Phil Glynn who was the deputy Orbiter Project Office manager comes out and he starts cussing at me, “Where the hell you been? Give me that damn thing,” and he goes walking in the OPF. That was it. I ended up finding a hotel room and spending the night, flew back to Downey the next day. Those were the kinds of things we did back then. We react quickly out here to do whatever it took to keep these vehicles going.

Ross-Nazzal: Things were on the fly so to speak.

Roberts: On the fly but within experience. These people out here in those days were all well seasoned from the Apollo program. Like I said, I was the young kid on the block. I was not around during the Apollo program. Those guys in the manufacturing and engineering workforce were all seasoned veterans from the Apollo program, so I was working with the benefit of that experience base around me.

Ross-Nazzal: It sounds like you had a partnership with KSC, and you had a NASA counterpart at JSC. What was their function?

Roberts: They reported to the OPO directly. They were the eyes and ears of the Orbiter Project Office manager. In those days my NASA vehicle manager counterpart at Johnson was Harry [W.] Byington. Harry has since retired, and we still stay in touch. Dave [David E.] O’Brien [III] was another counterpart. Al [Albert L.] Branscomb [Jr.] is my counterpart now in my transition and retirement job. He’s working for USA [United Space Alliance] right now.

All three sites, you had to have information transfer. Whatever was coming out of Johnson Space Center from a requirements point of view—new requirements, or mission operations—we would have conference calls with all of us to make sure that everybody knows what is going on in every site. This site here is my responsibility to let everybody know that our engineering products, our manufacturing, delivery of those mod kits, we’re all working per plan. If we had issues, these other two guys would know what our issues were and challenges. Joe Tague at Kennedy Space Center would talk about their issues. All three sites were fully up to speed and working off the same page.

NASA has got its own way of doing business. They’ve got a huge organization on this project. There’s multilevels of this project organization. We operate at Level III. There’s Level II and then there’s Level I. Everybody interfaces within those levels. There’s been a lot of times when there may be a Level II requirement or payload requirement or something like that and we get totally blindsided. It was the vehicle manager’s job down at Johnson Space Center to be the eyes and ears to make sure that we the turnaround flow guys don’t get blindsided that there may be a stop work situation. That’s how we all worked together.

Ross-Nazzal: Were there any major technological challenges that you faced as project manager while working on OV-103?

Roberts: The vehicle was designed in the ’70s. The GPCs [general-purpose computers] are old and slow, so all of our upgrades—especially the avionics and communication areas—were limited by the capability of our processing. We really wanted to improve certain areas of the vehicle, like TVC [thrust vector control] actuators to get better response, but it was all limited by the capability of our computers and the processors and the MDMs [multiplexer].

As the vehicle got older, the program realized that we were limited. Sure, there’s great things going on out there in other missile programs, fast processing of data, but we couldn’t do that because you couldn’t gut the vehicle to the point where you changed out your GPCs. One of the mods did improve the GPCs, but it was a small improvement compared to what the capability of computers are today.

When I was over on that missile program, the computer that we ran that missile with is so much more performance than the six that we have, five plus one spare. Just unbelievable. I would say the biggest challenge was to upgrade the vehicle as much as you could with the limitations that you had. There were certain areas you just could not upgrade because of those processing limitations as a result of your GPCs.

Ross-Nazzal: Talk about that first orbiter maintenance down period in 1992, which took place at KSC.

Roberts: There were two major mods that we did on that vehicle. One was getting rid of the beryllium brakes and putting in the carbon brake system. That was more of a health issue, and the beryllium brakes didn’t perform as well as the carbon brakes. We also put in nosewheel steering on the vehicle, because the vehicle was designed without nosewheel steering and basically utilized rudder pedal braking.

The other significant mod that we put on the vehicle was the drag chute system, which actually took chunks of structure off the vertical stabilizer and replaced it with another piece of structure that had the drag chute canister and mechanism and electrical all in it. Those were all built here at Downey. The folks at KSC during that OMDP felt that that was a little bit challenging for them from a ground operations point of view, so we put together a team from manufacturing here headed by Nelson Barter and they were down there for about three and a half weeks.

We set up a manufacturing team to do the physical work. We also had a dedicated engineering team and a release desk for any design changes at Kennedy Space Center. So we had a team of probably about 75 people that went down there and did that drag chute mod. We did the improved GPC mod at that time, but that was basically changing out some cards.

The next OMDP though was the significant one in the life of [OV-] 103, because that’s when they did all those weight savings mods. They took the payload bay doors off the vehicle; they stripped the AFRSI [advanced felt reusable surface insulation] off and replaced it with FRSI [felt reusable surface insulation]. Flipper doors mod was done.

Ross-Nazzal: This was the mod that was out at Palmdale?

Roberts: Yes.

Ross-Nazzal: How long did that modification last?

Roberts: Ten months. From the time it landed to the time it next flew is about a year and two months because there was some processing down at Kennedy Space Center. When they delivered the vehicles out here to Palmdale they had to pull all the OMS pods off, the APUs [auxiliary power unit] were out, they had to drain the crossfeed system on the OMS/RCS—they had to basically safe the vehicle to a certain level that it would be acceptable to work around at Palmdale.

Ross-Nazzal: You said they pretty much tore it apart. Did that mean that they really took it down to the various pieces?

Roberts: No. They gained access. They didn’t take the wings off, they didn’t take the vertical off, but they took the payload bay doors off. They stripped the thermal protection system off the upper surface of both wings; parts of the forward fuselage, parts of the aft fuselage. They removed the flipper doors off the upper surface of the wings and replaced them with new flipper doors. They rewired the whole crew module for MADS, the modular auxiliary data system.

Sechrist: You cannot underestimate the significance of chopping holes in the 1307 bulkhead to fix the OMS pod. That was really a major decision because that’s a main structural element. All the thrust from the engines goes through it, the wing box section comes through it, that’s the lifting point for the major weight when you’re raising it up—very significant exercise.

Roberts: I have photos of a lot of that. Part of this job that I’m doing right now, recordation, we’re building a document called the data pack for each vehicle. It basically shows the history of each vehicle. We have these drawings called build and flows, which actually is a graphic representation of what went on here at Downey utilizing all of our ground equipment and how the forward fuselage was put together. It’s a flow diagram, graphically. Somebody actually drew these things back in the ’70s. Then adjacent to that graphic we have the actual photograph of the work we’d done. In this document—it’s probably about 400 pages for 103—we have every component assembly photo taken. You basically watch 103 being built from skin to completion.

Part of that story is this clamshell event that took place up at the pad prior to STS-26, so I have those photos that we’re putting into that data pack. If you look at [OV-] 103 today, and you peel back those blankets on the 1307 bulkhead, you still see the cover plates that cover those holes. They’re there. The OMS pods are interchangeable. I’m not quite sure which OMS pod that was that flew STS-26 and where it is now. The pictures are pretty dramatic, especially when you see George Gallagher in his bunny suit lying on his stomach with his arms reaching through.

You know what a zerk fitting is? It’s a metal-tipped thing that receives a pump hose that clips onto it. You ever seen a grease gun in an automobile, how they grease fittings underneath cars? It’s similar to that, but this is a little bit more elaborate. It had a pressurized pump on it. Furmanite would not allow—and still doesn’t—NASA or Rockwell or anybody to do it but his guy. So George Gallagher gets the clamshell mounted, exposes the zerk fitting and call in the Furmanite guy. Here he comes, they have pictures of him hooking on—flips the switch, backfills this clamshell. That was it. Then we had to let it cure for 72 hours before we actually put the hypergolics in the plumbing system again. Pretty neat operation.

Like I said, that would never be allowed in today’s environment, because there was risks in doing that, huge risks. But it all worked. Our engineering guys did that whole analysis. Those holes were not just located based on George’s arms going through; it was also avoiding load path points because there’s stringers on those bulkheads. So it was an interesting operation, it was fun.

Ross-Nazzal: How much time were you spending out at Palmdale when they were working on the OMDP?

Roberts: I would go out there three days a week. I’d drive up on Monday morning. We had an 8:00 status meeting for the week’s activities, and I’d stay through Wednesday evening. Whenever you’re up there with the vehicle being worked on, there are always issues and challenges that come out of the floor. So when I’m up there I’m working with them and understanding the problems and where they need help, and I’m on the phone with our guys down here at Downey.

It also helps when I come back and go face to face with the guys in Downey and explain the situation. I’d spend Thursday and Friday at Downey, then the next week I’d go back up there. But it was good, it’s always good to be around the vehicle when there’s work being done. During the implementation of the requirements, I plan to be in Florida at least half the time. That’s all three vehicles. I’ve already forewarned my wife I might not be around as much as I have been in the past several years, but our kids are getting old enough. It’s not like they’re little toddlers.

Ross-Nazzal: I understand that they were installing the Orbiter Docking System during that upgrade.

Roberts: All the orbiters were built with the airlock internal to the crew module. Because of the upcoming planned Station rendezvous and docking missions, one of the mods was to put in the Orbiter Docking System. The Orbiter Docking System had to be put on top of an external airlock, so they removed the internal airlocks and put them in storage. They built up new airlocks here at Downey and put them in. That was significant because there’s a transfer tunnel involved, there’s specific loads.

I’ll never forget. I was there when they removed the manufacturing access panel off the 576 bulkhead of [OV-] 103. It’s a tall rectangular square piece of metal with a hole in it, and that’s where the 576 hatch was. There’s two O-rings adjacent to each other going around the perimeter of this manufacturing access panel. Obviously those need to be sealing, and one is redundant to the other. When we pulled off the access panel and looked at those O-rings there was some cracking.

We had already received our replacement O-rings in preparation for when we put this thing back together. It was the same supplier, so that made another big ripple effect and we had to come out with a redesign. It was a materials issue. They were supposed to be certified for 100 missions. At that time [OV-] 103 had flown 21 flights. Our materials and processing guys did some slicing and dicing and looked at it and figured out why they were getting premature cracking. They came up with a redesign recommendation, went to the supplier that supplied these O-rings, and they were available by the time we were ready to button up the forward fuselage.

The docking system itself is actually a Russian-supplied piece of hardware. There’s a lot of pyrotechnics on that Russian hardware; in case you can’t undock you got to blow it off. Also the Russians use materials in their electrical connectors that are a little bit hazardous that we don’t design hardware with. Part of that fleet safing document material—that came from left field. We did not know about that until probably two months ago, so we had to rewrite our documentation and identify the Russian connector pins. It’s cadmium, susceptible to corrosion that outgasses similarly to asbestos. The docking system was not put on the airlock at Palmdale, it was put on at Kennedy Space Center after it returned.

Ross-Nazzal: Were there other challenges that you faced during that OMDP? Or was that the biggest?

Roberts: That OMDP had more significant modifications and upgrades to an orbiter than ever before, and [OV-] 103 was the first vehicle to get all of those upgrades. Any time you put a mod on a vehicle for the first time you’re going to find surprises. Yes, we had significant challenges during that time, and we worked through them.

We had a system within the program called EO to follow. An EO is an engineering order, which is basically a small change on an 8 ½ x-11 [inch] piece of paper that you can attach to the D-size [22” x 34”] drawing. The guys on the floor document what the problem is, and they also document their recommendation to resolve this problem. That piece of paper is called a problem report or a test preparation sheet, gets forwarded through our vehicle project office, and then we work it with our design team to see if they get concurrence to their recommendation. If they concur, then we stamp it as an engineering order to follow.

Once they get that, the guys at either Palmdale or down at Kennedy Space Center can progress with work, meaning you don’t get a stop work. You don’t have to wait for your released drawing; you can work off of this paperwork. That system allows the floor to progress through their plan without waiting for engineering drawings release. That first OMDP we had a lot of EO to follow, I mean a lot. They’re small changes, but it was a real big mod period.

The best thing about it from my perspective that I ever did was spending all that time up there, because when you’re face to face with what the problems are and talking to the guys on the floor, you understand what the problem is, and the better you understand the problem, the better you can transfer the information to the designers and such.

Ross-Nazzal: Why were so many changes required? You said it was the biggest effort at that point.

Roberts: We were getting ready for the Station flights. The biggest goal coming out of that OMDP was lightening the vehicle and gaining performance, whereas the first one was lessening risk on the vehicle. We had all these Station flights planned stacked up through the ’90s and into early 2000s, so that was the big reason behind all these upgrades. Performance gain.

Ross-Nazzal: You mentioned that [the docking system] was provided by the Russians. Did you do any work with that? Did you ever work with the Russians?

Roberts: Myself no. At the flight readiness review the Russians were sitting right next to me, but I didn’t work directly with them. We had an integrated project team here that was the docking system team headed up by Steve Cavanaugh. Tim Cook was his chief engineer. They worked directly with the Russians preparing the external airlock and the docking system, integrating our requirements with the Russian hardware and Russian hardware with our requirements.

The docking system that the Russians handed over to us basically was designed to be compatible with the Mir [Space] Station. We took those design capabilities and evaluated them to make sure that they were compatible with our systems. Once we understood their system, we were able to design our airlock to make sure that it was compatible with their docking system.

Ross-Nazzal: Doing some research, I also found that in 1999 there was some damaged wiring on Discovery.

Roberts: Actually the wiring issue was a result of the first flight that Eileen [M.] Collins was the commander [STS-93]. It was on OV-102 [Columbia]. They almost lost an SSME [Space Shuttle main engine] nine seconds after it launched, but she did some quick flipping of switches. When it came back we did the initial failure analysis of that in-flight anomaly, resulted in some arced wiring in the crew module lower bay.

That investigation resulted in inspecting all the vehicles. I think we were down for about nine months because we did find a lot of insulation cracking, some hot spots. [OV-]103 had some hot spots—never arced like in 102—but 105 [Endeavour] even had some. During that down period all the vehicles were rewired. We found out through that investigation that there were some critical command paths that were redundant to each other that were adjacent to each other. If we had a wiring issue it would take out the redundancy, so we had to separate and relocate those paths.

That was a major setback in the flight schedule, but luckily nothing ever resulted. It was an eye-opening experience that our original design, we found that we could have done a better job twenty years prior to that. The nice thing about it is these orbiter vehicles are wired by hand, and they’re hand-tied bundles. They are not like the kind of wiring you see in an automobile where there’s looms of prefab [prefabricated] wiring. They have sets of various gauge wiring routing through the vehicle. At the original design, they were basically open and exposed with yarn tie bows. One of the things that resulted from that is that we did do some convoluted tubing protection of these wiring bundles.

After we did this wire inspection and rewiring of the vehicle and convoluted tubing modifications we have gone in and looked at and inspected those areas that were problematic earlier, and what we’ve done has resolved that problem. There are no more hot spots, and the cracking of the insulation is minimized now. Part of our normal turnaround maintenance is to inspect certain areas of the vehicle and eliminate those areas that we see any kind of aging in the insulation. Just normal turnaround maintenance process, where it wasn’t before.

Ross-Nazzal: We’ve talked about some of the orbiter maintenance down periods. Were there any additional plans to modify the orbiters that got scrapped because it would take too long or would cost too much money?

Roberts: The OMDPs are not contingent upon modifications. The OMDPs are scheduled based on time life cycle inspection requirements, structural inspection requirements. Every few years you got to go in and look at these things based on our OMRSD requirements. Those scheduled down periods are opportunities for upgrades. We also named them OMMs, which are orbiter major modifications.

To answer your question, yes. If we chose to fly these vehicles another 20 years, 10 years, even 5 years, there would have been OMDPs scheduled based on the inspection requirements. And yes, on my desk I have a booklet this thick [demonstrates] which is unincorporated upgrades on 103. There’s a lot of modifications that were bought and paid for by NASA that never got implemented on the vehicle because the opportunity never existed.

These modifications are not safety-oriented, because anything safety gets put in the vehicle. An upgrade could be performance, but it also could be improved ground turnaround processing. The greater majority of the ones that are not incorporated are typically ground turnaround processing. Those are the lowest on the importance level when it comes to implementation. Safety is number one, flight performance number two, and then ground turnaround processing. Ground turnaround processing just saves you a little bit of time during turnaround of these vehicles.

Ross-Nazzal: Let’s talk about your time at the Mission Evaluation Room. Was there ever a point during a mission where you were called on to fix an in-flight malfunction or anything at launch?

Roberts: My primary job there was to receive and understand the in-flight problems and work directly with our folks here at Downey. The one event that became the most challenging problem in my experience there in the MER was the drag chute door that came off of STS-[95]. The door fell off on launch, and Senator John [H.] Glenn was on the vehicle. I was here at Downey for the launch countdown, and I had to fly to Houston that night.

I brought with me six folks and we set up a team down there outside the MER to see what could have happened, why it could have happened, can we reenter with that thing open in the back? We were working directly with Linda [J.] Ham. She headed up her flight ops team across from our design team. We had meetings three times a day outside the MER. As you know we landed without doing anything, and that chute wasn’t even heated up. The entry heat environment on the back end of the stinger is nil, we found out.

That was probably the most challenging mission I ever was involved with because we were dealing with something that none of us ever really thought of as possible happening. We came up with all these thoughts as to why it happened. What it ended up being was the acoustic dynamics of main engine startup. If you hit this one unique frequency band, the shear pin on the drag chute door had the ability to break away.

We had flown with that design for I forget how many missions, and it never—but for whatever reason because of the atmosphere, humidity, it hit this certain frequency band that sheared those pins. The door fell off during launch. So we made a stronger pin, just bought margin with a stronger material. They still fly the same drag chute shear pins. They break away when they fire the motor on the runway. Just shear that door off, the door goes spinning off like a Frisbee and the drag chute pops out.

In the MER we’ve had a lot of issues. I remember early on when I was the vehicle project engineer I was working in the MER, and we had a debris hit on the underside of the vehicle. We didn’t have the photo capability like they do now post-Columbia. They have great cameras now that can see everything. We had a suspect impact on the underside, and we were being asked if we could clear for reentry. This was day one of the mission. We said, “We can’t. Rockwell has no input until we see the bottom side of the vehicle.” The vehicle rolled over so the belly faced out to space. Miraculously, we saw the bottom side of the vehicle, and we saw that the impact was minimal. We said, “Okay, we’re good.”

We got things done quickly back then, with a lot less technology. I think the program probably had a lot more influence then than they do from a programmatic governmental thing. The time it took for us to put in a request to get a picture of the bottom side of the vehicle to the time we got it there in the MER was maybe 15 minutes. That quick. We got lucky with our orbital dynamics. Everything lined up perfectly, and it happened.

There’s been other issues. We lost tiles on the OMS pods. We had some real good thermal modeling data. We designed these vehicles with a lot of conservatism, because we didn’t have any thermal modeling experience when these vehicles first started flying. But we started getting real smart real fast as we flew more and more. We knew that we overdesigned these vehicles from a thermal point of view such that there’s certain areas of that vehicle you could really take some substantial tile damage and still reenter without any issues.

One time we had a waste water dump nozzle that had a big icicle sticking out the side. It was probably six feet long, it was a huge icicle. It was blowing ice onto the upper side of the left-hand payload bay door. This icicle was actually connecting to the upper side of the payload bay door. So we mapped out the ice coming off the nozzle. We drew this map trajectory—it’s not going to hit the OMS pod, it’s not going to hit anything else. But when that payload bay door comes closed and it brings this icicle with it, what about the vertical tail? So we mapped it out.

We had a reasonable confidence that also if it broke loose it wouldn’t do any damage, so we reentered. And when they got down to the runway, part of that icicle was still on top of the payload bay door. It didn’t get hot enough to even melt off. So there’s a lot of conservatism designed in this vehicle. That upper side of that vehicle is very benign compared to the lower side of the vehicle.

Ross-Nazzal: Were there any major issues that you recall with Discovery when you were working the flight readiness reviews or with the mission management team in preparation for flight?

Roberts: Yes, the first flight after the second OMDP. Like I said earlier, that vehicle was torn apart and put back together. We had to present my presentation to the mission management team at the flight readiness review. I spoke for an hour and a half to them going over all the modifications, not only familiarizing them with the mods but also briefing them to have a certain level of confidence that this vehicle is ready to go fly. Not only telling them what the mod was, but how we tested the vehicle after we put the vehicle back together so that now the vehicle is capable to fly. That was a lengthy process. That was also the first flight of the docking system, so I had the Russians there with me. When I was done talking they got up and spoke about their docking system and why it would work with our Station requirements.

Typically when we had a significant problem that had to be presented at a flight readiness review, we would bring in the system expert. We had some MPS [main propulsion system] flow control valve issues and we brought in John Kramer who was a subsystem manager. We had APU problems early on, and we brought in Tibor Farkas who was the subsystem manager for the APU group.

We actually had an OMS/RCS problem that we had a burnthrough on the nozzle, one of the forward RCSs. It was the result of unburned propellant adhering to the nozzle material. It was termed FORP [Fuel and Oxidizer Reaction Products]. We had to do tests out at [NASA] White Sands [Test Facility, White Sands, New Mexico]. That problem was a challenging problem where we brought in all of our OMS/RCS and materials and processing folks. We presented that presentation at the flight readiness review.

[OV-]103, early on in the late ’80s, we had a left wingtip tile that actually burnt through and had structural damage. Not bad, it was just some blistering of the aluminum. We had to understand that problem and tell the program why it wouldn’t happen again. I think we came up with a redesigned tile, a little thicker tile. It was the only vehicle to ever do it. It was a DTO [design test objective], one of those tests that we were simulating the high cross range heating.

In fact Bob Wilkins, who was the lead structural manager for the wings here at Downey, years later I was in his office talking about some other problem. I looked over against the wall there and there’s the wingtip with the tile and the burn. He’s probably still got it in his garage somewhere.

Ross-Nazzal: You had mentioned after the OMDP that you do a testing and the checkout of the orbiter. Would you explain some of the different tests and how you check out the orbiter and say it’s ready to go back?

Roberts: As the system got put into the vehicle, you do limited system testing, start applying power, because in the OMDP we’re definitely in a power down. That vehicle is not to see any electrical power whatsoever. As you start building the vehicle back up with new wiring and new components, you start getting the vehicle to a configuration where you can apply power to it. So as you start progressing and building the vehicle back up, you identify certain configuration milestones that you can start applying power to certain areas of the vehicle. You start branching off throughout the vehicle to the point where you have full system power up. Before the vehicle is ready to ferry up there you get to the point where you power up the whole vehicle and start flipping switches.

I remember when we put 103 together there was smoke in the crew module. They obviously didn’t burn through, but it overheated some insulation. Emergency shutdown. They found out it was just wires arcing, laid wrong, bad solder joint. There’s a lot of these kind of things that have happened. But I remember that day, alarms went off. It’s just all part of putting things back together.

You get to the point where everything is operating properly. You put the rest of the vehicle back together, you get ready to ferry. Before you ferry you obviously test the upgrades here in California, and you hand over the vehicle back to NASA to ferry it back. Part of that handover process is we have a ferry flight readiness review. All the subsystems sign on the dotted line that this area got upgraded by this mod and it was tested on this date and certified safe. We get to ferry the vehicle back, then Kennedy Space Center would go ahead and redo all of our tests. I never really understood why they did that, but they did. So we had that data, more supplemental confidence to presenting at the rollout review or the flight readiness review why this vehicle is ready to fly.

Ross-Nazzal: I had two other questions for you. What do you think was your biggest challenge while working as the vehicle project manager on Discovery?

Roberts: Probably the biggest one was that clamshell thing, but like I said earlier, every day was a different day, and every day was something new. There were easier days, but there were more difficult days than there were easier days. The OMDP up at Palmdale, that was a lot of work. But it was good work, it was fun work.

Every launch countdown was difficult. I don’t mean difficult to get the vehicle launched. It was difficult mentally for me, because in my mind I’m responsible for that vehicle. I know there’s thousands of people here that all feel the same way. But I’m the guy with 103 stamped on my forehead.

I can’t really say one real event was challenging. They were all challenging days, and every day I enjoyed. The only reason I left that job was because I didn’t want to move to Houston, but back on the program for the last five years those two jobs that we’ll talk about tomorrow, they were just as challenging. With my knowledge coming off of managing 103 for all those years, it was a good fit for me to do those two jobs.

Ross-Nazzal: Anything that you can look back at and say, “This was my most significant accomplishment”?

Roberts: No, it’s all good. Really, it really is. Every day was significant, and I felt I was accomplishing something every day. There wasn’t one day I was bored. Every successful launch was a significant accomplishment, every successful landing. Every time I was I the MER and there were minimal in-flight anomalies I took that personally, because I took managing this vehicle personally.

The people I worked with on this vehicle, I had personal relationships. It wasn’t just a job; it was a life that I was doing during those years. For me to give it up because I didn’t want to move to Houston, that was something that I really had to think hard on. But I’d been to Houston enough to know that I didn’t really want to relocate my family, plus I grew up here in southern California.

It worked out for the best. Luckily there was other opportunities for me here within the company, and I came back to the program. So I was there at not the very very beginning of the program, but very close to the beginning, and now I get to put the program to bed. It’ll be sad but it’ll be kind of nice that I’ll be there at the end. I know there’s a lot of folks right now in our office that are leaving the program because the work is scaling down. They’re leaving the program, going on other programs, for the first time in their career. I’ve already experienced that, and it’s a good thing. I keep telling everybody it’s a good thing to go do something different.

A lot of these folks are saddened that they won’t be there at the end. They won’t have an opportunity to see the vehicles when they stop flying. I’m fortunate that I will be there. It’s an interesting process that has taken place defining how to safe the vehicle, but it’s even more interesting to watch the politics involved on who gets the vehicle after they retire. I just sit back because it really doesn’t matter where they go as far as I’m concerned, as long as these vehicles are safed per our requirements. These safed vehicles have a different set of ferry flight requirements than would a normal vehicle, because basically all the subsystems are inerted or deactivated except for one, the Hydraulic System 1, so we can deploy the landing gear when we get to a given site.

The ground ops guys at KSC, they’ve got certain challenges involved because wherever we land these vehicles at any site they have to do crane ops to demate the vehicle from the SCA. There’s only been two other occasions when crane ops have been used to demate an orbiter, and that’s been down at Alabama they demated for a World’s Fair [in New Orleans, Louisiana in1984], and then out at White Sands.

Ross-Nazzal: I just thought of one last quick question. When you were in launch control was there ever a point where you had to recommend that the mission be scrubbed or delayed because there was a problem with Discovery?

Roberts: No, at least when I was in charge.

One of your questions there about other vehicle managers and did we interact—yes, we did. When I was the vehicle manager there was Ed Burke who was OV-105 vehicle manager, Randy Rubens was OV-104, and Mark Debaro was OV-102. I was the first guy to come in. This is the next generation vehicle managers, because before that there were some older guys that are retired, and Bill Fleming moved up. Bud Williams retired, he was the 104 previous vehicle manager.

Ed Burke is actually a year younger than me, and he got named as the vehicle manager for 105 when 105 was being built up at Palmdale. Prior to that I had already been named as the 103 vehicle manager here. So Ed and I were the first two young guys coming into the vehicle project office, then Randy and Mark joined us later. When the four of us were working together, because we were all within one or two years of each other, there was definite competition between the four of us as to who could fly with the fewest in-flight anomalies, who could turn around the vehicle with the fewest amount of engineering changes.

We were all in our early 30s then. I think Ed was married, but the other guys were not. We all played golf together. Tournaments, we were always a foursome. It was fun hanging out with those guys in those years. We’re all good friends right now. Ed just left the program. He was the vehicle manager but then they decided all vehicle managers would be down at Houston. Well, Ed didn’t want to move to Houston either so he said, “Fine, I’ll be a vehicle PE and I’ll just support you from here in Huntington Beach.” He’s been doing that for the last three years.

He’s got a job where he’s the change board chairman for the Boeing 787 program up in Seattle [Washington]. So he’s doing that probably for a year. He’s going to be TDY [temporary duty] going up there for two weeks, coming home for a week. Randy works on some DARPA [Defense Advanced Research Projects Agency] projects out of Boeing. Mark works for Future Combat Systems out of Boeing Huntington Beach. So we’re still all around. It’s fun. We’re still good friends, we all had kids at the same time. But like I said, we all have those vehicle tattoos on our forehead.

Ross-Nazzal: Do you want to talk some about working with the Shuttle Program in general, and then working with USA? Anything there you’d like to touch on?

Roberts: Well, working with the Shuttle Program, my involvement with the program office out of Houston became active when we were briefing the program of our proposed modifications and turnaround upgrades. Then we would have to work with them obviously during the rollout reviews and flight readiness reviews.

Leading up to that, we were working intimately with their folks and their support staff, because presenting a presentation to the program office versus just our Level III project office, it was much more significant. You had certain protocols involved. So we got to know those folks quite well over the years. Especially Sue Pinch headed up that office down at Kennedy Space Center. That’s where all the flight readiness reviews were, down at the mission briefing room at Kennedy Space Center. All of the launch site flow reviews were in Building 1 at JSC. We worked with their folks quite closely. We all had a good working relationship.

When I went away to other projects and then I came back, I had to present the design certification review material to those program offices. A lot of those people were different people also than when I came back, because I was gone for four years. It’s like any other company, people move on.

I will say one thing; because I left this program and I went to that other program, I realized very quickly when I was on that other program just how mature this program was and how professional the people on this program were compared to other programs. That has everything to do with the amount of experience that the people in this program have.

The other program I was on, there was a lot of people with a smaller amount of aerospace experience. This program has been in existence 30, 40 years now. I’m still amazed how this vehicle got designed back in the ’70s without computers, especially since this data pack that I’m building right now and looking at this historic material. I have the drawings, not detail drawings, but the overlay drawings for that vehicle. I’m amazed what they tried to sell. It is truly amazing the capabilities of that vehicle with the air-breathing engines and the RMS [remote manipulator system] down the centerline. The forward RCS [reaction control system] was completely different. It’s pretty neat.

I’ve definitely experienced a lot of different things, worked with a lot of different people. Like I said before, these vehicle project jobs were something special, and not too many people got to experience that. I feel very fortunate about that and the people I worked with over the years.

Ross-Nazzal: Is there anything that we haven’t talked about about Discovery that you thought we might?

Roberts: I think you’ve been very thorough. You obviously have good knowledge on this program, because you can talk the language.

Ross-Nazzal: Well, we’ve been doing interviews for quite a while.

Roberts: Yes, and you’ve interviewed some very important people in this program.

Ross-Nazzal: It’s been a big program for NASA.

Roberts: I’m sure you’ve enjoyed it too, listening to all these people talk.

Ross-Nazzal: Oh, it’s great. It’s wonderful. Like you said, it’s amazing how the vehicle operates, and how there’s so many systems on the vehicle, and how it’s able to fly and return safely. Just amazing. So many people, so many different pieces of hardware.

Roberts: When I first started looking at the old build photographs and graphics—and I actually saw the Palmdale facility before what it is today and how they actually had to build Building 150 to accommodate two orbiters in build flow. Then you start seeing the ground service equipment, the tooling coming together. This is all in the ’70s. They’re building this space vehicle this size, and the largest thing they’ve ever flown before was an Apollo capsule. That’s pretty gutsy, especially without computers.

Sechrist: We had four IBM [International Business Machines Corporation] 3090s [mainframe computers], did a lot of mainframe computing. It wasn’t like today, that’s for sure.

Roberts: It’s truly amazing that they did what they did with the resources that they had. Obviously they had a lot more manual resources then. I think when I got hired there was 10,000 people working here. I believe it was 4,500 in engineering and several hundred in quality and logistics, but the rest were manufacturing. Parking lots were full. The lack of computer technology is conversely proportional to manpower because the more computer technology you have, the less manpower you need.

Ross-Nazzal: Well, I thank you very much for your time today.

Roberts: You’re welcome.

[End of interview]