NASA STS Recordation Oral History Project
Edited Oral History Transcript

George D. Hopson
Interviewed by Jennifer Ross-Nazzal
Huntsville, Alabama – 20 July 2010

Ross-Nazzal: Today is July 20th, 2010. This interview is being conducted with George Hopson in Huntsville, Alabama, as part of the STS Recordation Oral History Project. The interviewer is Jennifer Ross-Nazzal. Thanks again for talking with me this morning. I certainly appreciate it.

Hopson: Been looking forward to it.

Ross-Nazzal: Well, good. I thought we’d start off with an easy question this morning, if you could tell us briefly about your career with NASA.

Hopson: I worked at General Dynamics [Corporation] when I first got out of school. I got my master’s degree in mechanical engineering, and I worked there for about eight and a half years. I was a senior propulsion engineer. A Marshall [Space Flight Center, Huntsville] team came through Fort Worth [Texas] recruiting people for the Apollo Program, and I went down and interviewed. Shortly thereafter I got an offer from them, and I knew in advance it was going to be a cut in pay. My wife and I had been through Huntsville. She knew how much a box of Tide [laundry detergent] was supposed to cost in Fort Worth, and we figured it was taking a pretty big loss in pay by coming here because housing, everything, was more expensive except water and electricity. Also, my salary was about 10% less. Anyhow, I decided that the job interest was more important than the money. In the meantime we decided we might rather go to Houston [JSC], and I put in an application with them. The day after I accepted the job with Marshall I got a similar offer from Houston. I told them I was already committed.

I went to the University of Alabama [Tuscaloosa]. I was in the Marine Corps near the end of World War II, and they gave me the GI Bill [also known as the Servicemen's Readjustment Act of 1941]. They paid for my school supplies and materials plus a little money. I finished my bachelor’s degree in 195[0], and then I got called in—I was a second lieutenant in an Army Corps of Engineers Engineer Combat Battalion. That was when the Korean War was going on, so I was over in Korea during that war. Then when I came back, I had additional GI Bill and I went back down to the University of Alabama and got my master’s degree. Then I went to work for General Dynamics before I came to NASA.

Ross-Nazzal: Besides working on the Saturn, what were some of the other projects you were involved in?

Hopson: When I reported to MSFC I was assigned as the chief of the Propulsive and Main Jet Heating Unit. Back then—they don’t have units now—a unit was about ten people. I was chief of the unit which had responsibility for calculating and determining the protection requirements for the Saturn base heating. The first and second stages of all Saturn vehicles [rocket] had a base heating problem. Because there were several engines clustered together, exhaust plumes would interact with each other and some of the exhaust gases would flow back up into the vehicles base region. We had to determine what the heating rate was and what kind of heat protection were required. Later I was chief of the Fluids and Thermal Branch on Apollo, and since then I’ve had just about every job that you can have in Marshall, except Center Director. My assignments have included Director of Systems Dynamics Laboratory, Director of Systems Analysis and Integration Laboratory, MSFC Chief Engineer for Space Transportation Systems, Chief of Skylab Thermal and Environmental Control, and Manager of MSFC Space Station Freedom. My last position was “NASA Fellow for Propulsion” for all NASA Centers.

Ross-Nazzal: Quite a long career.

Hopson: Yes. I was a co-chief engineer for the Space Shuttle Main Engine [SSME]. We had a lot of trouble developing that engine. Management decided we needed one person at Rocketdyne and one person in Huntsville so there were two chief engineers, and you were chief engineer wherever you were, [either] at Rocketdyne or Marshall. We alternated on six week centers for about a year. Every other six weeks I was at Rocketdyne. I had an office out there, and they let me come to their technical meetings. We finally got the engine certified.

Ross-Nazzal: Would you tell me about that test program? When did you serve as the co-chief engineer for the SSME?

Hopson: I was formally made the Co Chief Engineer two or three years before STS-1 flew, but before then I worked on the engine before being assigned as the Co Chief Engineer. The SSME is a very advanced engine. It uses what’s called a staged combustion cycle, which is different than what we used on Saturn. On Saturn the engines used a gas generator cycle. The performance is higher with staged combustion but the design complexities and pressures are also higher, so you’ll have more test failures. It’s a more difficult job to develop a staged combustion engine than it is for a gas generator engine.

The heart of a rocket engine is the high pressure pumps. The pumps had turbines which were driven by hot gases. On the gas generator cycle you use the gas from a combustion component. The combustion exhaust gas goes through a turbine to power the pump, and then it is exhausted overboard. In order to get a temperature that the turbine blades can stand, you have to be either fuel or oxidizer-rich. We chose to operate fuel-rich for safety considerations.

On Saturn we used kerosene and liquid oxygen on the first stage and liquid hydrogen and liquid oxygen on the upper stages. On the Shuttle we used liquid hydrogen and liquid oxygen, but in any case you have to combust the propellant at a mixture ratio which gives lower temperatures than what’s possible. So when you’re dumping propellant overboard from a gas generator cycle, you’re really throwing propellants away.

The main difference in a staged combustion engine is that rather than dumping those gases overboard you put them back into the engine, and you burn them. You have to have much higher pressures to put those propellants back into the engine. The gases have to go into the main combustion chamber, so they have to be at a higher pressure than the pressure in the combustion chamber. SSME was our first staged combustion engine. The Russians had staged combustion engines which had oxygen rich propellants but I think they were only used on unmanned vehicles. Oxygen-rich engines avoid the coking (or soot) problems you have when you combust hydrocarbons at pressures higher than about 1000 psi, but because of the higher oxygen content, they have a greater potential for internal fires due to such things as impact of foreign objects.

We had quite a few SSME test failures on the test stand and had to do some redesigns as we went along when we uncovered some of these problems.

Ross-Nazzal: Would you tell me about the tests that were done out at COCA [test sites] in Santa Susana [California]?

Hopson: Yes, we had three test stands we would test on, A-1, A-2 and A-3 we called them. A-1 and A-2 were both at Stennis [Space Center, Mississippi]. A-3 was at Canoga Park [California], actually on Santa Susana Mountain. The test stands were basically structures that held the engine, propellants and the flame deflector. At Santa Susana the stand was up on a mountain, and the flame was exhausted down towards the canyon. We would test all aspects of the performance of the engine. We did extensive ground testing, and I think that’s why we never had a catastrophic flight failure.

We had the worst test accident that we ever had at Santa Susana. We were performing a half power head test. A power head has two high pressure pumps, the hydrogen and oxygen pumps. The half power head just tested one; in this test we just had the low and high pressure oxygen pumps. We were running the test and were using a flow meter, which looks like a propeller. One of the blades came off the flow meter, and it bounced down the pump discharge duct into the propellant throttling valve (used to load the pump). It caught the engine on fire, and it was terrible. It destroyed both the stand and the engine.

I got the job of seeing to it that the stand was rebuilt properly. They called it an Operational Inspection Review. I had a team of people, and we made several changes in the direction of safety. For example, we found that the water system for putting out the fire was too slow because some of the water lines were dry and the water had to flow in before it’d get to the engine spray nozzles. There were several water spray nozzles directed at the engine in case you had a fire. Some of them were stopped up because they had never been used, and they were rusty. There were several other significant changes that we made. Of course the report went to Stennis, and they considered our findings in the design of the Stennis test facilities. Maybe you’d like to hear in general how we went about the testing?

Ross-Nazzal: Sure, that sounds great.

Hopson: Any component design, even a thermocouple that goes on the engine has to be certified for flight. In order for it to be certified, each of two engines has to experience the equivalent of ten flights. Flight lasts 520 seconds or about eight and a half minutes. You have an engine on a test stand that you call your certification engine. Every part has to successfully complete the certification program, which is about 11,000 seconds of run time for each of the two engines, equivalent to about ten flights. We put the engine through all its paces then repeated the tests on the second engine.

One of the tests we run is a vibration survey. I used to have a car that when you got to a certain speed it made a lot of a noise and once you passed that speed the noise went away. It turned out to be a couple of steel lines that were clipped together. That noise, caused by vibration, is caused by a resonance frequency of the lines, excited at a certain engine power level. We looked for resonant vibration points on the engines because we definitely didn’t want to operate anywhere near a resonant vibration frequency. One of the tests would be a frequency survey which included all flight power levels.

Then we also gave the engine an overtest. We like to ask more from it than what it’d have to give in flight. In flight the highest power level is 104.5 percent. In the certification program we ran the engine up to 111 percent for 520 seconds. If successful, it gave you a good feeling that you had more than what the engine really had to give. We never failed an 111 percent test.

In the certification, there are a lot of tests. You run into problems, and you have to fix them, and then you have to start over on that part as far as test time goes. We never flew any part that an equivalent same design part had not been tested successfully twice the amount of time that we would use it in flight. That’s one of the reasons for running twice ten flights’ worth of time in our certification program. You weren’t necessarily limited to five flights, but that let you fly the first few missions. You’d already tested more than twice the time that the part would have to run on the engine.

As you got into the program and you flew and reflew different parts, we had what we called a fleet leader program. The fleet leader of a pump or valve or thermocouple design was the one that had the most test time without a failure. We never flew any part where the fleet leader hadn’t accumulated at least twice as much time as we flew that part. We wanted to stay well below what the engine was really good for. Fleet leader time included flight, and test time at both Stennis and Santa Susana.

We were flying pretty often, six or eight flights per year. When there was a test failure the Shuttle was grounded until the cause of the failure was determined and corrective action taken to assure that the flight vehicles didn’t have the same problem. Both Rocketdyne and MSFC had standing Failure Investigation Teams which worked together to determine the cause of the mishap. The investigation was treated with urgency, since flight was put on hold until the cause of the failure was identified and corrective action taken. I was chairman of the MSFC Team during most of my SSME work.

Ross-Nazzal: Can you explain for a layperson how it’s possible to run an engine at 111 percent?

Hopson: Well, actually that is a confusing thing. When we first started designing the SSME, everybody—external tank, orbiter, everybody—did their thing, and decided what weight their element required in order to complete a mission. So the engine requirements were based on those estimates. At that time when we first started developing the engine, the requirement was 100 percent power level. We called that rated power level. Later on the weight went up on the orbiter and other elements and we really needed more than what we’d been planning for. We had enough margin in the engine—actually, as I recall, we certified to fly 109 percent. Then later 104 percent, then 104.5 percent were baselined for flight. Really all that means is that flight power levels were 4.5 percent higher than the early plans as to what the engine was required to do.

Ross-Nazzal: It’s an interesting concept when you try and think about it; you wonder how is that possible.

Hopson: Yes.

Ross-Nazzal: Tell me about testing with the integrated subsystem.

Hopson: In a rocket engine, all the components have to play together. They have to all be compatible with each other. The low speed fuel pump has to deliver satisfactory inlet condition to the high pressure pump, same way with the LOX [liquid oxygen] pump. When you talk about a pump, you’re talking about a component. When you talk about an engine, you’re talking about all the components, and they all have to play together in order for it to operate satisfactorily.

Ross-Nazzal: There was some testing done with the integrated subsystem test program out at Stennis. Were you involved in that?

Hopson: Yes. That was where we not only integrated the engine within itself but the testing also included an external tank. The external tank is part of the propulsion system. So what I said about the engine, all the parts having to play together, goes for the whole Shuttle really. That’s where we did that kind of testing. We’d test at external tank at flight temperatures and pressures. We had a propellant depletion cutoff system so that [if] you started running out of propellant it shut the engine down; if you started running out of hydrogen or oxygen it would shut the engine down, because it gets to be catastrophic when you lose the fluid and the pump is operating. First you have cavitation and then you have parts rubbing together.

Ross-Nazzal: Was there ever any point when you were working as chief engineer that you thought this is not going to work? There were fires on the test stand?

Hopson: Yes, we had a lot of problems. Normally in a space program, you really need to start work on the engine earlier than you do the other stuff, because engine time to design and develop is longer than other components as a rule. We used to talk about seven years being about the time that it took to design and certify an engine. I think that’s probably true for the gas generator cycle. With staged combustion it could be longer than that, depending on what kind of problems you run into.

Ross-Nazzal: Were you facing any pressure from [NASA] Headquarters [Washington, DC] or from JSC with the program office? Because the engine was almost a pacing item for the orbiter, along with the thermal protection system.

Hopson: The only real pressure we got is when they thought it was taking us too long or we were spending too much money. They didn’t get into the technical aspects of the engine, except when they limited flight of a redesigned high pressure fuel pump to one of the three engines.

Ross-Nazzal: How long were you chief engineer during the development, design, and test?

Hopson: Two, three years, something like that.

Ross-Nazzal: You mentioned that you were co-chief engineer. Who was your co-chief?

Hopson: A fellow named Jerry Thomson. He started out being the chief engineer for the engine and we ran into all these problems. The program manager, who was J. R. [James R.] Thompson, decided that we needed a chief engineer at Canoga Park at all times. He told me that he thought that I should go out there and stay. Fortunately, he made me co-chief engineer so I only spent half of my time at Canoga Park.

It was bad then, the weekend was bad. During the week there’s action going on. On Saturday I’d go down and work half a day till 1:00. I’d be occupied until about 1:00 on Saturday, then there would be nothing to do for the rest of the weekend. It seemed like everything in California closes down on Sunday. You get that big thick Los Angeles newspaper— [I] really wasn’t interested in a lot of the local news—the weekends were terrible. It was almost like you’d been overseas and coming home for the next six weeks. Those six weeks at home went by real fast.

Ross-Nazzal: Did you play a role in closing out the test stands out in California?

Hopson: No, I didn’t. I guess it got to the point where they felt like they could do all the testing they needed at Stennis. There used to be a lot of action on the A-3 test stand, up on the hill, as they called it, or Santa Susana.

Ross-Nazzal: Would you tell us about some of those tests? How long did they last?

Hopson: A normal flight is 520 seconds, or eight and a half minutes. We didn’t very often run tests that were less than that. An abort type test was more than 800 seconds. It was about half again of what a normal flight would be. I remember one time down at Stennis, J. R. Thompson and I were down there, and they ran a single engine continuous back-to-back abort test. They ran that engine for almost 30 minutes. It seemed like it was never going to come to the end of the test.

Ross-Nazzal: I bet you were biting your nails.

Hopson: Yes. One thing that surprises a lot of people about the SSME is that each of those engines burns 1,000 pounds of propellants a second. When you combust hydrogen and oxygen, the exhaust is water vapor. So when they run a test, there’ll be a big cloud of exhausted water vapor. If the wind conditions were right, and the cloud of vapor floated over you, it would condense because it was cooler in the atmosphere than the exhaust, and it would pour down rain on you. We got wet once in a while.

Ross-Nazzal: I thought we would also talk about your time as manager of the Space Shuttle Main Engine Project, which you accepted in 1997. I did a little research, and I noted that the Block IIA engine was flown the next year. Can you tell us at what stage the Block IIA engine was when you accepted the position?

Hopson: The new engine had not been certified for the safety enhancements included in the Block II engine when I was assigned as SSME Manager. We designed the engine that was supposed to satisfy the flight requirements. Rocketdyne did the engine design. Then as the program moved on, a lot of improvements, such as the external tank weight reduction were made once we had real data to know what was going on. There were two things we could do. One thing, we could increase the payload of the Shuttle. The other thing was to put some of that performance gain (weight reduction) into safety improvements for the high pressure pumps. We did both.

In the beginning, things like structural loads on the whole vehicle had to be calculated. As we got into the program and we had strain measurements from different places and other types of measurements, we found the places where we had overdesigned. The biggest thing was the external tank. We did two weight saving exercises on that. One had to do with cutting out weight where we were overdesigned. The other was later on—I think it saved something like 7,500 pounds. They went to aluminum-lithium alloy from the types of aluminum that we used on Saturn. It was very high strength aluminum. This aluminum-lithium had special strength, [but] also had special problems. There were problems welding it, and it tended to be more susceptible to surface cracks. They worked all those problems out.

Let me tell you about [Block] II and IIA. There wasn’t supposed to be a [Block] IIA. It was supposed to be [Block] II. There were several changes that were going to be made; some of them kind of fine-tuning. The most important one was the new fuel pump. As we went along, the easier type changes that we made, the fine-tuning type things were done pretty early, but the fuel pump was a bear. The redesign of that fuel pump was tough. When I was made program manager, the pump had not been certified. Gene Goldman, SSME Deputy Program Manager, and Len Worlund, SSME Chief Engineer, played key roles in SSME certification, and all other aspects of the SSME program. We got to a point in the program where we’d made some of the changes that were in the direction of safety.

Most weren’t hard to do, but the pump wasn’t ready. So we said, “We’ll fly those. We’ll call it [Block] IIA and that’ll be all of [Block] II except the pump.” Really the plan was to go to [Block] II, but the pump wasn’t ready in time. We went through testing so we had that interim IIA. The IIA engine was certified and flown but it did not have the most important safety feature, the new high pressure fuel pump.

Ross-Nazzal: What things were changed in the [Block] IIA engine from the [Block] I?

Hopson: Of all the IIA changes the most significant one was that we decreased the nozzle area ratio about 10 percent. Basically what that amounted to was increasing the nozzle throat area. And what that did was to lower the outlet temperatures and pressures of the preburners, which supplied the gases to run the pumps. The hydrogen pump was running pretty close to its limit on turbine temperature and pressure, and we wanted a little bit more headroom there. We wanted more margin. When we decreased that area ratio, it reduced the temperatures of the gases that drove the pumps by about 100, 120 degrees Fahrenheit, so that gave us extra margin. As a result we lowered the pump redline limits by about 100 degrees Fahrenheit.

[Makes drawing to demonstrate] This is engine run time, and this is turbine temperature. Of course you start off at the pump ambient temperature, and then you ramp up the engine power to the temperatures that you normally run at. We have something we call a redline. Almost anything that goes wrong in an engine will increase the turbine temperature, so this redline was there in case a temperature increase indicated a serious problem. When it hits the redline, the engine controller shuts that engine down.

On the early pumps the problem was that it takes a finite time to get from the normal temperature to the redline temperature. The pump was designed for minimum weight, and testing showed that if it were damaged it never would survive to reach the redline. It would explode before we ever got there, which would mean you’d lose both the crew and the vehicle. Because those engines are clustered together, one pump explodes and it wipes out all nearby engines and components. The cause of pump failure is sometimes caused by an upstream failure, not necessarily within the pump itself. One example of this was excessive pump temperatures caused by a problem within the upstream preburner element.

I went down to Stennis one time when we had an engine test stand failure, and the Rocketdyne technicians were down there. The pump was partially melted, and they had a crowbar and what’s called a come-along—it’s a thing with gears and a cable, something that you can really pull something with—and they were trying to get that pump out. The point I’m trying to make is it really makes a mess of the engine and would cause failure to the whole engine compartment if you had something like that. The new pump was robust enough so that, even with significant internal damage, we could make it to the temperature redline and safely shut the engine down.

I was on a console at the Cape for the launches. The chief from Rocketdyne and I had a console we looked at. Our job was if there was anything wrong to tell them, and let the flight director decide what to do about it. I would be at the console with Jim Paulsen, the engine manager from Rocketdyne. You had to be at the console three hours before the launch. When they start loading propellants, and the propellant guys are looking at temperatures, pressures and the amount of propellant loaded for the engines Jim and I don’t have much to do for the first hour or two, before propellants are dropped, so Jim and I could look at where there was something interesting somewhere else. We had two TVs. We had one that you could use to call up the external tank, the weather, or anything you wanted to, and we had another one that was just for engine data.

When they say they drop propellants, what they mean by that is they open the valves that let the cold propellants flow into the engine. The engine has to be chilled down in order for it to start. You get bubbles, and boiling, and you want to get all that stuff gone. You want to have liquids at the right temperature and pressure to start the engine. Sometime in the last hour, after they dropped propellants, is when Jim and I really had our job to do.

One thing I need to clarify with you—I told you Jim and I were sitting at the console, and we were the interface with the launch control people. On the headphones we had Honeywell [International, Inc.] down at Clearwater [Florida]—they’re the control system supplier—we had Rocketdyne in Canoga Park, we had MSFC at Huntsville, and we had Pratt & Whitney at West Palm Beach [Florida]. Those were the main ones. In other words Jim and I weren’t the geniuses who knew everything. We had those guys looking at the engine data, and if they saw something funny they’d tell us. Or we might ask them to check something, and they’d do it. It was really a team type thing, during launch countdown.

The only time that I remember that we ever scrubbed a launch for engine reasons was when they were looking around with binoculars they saw a loose test stand pin. The pin was a pretty good size. What the pin was for was holding the rails together for the test stand walkway, and somebody left the pin up there. The question was would it hurt anything when it dropped? Because when they’d start those engines, that pin was going to fall. Jim and I decided that this was not a good thing. It could damage an engine nozzle because it would be sucked right into the airflow going by the engine. So they scrubbed, got rid of the pin, and then launched the next day. The most important thing we were there for was to tell them about any problem with the engine, so I was always at that console with the head guy from Rocketdyne.

Ross-Nazzal: What other type of support did you provide for the flights? Were you involved at all in the flight readiness reviews?

Hopson: I was the chairman of the Level III flight readiness review. For my flight readiness review I had a board of senior engineers. We had one from the Cape, several from Canoga Park, several at Pratt and Whitney and one each from Stennis, KSC, JSC, and USA [United Space Alliance]. So we had a diverse crowd there, all with different interests and different inputs. We would have the contractors go through a presentation of everything about the engines that we were going to fly, what temperature they thought they were going to run at, how much margin did we have between that temperature and the redline.

Those meetings would usually last four or five hours. The people at the remote sites—some of them were hooked in by telephone with a squawk box, and some of them attended our board meeting. The main purpose of the meeting was to decide what we wanted to tell the final flight readiness review board at the Cape about the flight engines. We also discussed the things that were a little bit abnormal but we thought were okay. Usually there’d be about three of those type things at my review. I would poll the board about what to bring up at the Cape flight readiness review and then declare the review over. Then when we’d go down to the Cape for the final flight readiness review and I would tell the board what the issues were. Then the engine contractor, Rocketdyne, would give the engine predictions, such as temperatures and pressures and would expand on the issues and say what our recommendations were. It was up to the board, and they’d vote on whether they agreed with what we recommended or not.

Ross-Nazzal: Would you share some of the details about how the engines are prepared for flight?

Hopson: When an orbiter lands they take all three engines out, and they go to the engine facility at the Cape, a place where Rocketdyne people inspect and repair the engines. The engine has a lot of parts, and those parts all have an operational life. For turbine blades, the operational life in the fuel pump was something like 4,300 seconds. For the upcoming flight we’d tack on an abort flight duration (which was the longest flight) to the accumulated operational time. If the accumulated time plus the abort flight time and green run time exceeded half what we had experience with, then we would change the part out.

Probably the most important thing that we did was run a “green run.” When an engine had flown it would be inspected and overtime components replaced. Then we’d do what we called a green run. We’d send the engine to Mississippi, and they would run a full duration test on it. If it passed the green run you’d put the engine back into the flight engine pool. Our contract with Rocketdyne said that we were to have 9 flight-ready engines at the Cape at all times. They’d have a stockpile of engines down there because we had three orbiters, and that’s 12 engines. We didn’t want to hold up work on anything because of lack of engines.

Ross-Nazzal: How long does it take to prepare an engine for a flight.

Hopson: I don’t know, it’ll vary a lot from one engine to another. If you have to change a lot of components or do something really complicated—time would vary a lot. If you were lucky and you had an engine that didn’t have many overtime parts on it, you would put it on the test stand and green run it, and then put it back into the flight engine pool.

Ross-Nazzal: How well did the [Block] II engine work the first time it flew on STS-89?

Hopson: In the certification program we ran tests at 111 percent and also did the resonance surveys, but most tests were at the nominal flight power level.

There were some unique problems that went along with the new pump that cost us a lot of time and money. A new contractor was building the pumps. Every engine assembly used a dry lubricant. By dry lubricant it means it’s not liquid, it’s a grease-like solid. When you fit a turbine blade into the disk, anything where there’s any moving or rubbing, you use a dry lubricant. We used a dry lubricant called Braycote. There’s a family of Braycotes, and we used one particular member of that family. The main lubricant was molybdenum disulfide, and the Braycote also had some Teflon particles.



Ross-Nazzal: Yes, it’s amazing how that machine actually gets into space. There are so many different components, subsystems, and so many people working on it.

Hopson: Yes, it is. It’s thousands of parts, and there’s a lot of so-called redlines that can keep you from launching. One of the big threats to launching is weather. In order to launch, the weather has to be okay to land at the Cape. If it’s socked in where they can’t see, that’s a “no go” for launch. You have a NASA astronaut pilot that flies around the Cape area, to report what the conditions are at the Cape landing field. So number one, that site has to be open.

Number two, you’ve got two places in the United States to land, and one of them has to be open. One of them is Edwards Air Force Base [California] and the other one is White Sands [Space Harbor/Northrup Strip], New Mexico. Then you have two sites, one of which has to be open, overseas. They call it TAL, transatlantic landing, and the location of those vary. You like to have the two landing sites as far apart as possible, because you don’t want one weather system to get both of them. Before the political situation got to be a consideration we usually had a TAL site in Spain at Mor?n or Zaragoza, and the other one was in North Africa, such as Banjul in Gambia. Later on we changed to where now we could land at sites both of which are in Europe. Usually we use Zaragoza, Spain and Istres, France.

So you have the Cape, you have one of what they call CONUS, continental US, landing sites, and you have one TAL site which have to be open. We have never had an abort landing. The SSME engines are started at T-6 seconds in a sequence that minimizes Shuttle structural loads. If the engines all operate satisfactorily the solid motors are ignited at T-0 seconds. This is to preclude lift-off with a faulty SSME engine, because once you start the solid motors you are leaving whether you want to or not.

Apollo 12—that’s the only Apollo launch I ever saw, and that’s the one where lightning struck the vehicle. When they light the engines the ionized exhaust conducts electricity, so when you lift off the vehicle and its’ exhaust is a huge lightning rod. Approval to launch is very sensitive to lightning. If there’s any chance of lightning or rain within 25 miles of the Cape, you get a hold on account of weather. They predict in advance what the launch weather is going to be, and they give a percent possibility of a no-go. A lot of times you’ll go in when they predicted you’re not going to be able to launch, but every once in a while things clear up.



Ross-Nazzal: You were talking about the [Block] II turbopumps, and you were talking about the dry lubricant that they were using. Can you share some other details about the turbopump and the testing program?

Hopson: The highest temperature of any major part in the engine is the turbine blades on the high pressure fuel pump. You’re operating fairly close to the limit and they’re glowing red; they’re operating around 1,000 degrees or a little more. Those turbine blades are very high temperature super alloys that contain nickel, chromium, and cobalt.



Ross-Nazzal: The new turbopump, was that tested on the E-8 test stand?

Hopson: It was tested on the A-1 or A-2 stands at Stennis. I think they’d already closed A-3.

Ross-Nazzal: And they were subjected to the same type of test that you conducted earlier in the program, the 520 [seconds]?

Hopson: Yes. We ran the complete certification test series, plus we imposed the fleet leader requirement. We used to test regularly at all three test stands. I remember one failure we had at Canoga Park where the engine actually burned itself out of the test stand, bounced down the flame bucket, and then bounced down the canyon for a ways. An engine accident is spectacular. One thing that we used in testing was cameras that have speeds (frames per second) that are tremendous. When you have an accident you look at the frames: frame by frame. There’ll be one frame, where the engine will look perfect and the next frame you can’t see anything because of steam and smoke and fire. When they go, they usually go pretty quick. Sometime, though, the film will show the approximate location of the failure, which is very helpful in determining the cause of the failure.

Ross-Nazzal: How did the [Block] II engine operate the first time you flew it?

Hopson: Great. We wanted to fly the new pumps on all three engines.

Ross-Nazzal: Yes, I noticed you only flew one. Why was that decision made?

Hopson: It’s was a judgment call by our management.

Ross-Nazzal: A safety issue?

Hopson: Yes, that’s what it was. The SSME team didn’t think it was a safety issue, but management did. We’d have liked to have flown all three. We had tested for all flight conditions on the ground and we thought that it would be safer to fly three of the new, more robust pumps, than flying two of the old pumps which had low damage tolerance.

Ross-Nazzal: So it worked exactly like you hoped it would? No changes?

Hopson: Yes, no problem at all. That’s really been a good pump. To my way of thinking, that’s by far the most important safety feature. A very important engine safety feature is a “redline” temperature approximately 250 degrees above the normal fuel turbine outlet temperature. If exceeded the engine would shut down to preclude catastrophic failure of the engine. Pump testing of the original pump showed that the pump and engine would fail (hot gasses would be released and pieces of the pump expelled) before the redline temperature would rise enough to trigger the redline. The redesigned Block II pump was robust enough to contain internal damage until the redline was reached and would shut the engine down safely.

We had 11 pump accidents on test stands with the original pump. All 11 of them would have caused loss of vehicle and crew. The shuttle would be destroyed before it got to the redline. A lot of people thought, “Well, we have a redline to protect us, to shut that engine down.” It wouldn’t do that, but with this new pump, you can lose half the blades, and it’ll contain the damage. The old pump weighed 700 pounds and the new pump weighed 1,100 pounds. So we put an extra 400 pounds into the new pump to increase robustness.

One early problem we had in the pumps, were the bearings, because the rotational speeds of the pumps are very high. We were having trouble with the pump. Rocketdyne engineers and I went to Evendale [Ohio, headquarters of GE Aviation], and we talked to the rotating machinery people. There’s a rule of thumb that gives you an idea about how hard the requirements are going to be for the bearings. If you use the diameter of the bearing and you use the speed (rpm) of the pump, and you plot one against the other, they call it DN. If you plot DN for the SSME pumps, we were out considerably further than any bearing used in the past. GE said, “You’re not going to make those bearings work.” To make the bearings work we used ceramic bearings instead of steel. This was an important feature that let us develop the pumps.

Ross-Nazzal: This was for the original engine that flew, for the STS-1 and the earlier flights?

Hopson: It was for both of them really. Just to give you the idea, in the beginning you had all this stuff about what the engine had to do to be able to do its job. The engine had to be a real performer, far better than anything else that had ever been built before. It’s lighter to weld things than it is to bolt them together, so the engine in the beginning was almost all welded, and the pump was also.

The real sensitive part inside of a pump is the rotating part. It has a shaft, and it’s on bearings. And at the end of that shaft there’s a disk that the blades are put into. Those blades have to move a little bit during operation. You slide them in, they have to have a certain amount of movement or they’ll hang up and crack. The rotating part of the old pump included a shaft and a turbine disk which were bolted together. It was the best pump that you could design which has the lowest possible weight and could also do the job. But if you lost a half of a turbine blade you bought the farm, because the next thing you knew the rotor would be into the case because of the unbalance. You’d have an explosion.

For the new pump, we used some of the weight saved due to performance increases and minimized the welding. Also, bolted components can be dismantled for internal inspection, which is especially important with reusable components. The newer pump had three bearings instead of two. The previous pump had one bearing in each end; this one had an extra one, and it was a roller bearing. All the others were ball bearings. On this new pump we had two ball bearings and a roller bearing, and the shaft and turbine disc were all one forged piece. In other words if something happened to it, it was a lot stiffer. It could withstand unbalance better than the other one could. We put 400 extra pounds into that pump. Even if you lost half the turbine blades, the pump would contain the damage until the redline would shut down the engine safely.

Ross-Nazzal: Pretty impressive.

Hopson: Yes, it was a good pump. The first pump was really good because it let us fly. It was right on the ragged edge, but it let us fly. We went through all the safety wickets, certifying the original pump and all that, so we weren’t flying something that we thought was going to fail. Over time in our test program we found out that the original pump had very little damage tolerance. Lose a little bit of a blade or something and the pump comes apart. So, to me, the new hydrogen pump was the most important safety improvement we ever made to the SSME.

Ross-Nazzal: That’s interesting. I was reading that there was a decision to develop an advanced health management system for the SSME. Can you tell me about that system?

Hopson: In the beginning there were thoughts about safety, and pumps were a main concern. The high pressure pumps had two redlines that were considered as a basis for shutting the engine down before it came apart. One of them was the temperature redline I was telling you about. The other one was a vibration redline, which amounted to vibration sensors on the pump, and if the vibration exceeded a certain value it would shut the engine down. The advantage of this over the temperature redline is that if something happens to the pump the controller shuts the engine down instantaneously. You don’t have to wait for the temperatures to reach the redline to shut down. We flew that system but never made it active.

The reason we flew it inactive—it’s a pretty serious thing if you shut an engine down, especially right after liftoff. They’ve got what they call RTLS, return to the launch site. If you shut an engine down early, then they can’t make it to any of those other landing sites; they have to turn around and come back. That’s never been done in flight. It’s been done in a simulator. Some people say it’s a piece of cake; some people say there’s slim to none chances of making it. You have to wait until those big solids burn out before you can really have much control of the vehicle.

The advantage of the vibration safety system is that it acts so quickly. Anything that happens to a pump, you’re going to see vibration. The reason why we didn’t make this redline active in the beginning was the fact that first we wanted to fly it inactive and see if it had the potential of shutting down a good engine. Well, we found out that it had a serious problem. We found out that in the wiring, especially if you got some moisture in the wiring connectors, you’d get noise which would be interpreted by the redline system to be pump vibrations which would cause the engine to be shut down when there’s nothing wrong with it. The noise was spurious signals that could trigger a shutdown.

The AHMS [advanced health management system], the big difference between it and the first vibration shut down system was that the first system used composite vibration. In other words the old system used the whole spectrum of frequencies, so any noise was considered along with real vibration. The AHMS only considered synchronous vibration. The pump turns X number of rpm [revolutions per minute]. If you have a vibration that matches the speed that the pump is turning, then that’s synchronous. And if it’s synchronous it’s real.

Ross-Nazzal: When did you start using the advanced health system?

Hopson: They were still working on it when I was there.

Ross-Nazzal: When you left, 2004?

Hopson: Yes, but they had run it on test stand. They were about ready to incorporate it.

Ross-Nazzal: Can you tell me about that test program?

Hopson: They added the vibration sensors at what they thought were the appropriate places on the pumps for measuring the synchronous vibration, and also changed some printed circuit cards in the controller. Then they installed theAHMS on a ground test engine, so as to evaluate its’ performance and to assure that it didn’t create an engine problem elsewhere. When I left, the early test data showed no problems.

Ross-Nazzal: I read when you were project manager that the agency grounded the Shuttle fleet because they found cracks in the flow liners of the main engine. Can you talk about that issue and how that was resolved?

Hopson: The flow liner was transition sheet metal between the propellant supply, which is the external tank, and the engine. They found some cracks in it and they had a big program to deal with that. I always thought they overkilled that problem, because before the interest really focused on it, we knew about it and just welded the cracks. You could inspect for cracks after every flight and then weld the cracks in the sheet metal, the way the sheet metal other places in the pump were welded. It took several flight duration engine runs before the cracks started and the cracks were in a location which was relatively easy to inspect

Ross-Nazzal: You said it was a pretty simple fix?

Hopson: I don’t know. That thing went on so long that it seemed like people were making a career out of it.

Ross-Nazzal: Did the Columbia accident [STS-107] have any impact on the SSME Project itself?

Hopson: No, I don’t really believe it did. It was mainly aerodynamics and the propensity of the foam to come off. No, I would say that it didn’t have any significant effect on the engine.

Ross-Nazzal: In 2004 President [George W.] Bush announced a new Vision for Space Exploration. Did that have any impact on the improvements or upgrades that you were planning on instituting?

Hopson: No, not a bit.

Ross-Nazzal: You left in 2004. Were there other improvements or updates that were still in the works as you were leaving, to the engines?

Hopson: I’m not sure whether the advanced health monitoring system had been baselined or not. That’s the only thing that I know of that could have been.

Ross-Nazzal: I think I’ve exhausted my questions. Are there any other topics that we might have explored that you thought about?

Hopson: I don’t think so

Ross-Nazzal: Is this your bio [biography] and a copy of your Lessons Learned? Great. Do you mind if I keep this copy?

Hopson: No. You may keep them. The retired SSME Chief Engineer and I give a propulsion course every once in a while. There are some significant differences between the Shuttle requirements and going to Mars or the Moon. So we tried to put out something for the young guys that would not only talk about the problems that we’ve had recently, but also some that we had on Apollo that we didn’t have here. That was one of the propulsion course handouts [demonstrates].

Ross-Nazzal: That brings to mind a question. Why did NASA continue to evolve the engine after STS-1? Why spend millions of dollars to enhance the engine?

Hopson: All of the engine changes were safety improvements. Anybody that would fly STS-1 would be taking a large risk because of all the problems that we discovered and fixed later. We have a little newspaper we call the Marshall Star, and a couple years ago they had a picture of STS-1 pilot Bob [Robert L.] Crippen. He was smiling and waving as he was getting on STS-1. I cut that thing out and sent it to him with a note that said, “Crip, you must not have understood the problem.”

Ross-Nazzal: That’s interesting.

Hopson: One of the things that cost us a lot of money and a lot of time in developing the new pump was that they run a test pump and then always inspect the parts for problems. [If] they look good, they’ll say, “there was no damage to the pump.” Then later we would run a test, and there would be a failure, or something about to fail. So we would go back and pull the previously tested hardware off the shelf and look at it. We would find a crack in the same place. The inspection hadn’t found the earlier crack. This occurred on three different pump parts. We lost several months by not being able to detect cracks soon after they occurred.

It turned out that the solid lubricant, Braycote, was part of the problem. They used that very liberally and thought that there wasn’t anything it would hurt by putting a lot in versus a little. They put the pump together with Braycote, and you have engineering, you have inspection, and you have cleaning organizations. The cleaning guys said, “It wasn’t our fault. That Braycote is almost impossible to get off.” Inspection guys said, “They didn’t clean it well enough so we could find the crack.” Then the engineering people said, “Well, inspection didn’t find the crack.” What that means to me is that there was no one who felt accountable for the complete pump. These organizational charts say cleaning, inspection, engineering. Well, each one of them has a little warlord that’s over that particular group. To me, that can hinder accountability.

There’s nothing wrong with having those groups. You have to have some kind of organization chart, but there ought to be one guy—I don’t mean the captain of the ship, that if it runs on a sandbar while he’s asleep, and they demote him. He has to be accountable for any mistake that’s made aboard that ship. If you want to treat the program manager that way you can, but he really can’t know everything and be everything. So, to me, there ought to be one guy for each major component whose job is seeing that treatment is correct and good everywhere. I think that one of the big problems we had was lack of accountability. That doesn’t just apply to the pump that applies to everything. “Wasn’t my fault, he was supposed to so-and-so.”

In retrospect, every major engine component should be assigned to someone who felt totally accountable for that component. The engine nozzle was another component where lack of accountability cost us dearly. As SSME manager I regret not having recognized this fault in our program. Some might think the accountable person is redundant and not required. This might be true with some parts, but rocket engine parts deserve special treatment because they usually use high technology design, precision manufacturing, and are very expensive.

Ross-Nazzal: The new pump, did it benefit you in terms of overhauling the engines, maintaining the engines, and improving what you might call accountability?

Hopson: It should. It was a better pump, there should have been less things done to it to make it ready to fly again. But I can’t really say with certainty. The accountability problem had not been recognized and dealt with. Hopefully, lessons learned, such as contained in this document will help to avoid similar problems in the future.

Ross-Nazzal: How many times could the new turbopump fly versus the old pump?

Hopson: I don’t know. The fleet leader program was not going long enough to set lifetimes of parts. The certification testing was used as a basis for the first few flights.

Ross-Nazzal: Anything else to add any parting words about the main engines?

Hopson: I don’t think so.

Ross-Nazzal: Well, I think we’ve covered everything pretty well, so I thank you very much for your time.

Hopson: If you have any questions that come up, I’ll be glad to find the answers for you.

Ross-Nazzal: We certainly appreciate that. Thank you again.

[End of interview]