NASA STS Recordation Oral History Project
Edited Oral History Transcript

Robert J. Schwinghamer
Interviewed by Rebecca Wright
Huntsville, Alabama – 20 July 2010

Wright: Today is July 20th, 2010. This interview is being conducted with Robert J. Schwinghamer in Huntsville, Alabama as part of the NASA STS [Space Transportation System] Recordation Oral History Project. Interviewer is Rebecca Wright. Thanks so much for coming in this afternoon and offering all the information for the history. You’ve been involved in aerospace history for more than 50 years, and you were here when they were talking about some of the concepts. One of them that we talked about was the metallic reradiating heat shield.

Schwinghamer: In the early days of [Space] Shuttle, we looked at the prospect of using some metal plates applied like shingles overlapped on a roof. We had a columbium alloy that was good up to about 3,500 [degrees] Fahrenheit, but it eroded badly during testing so we had to come up with a coating to protect the columbium. We did develop a ceramic coating, but in order to apply it to these plates we had to go to Fansteel Company [Fansteel Inc.] in Canada. That was a disadvantage, but they did the coating of those shingles.

The advantage of a metallic reradiating shield certainly would be you could fly through rain—it was much more robust than the nonmetallic tiles [and was] not very subject to damage in any way. It would have required a whole lot less maintenance had [one] done it this way. There were some significant disadvantages that developed. Because it was so much heavier than tile, we’d have had to beef up the structure. We needed some more thrust from the engines because it was heavier. Then, not to be considered lightly, we got a lot of complaints from [NASA] Headquarters [Washington, DC] about depending on [foreign] help [to] build the Space Shuttle. I think the extra weight and the [foreign] dependence killed the metallic reradiating shingles heat shield.

You wanted to talk a little bit about thermal protection material for the solid rocket boosters. In the 1970s the normal [thermal] protection material used was cork, [relatively] expensive ablation [material]. The material was laboriously applied by hand; it was hand labor. We knew this would never meet the Shuttle schedule, at least as anticipated at that time, so we developed an excellent thermal protection material in the lab [laboratory] for use on the solid rocket boosters. We then learned how to spray it instead of putting it on laboriously by hand, and we used some just developed robotics to do the job and we got wonderful consistency and super adherence.

Then the design guys said, “But we have to take it off for reuse.”

So I said, “What do you want? Make up your mind. You want it on or you want it off?”

They said, “Both.” [Well] we had to come up with a way to take it off, and it was really sticking on there good. Luckily right about that time we had something in the lab called a hydrolaser, which is a very high pressure water jet. We had bought one, so we roboticized it, and it took the thermal protection material off like gangbusters, no problem. Because it was automated, we were able to use precise control both during the application and the removal of the thermal protection material. That really worked out great.

A little bit on the spray-on foam for the external tank. I’m going to let you in on a little episode here. I wouldn’t have done this 20 years ago but I’ll tell you today. When we looked at the external tank we had the same huge acreage problem as we had in the solid rocket boosters. We had to cover acres of the thing with the foam [applied] with very close control. We had completely characterized about three of the most promising materials in house at the [NASA] Marshall [Space Flight] Center [Huntsville (MSFC)] in the materials lab. We did that for the [candidate] materials we could find. They were three pretty good [ones], but we picked one.

Well, from an obscure technical paper I found out that the foam we selected was highly toxic when combusted—so said the paper. I thought maybe the [author] was a hot dog, I didn’t know. We had already given the go-ahead to the contractor on that foam so it was a serious thing to consider changing at that point. Because I didn’t totally buy the paper’s warning we got some white mice and we gave them a foam combustion products breathing test. The mice kicked the bucket in an alarmingly short time so we switched foams in midstream and by now we were robotics experts, [so] we devised an automated system to spray the large areas very successfully. After this though, we always did extensive toxicity testing on the foam we received for use on the Shuttle.

I remember when we did the first one down at the Michoud plant [Michoud Assembly Facility] in New Orleans [Louisiana]. I was standing there with my boss and he looked at it and he said, “Bob, that thing looks like it was done by a drunken plasterer.” I said, “We can fix that, doc, it’s just the software.” And that’s what it was. We fixed the software and the second one was beautiful.

On the first Shuttle flight we had some foam coming off of the tank as it sat on the launch pad. I took a crew to KSC, the [NASA] Kennedy Space Center [Florida], and after days and nights we found the trouble. There had been inadequate activator application. It wasn’t uniform enough over the surface. That brings us to the subject of foam bonding on the external tank. That foam incident for the first Shuttle flight was disturbing to me because we learned for the first time—and we were material types, [and] should have [guessed] this—there was [really] no adequate quantitative method of determining cleanliness of the tank before spraying it.

They had that old-time water test where you poured water on it and if the water globuled up it was dirty and if it didn’t globule up it was clean. But that was sure not quantitative; that was a qualitative test so it wasn’t good enough. We joined with a small instrument company that we liked very much, and we decided we were going to develop a quantitative diagnostic device that could tell us the cleanliness level before bonding or spraying the hardware. We called the device the optically stimulated electron emission [OSEE] instrument. We put [the scanner] on a robotic system and we scanned over the surfaces to be bonded, or sprayed, at fairly high speed and it did a great job. You could [quantitively] tell if it was clean.

This was also of great significance to the guys building the solid rocket motor. The manufacturers there had concerns always about safety and reliability, because it’s extremely dependent upon superior bonding of the propellant to-liner-to insulation internally in the rocket. If that stuff comes loose you’re going to have a bad day. As a result by using this OSEE, there was a great improvement in reliability and safety for the solid motors themselves. This wonderful little device was warmly received, and it’s now widely used throughout the aerospace industry.

Our squabble with the Air Force. For a while we flew some Air Force payloads in the Shuttle payload bay, and they were called IUS, it was the inertial upper stage. Well, when we examined the design of the IUS proposed to be flown aboard the orbiter, we noted that the critical payload bay IUS support structure used a material that we knew had a critical flaw size so small there was no means of detecting it before flight. That means we could have a flaw that would fail in flight because we couldn’t [find] it.

We immediately insisted that until we could do better than that, every time the Air Force flew an IUS structure there had to be a proof test before flight to make sure it wouldn’t break in flight. There was still [an exceedingly] small risk that [after proof testing, the flaw could possibly] grow to critical dimensions before you got it flown, but that was a small risk. Since one couldn’t detect the flaw that could fail in flight, this was just too big a risk for NASA to take [without a proof test]. That made the United States Air Force a bunch of unhappy campers but we stuck to our guns.

It happens that in the lab at that time we had a fully characterized new alloy. We developed [and characterized] alloys in the materials lab at that time, and this one was called Custom 455. It was an excellent substitute material, but turned out that nothing the size of that IUS support structure had ever been forged before. They couldn’t forge anything that big so we worked with a contractor to build such a large forging. It was accomplished and it became the material of use for the IUS in the payload bay, so it worked out pretty [well]. This proved completely satisfactory and improved safety without a proof test before flight. That brought peace in the family.

Wright: It took a few years ago to get that, right? Did this take you a while to determine this?

Schwinghamer: Yes. At least months, approaching a year. In the meantime they still had to proof to fly, but after that no more proof testing. We were comfortable and they were too, so that was an internal peace between organizations.

One of the really difficult challenges was to weld the big external tank. We had developed a technique called variable polarity plasma arc welding. The external tank at that time was made from 2219 aluminum alloy, and it was a fairly new development back then. Not much was known about its weldability, especially in such large structures. Nobody ever made structures [of that size] with that alloy. At first the welding process that we selected just couldn’t cut the mustard, it wouldn’t do the job, but we did several things.

We added additional diagnostics to better control the welding arc, we beefed up some of the [welding machine] components because they actually failed during the extremely long run times—it took us three and a half hours continuously running the machine to make a circumferential weld around this tank. The guys making the welding machines hadn’t counted on that. They were talking about 30-minute duty cycles or something. So we went in the welding machines and altered them ourselves to make them able to run three and a half hours without stopping. Then we used some design-of-experiments smarts to optimize the weld current wave shape. When we did those things we finally had a process that worked great.

We were able to perfect the welding at the Marshall Center because we had a large tower and we tested welding techniques on full-scale hardware in that tower, so we could handle the 33-foot diameter tanks when we tested. But even at that, at one point we couldn’t weld at all. We were down three weeks one time, we could not make a weld. I didn’t get too much involved at first and then it got worse and worse so I went over there myself and we examined that thing and we found that the cable that took the [welding] current from the weld back to the [welding] machine was continuously moving as the tank was rotated to weld. So we designed a fitting to allow that cable to stay in a fixed position at the center, at the bottom and it never moved and that solved a nasty problem. No trouble after that.

Wright: Something simple, but what a difference it made.

Schwinghamer: Simple, but you had to understand the physics involved.

I got much involved in the decision of how we would prevent the components in the solid rocket booster from being destroyed on recovery. The parachutes of course brought it down but [it was] still moving pretty fast, and every time that thing splashed in the ocean it buggered up [a lot of components] in the aft skirt. So we investigated several foams and we finally found one that was very good at cushioning and we sprayed the components with this cushioning foam and after that we could reuse the hardware.

In the area of the engines, the turbine blades and bearings were problems at the start. Serious problems. I think what caused the problems were the enormous temperatures and pressures that [the] engine [had to sustain]. Other engines were not running at those temperatures and pressures but to get [both] the kind of specific impulse we [needed] and low weight [too], that’s how the design ended up. So now it’s up to the materials guys. “How come you can’t get me something that can stand these temperatures and pressures?”

With the turbine blades it became immediately apparent we had to do something there, and I started an in-house, in the materials lab, metallic materials division, a program to develop a suitable turbine blade material. We had one which looked promising, and then we tweaked that thing over a period of many months. Tweaked and tested, and we finally tweaked it into the kind of durability and usability that we expected. We did that pretty much in house. In the bearings, they were another special problem. Suffice to say, again, we had something going in house that proved to be the ultimate solution. Materials got so important that management decided that the materials people had to sign off on all the MSFC drawings for Shuttle hardware, so they never got issued until the materials people signed off.

I guess you could say I was a prime mover in the decision to go to a better aluminum alloy for the external tank. We called that one the super lightweight tank because it used a new alloy, 2195. That was a very successful program. We had some problems, but it was finally successful and saved about 7,000 pounds of payload by switching to that new alloy. Along the way we also developed and applied some hydrogen environment embrittlement data and stress corrosion cracking data, and we made two major specifications in those areas. They’re [still] used by the aerospace companies.

In 1990 I headed a leak team for NASA when the Shuttle fleet was grounded. That was a tough problem and lasted several months. I [also] headed a NASA failure investigating team on the big casting pit fire and the mix pit fire we had at a contractor in Utah. Huge fires, big damages. We found out what they were however, and we used a fault tree to do that. I was [a dedicated] user of the fault tree.

I was also the solid rocket motor contingency team leader [of the] Marshall Center. When the Shuttle came along, [MSFC] named contingency teams and contingency team leaders for each of the major elements; the engine, the tank, the solid rocket motors, payloads, etcetera. [Those team leaders] were committed to take action immediately, [one] didn’t have to call and ask, “Boss, should I go?” When anything happened the team leader convened the contingency team. I was the solid rocket motor contingency team leader, and so I did what we were supposed to do and I set up the fault tree and we worked four months on the [Space Shuttle] Challenger [STS 51-L] accident investigation.

Back in the Saturn [rocket] days, [the] Apollo program had a definitive components development and test program, that ran right along with the major program. In fact it had to precede it a little bit because what came out of the components development and test program was what you then put in your [final] design. Sometimes we had as many as three contractors doing in-house work. It’d be ongoing at one time, and we’d pick the best of the three. [One] really had a pedigree in the system when that [was done]. That [meant] that the final selection had a significant development and test pedigree, and when the whole system was complete and put together, all the elements played together, largely because of that pedigree. I think it was very beneficial.

Of course the key to that kind of an approach just [depends on] funding. [One] cannot do that without [adequate] funding. I guess in Shuttle we never really had the funding that Apollo had. It never was there, but [on the other hand] there were some advantages that the Shuttle had in hand. We learned for instance a lot from Apollo, and that proved very useful for Shuttle so it just carried on into [that] program. At the same time, new analytical methods were being devised in areas like computational fluid dynamics, we had new stress analyses and improved materials and processes. Even so I remember at one point we finally decided we really did need a fracture control program. We were trying to get by without it but we needed it after all. We got a late start on that one but we caught up.

Early-on the turbine blades were already a problem, so we began, as I mentioned, an in-house project on the turbine blade materials and we tweaked the alloy several times over a period of [many] months to get the durability we needed for Shuttle. We finally made it and we passed that along then to the contractors. As I indicated earlier, it was the very high temperatures and pressures that drove us to these extraordinary efforts. [One] just couldn’t take something off the shelf, no alloy off the shelf—it just wouldn’t hack it. [One] had to tweak it and make it fit. We ran many tests until we finally hit pay dirt. Then of course, as I indicated, we passed that along to the contractor.

Turbopump bearings were an even bigger problem. Sometimes we had to change out bearings after only one full duration run on the test stand, and we burned up several engines. Changing after one full duration run is like your car is good to drive to Birmingham [Alabama], but that’s as far as it goes. It just wasn’t acceptable. In order to test the bearings we had to use whole engines at the [NASA] Stennis [Space Flight Center, Mississippi], and that was dangerous and expensive because here you have something in very much a development stage, and now you’re risking a full all-up engine on that test. So we built a bearing tester. It was just a simple system that sat aside and didn’t have to have the whole engine attached to run the test, and that speeded everything up. We would run tests in a couple days and change the material and put them in and run them again. Boy, it saved a lot of money and certainly expedited the solution of the problem.

When we got that thing going, we did find immediately that the bearings as they were in the design were undercooled. They were running too hot, they needed cooling. Well, we supplied additional cooling, and it got better, but not good enough. Fortunately, we did a lot of anticipating in those days, “What might we get into?” [We] had a lot of experimental stuff going in the lab to head off possible problems. In this case we had anticipated this to some extent and we had a lab project going on bearing balls made of silicon nitride, a ceramic. They were superb.

There was no interest at all from the United States bearings makers, none whatsoever about the silicon nitride. I couldn’t get them interested to provide what I needed, so we had to get the best silicon nitride we could find, and it came from Japan. Then we sent that silicon nitride material to Germany, to of all places Schweinfurt. [Now you may not be] old enough to remember Schweinfurt in World War II, but Schweinfurt was where the Germans made all their bearings. We [the US Army Air Force] bombed Schweinfurt and that ended the war months earlier because they had no more ball bearings. I thought [our bearing arrangement was kind of] ironic. Anyway, the first user of silicon nitride in cryomachinery was the Space Shuttle main engine. That was a first. Nobody in the world had ever done that before.

I put my reputation on the line in the beginning to sell the silicon nitride. I was sold on it. We had test data out the kazoo; it looked good. I went to the Center Director and he said, “Oh you [surely wouldn’t] put glass balls in these turbines.” I said, “It’s not glass.” We took a piece of steel, put a silicon nitride ball [on it], and hit it with a steel hammer. Nothing. That convinced him. The stuff worked so well that they’re now used widely in high performance aircraft. In fact one aircraft company put those silicon nitride balls in their bearings, they flew the airplane, they shut the coolant flow off to the bearings with the plane in flight, and [the bearings] were still good enough at high temperatures [so that the pilot could] land the airplane without incident. That’s how good they were.

It’s still a little tough for me to talk about Challenger. I mentioned we had contingency teams for each of the elements on the Shuttle. I had the contingency team for the solid rocket motor, so [setting up the fault tree] fell in my lap when we had Challenger. We went into action immediately, which we were allowed to do. We set up the construction of the fault tree to guide our part of the investigation, but that investigation, the whole thing, I still have bad thoughts about it. It was especially disheartening because of the loss of the astronauts. [None of my previous] problems were of that nature. It was an exhaustive grueling investigation. It started in January [1986] for us and it lasted until June, day and night.

At least half of my team members came down with flu, colds, depression. I learned you can’t go four months day and night without suffering the consequences. I believe our efficiency must have dropped to about 40 percent after four months or so but we just kept plodding on. During one of the many 10:00 evening dinners I was having [at home] while that was going on, my mother, who then lived with us, promised to make two big pans of rolls [every weekend] for the team [as a morale lifter]. And they were big pans. She did until one weekend she got sick. I’ll never forget, as I came to work that Saturday I was personally roundly, castigated. Heads popped out of doors and shouted, “Where are the rolls, Schwinghamer?” It was a morale booster to say the least.

Wright: Something to look forward to.

Schwinghamer: I mentioned earlier about the hydrogen leak grounding the Shuttle fleet in ’90. After an unsuccessful tanking test and three aborted attempts to launch the [Space Shuttle] Columbia [OV-102] in ’90 the entire Shuttle fleet was grounded. Since we didn’t know what it was or where it was, we couldn’t risk flying any of the other Shuttles on the premise that only the Columbia has it, the others don’t. So the fleet was grounded, which was a good decision.

The NASA director for Space Shuttle named me as a team leader, and I picked up the investigation team myself. I named the team members. J. R. [John R.] Thompson, who was [NASA] Deputy Administrator at that time, sent me to Kennedy Space Center with a one-way ticket. He says, “You’ll get a ticket back when you solve the problem.” That was a little bit of an incentive. We were down there from September to December. We set up a fault tree immediately, with instructions to everybody in the system there that there would be no confirmatory tanking test until the complete gamut had been run on the fault tree master plan. Project managers, they get antsy. They can’t wait, they want the answer, and I kept saying, “No, no, no, not until the fault tree gamut has been run.”

This was a big operation. It involved three NASA Centers and many many contractors. After about six weeks the fault tree was complete, the tanking test was successful, the major leak was found, and even other leaks were found because we used the fault tree. Those were repaired, and Columbia was the soundest leak-free orbiter at that time in the fleet. We [all developed a special affinity for] Columbia [due to] the time we spent down there with our team. The eventual loss of Columbia [STS-107 accident] was really a blow because we had become [so] intimately acquainted with that bird during the leak investigation. That was the end of the kind of things I was used to doing.

Wright: I have a couple questions, I made notes as you were going through. You said you went out to Utah, about the fires. Can you share more information about what you went out to investigate with those fires and problems?

Schwinghamer: It was early in the Shuttle Program; it was late ’70s or early ’80s. I’d just have to go pick a team and go and we’d go and stay there till we found [the cause of the failure—the smoking gun]. Most of those things took several weeks typically. You’re away from home that long, and I think what I learned about that was [that one] can’t just go in with your investigating team and [dominate the investigation]. I made sure when we set up the fault tree, at every block or element in the fault tree the contractor [also] supplied who he wanted to be in that block. It ends up being a team effort when you do it that way.

If [one] just sends an investigating team into an existing organization and trys to work it that way it’s very difficult. I learned early-on that every block on that fault tree [needed a person’s] name, and it had two names, it had a NASA guy and a contractor guy. That encouraged their cooperation, also contributed knowledge that you wouldn’t have been able to dig out, in every instance on every subject, from the contractor. That’s why [the] fault trees were successful I think.

Wright: I guess you start in a way with objectivity because of the way that the tree is laid out, so that you have to supply facts?

Schwinghamer: Yes. The top block is always: this is what happened. [It must be comprehensive]—this might have been the problem, [or that] might have been the problem, or this other thing—all conceivable things that might have influence. Then there were subheads that would be further broken down. You’d name an expert—if it was structures you’d pick the smartest structures guy you had and so would the contractor, then they’d wade down through, “Well, could it have been stress relief, could it have been fatigue?” All those things, each one of those questions get either exonerated or indicted. “Oh, couldn’t have been that.”

“Prove to me it couldn’t have been that.”

“Here’s my test report and here’s the test data.”

“Okay, wasn’t that.” [One finally] gets down [near the end] and here’s the one that’s indicted. “Well, how do you know that that’s the case?”

“We did an analysis and it looked bad, we ran a confirmatory test and the test showed it was bad.” [One] just [has to] wade down through the whole system. If they’d been able to do this with the oil leak in the Gulf [of Mexico, BP 2010 oil spill], they’d have [gotten to the culprit earlier].

Wright: I believe when you were doing the investigation of the solid rocket motor that some of what you discovered was that they didn’t need to change the materials.

Schwinghamer: I think you’re talking about the O-ring situation on the solid rocket booster failure. We had a material that we had selected because it was the toughest O-ring material we could find, which meant that when you put the segments together you were not very likely to damage the O-ring, and it had extremely high temperature capabilities. In our opinion, after months of testing of many materials, silicones, all kinds of rubbers, we finally decided that this one material [Viton] was far superior.

After the Challenger failure the O-ring was involved. Really wasn’t the O-ring’s fault, it was a deficient mechanical design, but because the O-ring was involved there was a hue and a cry, “Let’s get rid of that O-ring.” I kept saying, “You can’t find a better one. There is not a better one. I’ll show you the data. I’ll show you the rigidity, the toughness, the temperature capability. You can’t get a better O-ring material.” I finally prevailed, and they finally listened to reason.

But you know how there’s [always] an overreaction when there’s an accident, “Let’s do this, don’t want to see any of that anymore.” There was a pretty strong contingent at very high level in NASA, and I just kept lying down in the flame bucket and saying, “No, no, no, don’t change this O-ring material.” That struggle went on for several months too. Early-on the people at the Marshall Center supported me. I had very good luck with that, they always supported me in whatever argument we got into. The Center Directors and all the rest of management were fine. [But it] went on and on, and then finally—it was even suggested by the NASA Administrator that we change the O-ring material. That’s kind of impelling when you get that kind of suggestion, but I kept saying, “No, no, no, don’t do this.”

I think also it became more evident as we analyzed our design. We redesigned that whole thing, and we had a test facility where we could test those O-rings during a hot firing at Marshall. After we saw that the clevis joint was mechanically [deficient], then the heat came off a little bit on changing the O-ring. Initially everybody thought it was a lousy O-ring. There were some people too willing to want to bet on that. But had we only changed that and not changed the clevis design we’d have had trouble again later, although I doubt if we’d ever launch at 27° [degrees] Fahrenheit anymore after that.

Wright: You also mentioned something about—I think you called it a fracture control problem. Can you talk more about those?

Schwinghamer: Yes. Fracture mechanics is a method of examining your materials, in their design, to see whether they will meet the endurance requirements—will they fail before [reaching the design life] or not? In order to do that you’ve got to get off to the side and get extensive materials data [so one can] plug these material data into these fracture mechanics calculations to determine whether fatigue [life] is going to be adequate. It’s a [comprehensive] method, and you have to gather the data through the whole system both for the materials and for the structures. That’s time-consuming, it’s expensive. We had it in Apollo and we decided at first not to have it in Shuttle.

Then as we went on, from a materials perspective it became more and more apparent we were stretching the limits in some [situations]. “How far could [one] go?” Well, fracture control program would tell [one] how far [one] could go. [At first] we weren’t going to do it and then it wasn’t too long before it became evident this is not [just] a nice thing to have, it’s a necessary thing to do. So we did that, we had a fracture control program for the Space Shuttle.

Wright: Part of what you had mentioned was the need for automation of processes. How were you able to, during the process of trying to meet schedules, find better and newer ways to automate and structure those processes where they were superior?

Schwinghamer: I’ll tell you what the secret is. Anticipation. [One has] to seriously sit down and consider what [will be necessary]. What your structure should do, what [one’s] materials should stand. Then, at that moment, [one does] the best [one] can with what [one has], whatever is in the system. Then [one] looks at it and says, “The margins here are less than I’d like to see,” so [one] starts a program off to the side to improve that situation. If nothing happens you can elect either to leave it as it is, good enough, or you can still change it and get a higher factor of safety. [One] can make a choice. A lot of it has to do with scrutiny and anticipation.

If [one delays] it falls on [one] like a ton of bricks, it [may be] pretty late to start a program to improve that situation. You will start a program but you’re late, you’re a day late and a dollar short. I always prided the Marshall Space Flight Center—and all the [center] elements [anticipated] that. We urged people to use ingenuity and think outside of the box. “If so-and-so happens what am I going to do? What’s the backup position?” Of course funding constraints put a damper on that, because there’s a limit to how far you can go outside the box if you don’t have the money to do it. I attribute that ability at the Marshall Center to everybody thinking out of the box and anticipating what might happen. Sometimes it did, and sometimes we headed it off. I know we headed it off on bearings and turbine blades.

I knew the bearings were going to be a problem from day one. If we hadn’t had that ongoing program in the lab and brought the silicon nitride along at the same time it wouldn’t have been there. And not everybody can do that. It takes ingenuity and most people can’t or don’t want to think outside the box. It’s uncomfortable. When [one] finds people like that in the organization [one] cultures those, you treat them nice, because not everybody can do it, but we had a bunch of people who knew how.

Wright: You were able to start your materials lab. Are these the type of people that you pulled to be in that, your system of finding those answers?

Schwinghamer: [Let me say that I was very fortunate in that my two predecessors had already created a disciplined, capable and smooth-running organization when I took over.] And yes, we had a great esprit de corps. Part of it comes from being under the heat of a requirement. That makes you all pull together. But I always went out of my way to cultivate the mavericks. I got a lot of good stuff out of mavericks. Sometimes [people would] tell me, “What do you want to hire that guy for? What do you want to transfer him over to you for? Are you kidding?” But they had [special] abilities, and [one has] to treat them accordingly. [One] can’t treat everybody the same. Some people are satisfied to say, “Go fix it, Joe,” [Joe’s] tickled to death with that. Other people have to be coddled a little bit—they got to be stroked. [One has] to recognize when [one has] people like that. Then the other thing is, I was always a walk-around manager. I didn’t stay in the office very long. At 4:00 I went to the office and then I stayed till 6:00 or 6:30, but during the day I’m walking around. “How’s it going here? What does that tensile test look like on that material we’re testing?” Walk around.

I’m really proud of one thing. I was in the materials lab for 20 years before I went to Associate Director’s job, and I never had one union complaint out of anybody. [There were always some complaints, and the legal group had one department that spent a lot of time arbitrating] those complaints. [But] I didn’t have one in 20 years. I think it’s because you have to recognize people are different. And [one has] to be honest, upfront.

I thought the worst thing that ever came along was the leaving [of important interpersonal] messages on the telephone. [One has] to talk to people [eye to eye]. It’s especially so if it’s a sensitive interpersonal relations thing, like “I wish you’d stop doing so-and-so.” I always tried to tell them why I thought [the action taken] was necessary, and then, “[Or] do you disagree with that?” and then we’d discuss it. But it’s so easy to pick up the phone and leave them a message and say, “Joe, I don’t like what you’re doing, [do better].” No good. It’s one on one, that’s how [one does] that. We really had it [going that way at the center].

That was something that was always prevalent at the Marshall Center. [Wernher] von Braun was a hands-on walk-around guy, and he talked to [one]. He did a lot of that. It was a team from day one and it stayed that way all the way through Shuttle. I [think] it’s still going.

Wright: I would have to think that with your background there were a lot of important priority issues all going on at the same time. Was there one that was more prevalent? Or was there one that sticks out in your brain as more time-consuming, maybe just more important for you to resolve?

Schwinghamer: The whole thing was a kind of a problem du jour. Whatever was hottest is what we did. I must confess the one that was potentially the most damaging to [my career] was that leak investigation. What I haven’t mentioned [before] is that shortly before the hydrogen leaks were discovered [in Columbia], they had moved an orbiter and some of the people responsible for working in the aft, where the engines were, had neglected to do something with a beam and it came loose, and rattled around and did damage in the engine compartment.

I [was aware of] that, and [I knew] they got a seven-day unpaid vacation. Here we are a couple weeks later, I’m down here with a tiger team trying to find out why we can’t find this leak. I lay awake nights in the motel thinking suppose there’s a hardnose in there that says, “I’ll fix you for that seven weeks’ [un]paid vacation,” and he takes his wrench and he cracks a valve and lets it leak at the right time and shuts it off so you can’t find the leak afterwards. I thought, “Oh my, it could be sabotage, we’ll never find it if it’s that.” I spent days out at the pad with those guys working in the aft, and I crawled around in the aft with them, on Saturdays and Sundays I was out there with them. I finally just intuitively decided that these people’s heart was in the right place. That is not what is happening.

Of course I had the fault tree going down all the blocks at the same time, but had it been a sabotage the fault tree wouldn’t find it. They were all contrite and they didn’t have any hard feelings about their seven-day unpaid vacation. It took a while for me to be with them and get [with] them on a first name basis. They’d come up to me and we’d eat lunch together. [One] learns a lot, a lot you don’t know otherwise when you eat lunch with them. That was the one big concern. I thought, “Oh my, my career is dead.” I had a pretty well established career for a troubleshooter by then. I thought, “Oh my, if this has been sabotaged I’m dead.” But it wasn’t, fortunately.

Wright: You mentioned you’d been to the Cape [Canaveral, Florida] and that you’d gone to Utah to do some investigations there. Did you find yourself at the Michoud facility or any other places? Did you go down to Stennis for any of the tests that were there as well during the time that you worked in the Shuttle program?

Schwinghamer: For instance at Utah when we had those terrible fires, the first day I got there I said, “Okay, I need the name of everybody that was working when this occurred.” Then I took the time to take each individual that was working the day the incident occurred. I took them in a room privately just one on one—no overseers, no nothing. I talked to them, and tried to find out what they might know about the incident because when we made the fault tree you had a whole bunch of possibilities that either had to be exonerated or indicted, and I thought if I talk to the people I may get enough information that some of those can automatically be removed [as potential causes]. They [might] remove my concerns therefore.

I’ll never forget. [I] got this one guy in there, and he came in and he sat down, and he was obviously nervous to begin with. He was really nervous. I started to question him, I said, “Just tell me. I’m not looking for an opportunity to blame somebody here. Please tell me what you saw that day, the day it happened.” He said, “I was afraid to say this to anybody. As I was working there I could see that this cart that was carrying propellant was running on tracks and every time this wheel came around on that track it made sparks.”

I said, “Were the sparks falling into the pit with the propellant?”

He said, “Yes.” There it was. I didn’t have to run the fault tree very far after that. If I hadn’t talked to him, hard telling how many hours we would have spent. I guess eventually we would have [come to the proper conclusion] after two or three more months. But he said, “Yes.” And he cried, I felt so sorry for him.

I patted him on the back until he moved on and said, “This is between you and me. Don’t worry about it, nobody’s going to come down on you for this. I’ll see to it that they don’t.” Then I talked to [the company] management, I said, “Come on. This guy saved us weeks and weeks. Leave him alone.” It wasn’t his fault anyway. It was the design of the system that allowed this propellant to come out of the hopper, fall on a steel rail, and then comes the steel wheel—that wasn’t this guy’s fault. He could have been criticized a little bit for not coming forward earlier. He was afraid, he was afraid they’d blame him.

I really had [invigorating] times on those failure investigations, typically.

Wright: You were able to learn and investigate and probe all different types of areas within the Shuttle.

Schwinghamer: One of the guys at the Aerospace Safety Advisory Committee in Washington said, “Schwinghamer, you’re a technical gadfly.” I said, “Okay. I don’t mind that a bit.”

Wright: One of the other things that you mentioned was computer-aided design, computer-aided manufacturing techniques in house. Can you talk the impact of that?

Schwinghamer: I had both [the] materials and processes lab. The guys in processes lab did their best to keep up with the latest state-of-the-art in things like automated controls, numerically controlled machinery, high speed machining, things that were just new and developing—and new welding techniques. We had the money to spend to do that. We did processes research and materials research. Very frequently then because of that capability we would find things that fit very well.

Sometimes it allowed us to head off a problem. Many times it helped us to improve the process. We also became very knowledgeable about two things that are awfully important if you’re going to manufacture something. We did process sensitivity analysis. That’s very important, process sensitivity analysis. [One] determines the limits within which your process can operate and the limits so that you don’t get outside the processing limits. [One does] process sensitivity analysis that tells you where the limits on the process are, then [one does] statistical process control. [One] sets controls based on what [one] knows those limits are and [one] sets those controls, and data is extracted [continuously] where these various operations are going on at these various machines, and it’s looked at statistically to guarantee [the process] stays within those limits.

We got into that early, we did that here. We insisted that all our contractors do statistical process control and process sensitivity analysis. Those are two kingpins if you’re going to build anything. [One must] do that. Otherwise, if [one doesn’t] do the process sensitivity analysis, [the process will] be making widgets and all of a sudden the [lousy] widgets, they don’t work anymore. Why? Because [the process] got outside the limits.

The parts themselves [one was] putting together had variability, and sometimes that variability stacked up against [one]. Most of the time it’s random but sometimes it stacks up against [one]. By doing statistical analysis and setting statistical analysis on the machines that are making the project, [one] can tell whether [the process is] in or out. [One] can tell [the process] made a bad one even before you test it. The process guys got very fluent at that. Those were the two things that we hammered with every contractor, process sensitivity analysis and then [use] that data to do statistical process control.

Now, they don’t like that. That’s like Big Brother looking over [one’s] shoulder while doing [the] job, but all the successful companies today use that technique. The only thing wrong with it is it’s expensive. When [one] starts doing the process sensitivity analysis, that comes way before [the process] starts cutting chips and making parts, and that takes time. [One does] things like design-of-experiments, which are statistical methods to determine which of these variables are the important variable. [Finally one has the important variables under control.] It all plays together. It’s all time-consuming, it’s all expensive but when [one does] that what [one] builds will be a quality product.

Wright: What do you think was the biggest challenge that you had to face during the time you were working on the Shuttle program?

Schwinghamer: It’s hard to say. I told you before, it’s the problem du jour, whatever [it is] that [day]. The leaks stand out, bearings [too]. It just all runs together for me, and nothing really irritated me to the point that every day it bothered me. Never had anything like that.

Wright: Did you spend much time at Michoud with the external tank?

Schwinghamer: Yes, especially on the external tank welding and then the implementation of the new 2195 alloy. I spent a lot of time at Michoud, and also had two really good engineers that saved that program. He called me one time. I’ll never forget, he says, “You got to come down.”

I said, “My goodness, it’s two days before Christmas.”

“You got to come.” So I went down there. He said, “I wanted to show you something.” They had welded up the first bulkhead and we ducked under it and looked up there and here was the weld and I could see daylight for 12 inches. He said, “We’re in trouble.” I said, “Yeah, we are.” So there went the Christmas vacation. But like I said, it was really the problem du jour.

Wright: Had to identify it and solve it.

Schwinghamer: Yes, identify it and solve it. Fault tree if necessary. I always told the [project] guys about fault trees. They’d say, “It [delays] the project [schedule]. I don’t want to do that.” I said, “Look, if [one is] driving [one’s] car and the wheel comes off [everyone] knows why you came to a stop. [One doesn’t] need a fault tree for that. If there’s a subtlety going on inside [one’s] engine [one doesn’t] understand, [one] needs a fault tree.” I tried to give [project managers an] analogy why [a fault tree] was necessary, because they are time-consuming.

The worst of it is, I kept saying, “I [must] have stress analysis X.”

Project manager says, “I can’t afford to let him work on your fault tree for two months.”

Then I’d have to go to higher management and say, “I can’t do the fault tree if I don’t get the right people.” The fault tree is only as good as the people that populate it, the people that are doing it. But if you pick the right people, you’ll get the right answer.

Wright: One of the statements that I read that you had mentioned was the reusability element plus durability impacted a whole different way you put the Shuttle together based on how you did the Apollo. Did that keep coming up during all the factors when you were solving these problems, the fact that it had to be durable and it had to be reusable? Did that continue on through the program?

Schwinghamer: Yes, there was really a hard drive on reusability and durability because before [Shuttle], we threw [our stages of the rockets] away. So [reusability] was an ever present consideration. There wasn’t a day [gone by that reusability wasn’t considered]—and of course [this aspect] manifested itself in having to have materials that could be used longer, could be used at higher temperatures, that had to have higher strength without sacrificing the toughness. Usually when you get strength in a material you sacrifice toughness, but in the case of the engine we had to develop things that had both the toughness and the strength. And durability and reusability, it was just there every day. [One] knew [one] had to do better because we had to reuse it. It was a very evident aspect of the whole problem, and it drove a lot of tests and studies that were required because of that aspect, reusability.

Wright: Originally you were going to make 400 tanks. In your mind did you believe that was an achievable requirement based on that plus how you were designing the rest of the components?

Schwinghamer: Well, 20/20 hindsight is better on Monday morning. We were all young guys, we thought we could do anything. And we pretty much did everything, except for a few things, few you don’t want to remember. But at that age nothing’s too tough. Everybody was all about that same age. I couldn’t do that anymore, I’m 82 now. Man, I couldn’t do it.

Wright: You mentioned one time that flying the Shuttle is very scientific. Could you expand on that, why you feel it’s scientific?

Schwinghamer: After having had some insight into the amount of intelligence required and even transcending engineering into science, it almost became a science aspect because of the sophistication of the machine and the way it was put together and the high performance that was driven from the machinery. To me, that smacked almost of science rather than bridge building, and that’s why I alluded to it that way.

It was an absolutely outstanding accomplishment. The horsepower-to-weight [ratio] on the engines—it’s just enormous. It’s like 0.5 for a high performance racer, and ours was over 100 or 200 to one, horsepower-to-weight ratio. All of that doesn’t come for free. You’re working in a terrible environment, and you’re stressing things beyond what you normally would stress them to. So it almost became science more than engineering in many respects.

Wright: I remember reading that you had said, “It’s not a truck, it was a race car.”

Schwinghamer: Yes, right.

Wright: Did you have some other notes of contributions that you wanted to talk about today? I think we went over the list that we had prepared, but I didn’t know if there were some other areas that you can think of?

Schwinghamer: About Shuttle specifically you mean? I had to spend a little of the time on the other things going on, Skylab and a few other things.

Wright: You had developed a hammer, could you share with us about that tool?

Schwinghamer: I didn’t say too much about it because [although] it is still used, the most extensive use was in Apollo, Saturn V. I think those bulkheads were 33 feet in diameter, and thick. When [one] makes a weld the weld metal becomes molten. Then when it solidifies it shrinks and that puts tension across that weld joint. So when they were welding up the bulkhead on the first Saturn V, the first stage, the S-IC stage, they made the measurements and the bulkhead was down two and a half inches from where it was supposed to be. The stress guys were having a fit about, “Oh my, our stress calculations, they’re based on this, not that.”

I had made contact with a plasma physicist in California, and his name was Dr. Waniak [phonetic]. He worked for Atomic Energy Commission for a while and then he had his own company. I read an article of his—he was trying to design coils to maintain a high magnetic field inside, and [he was blowing his] coils up, just with one pulse of electricity through the coil. So I thought, “I wonder what we could do with that about moving metal. If he could blow these coils apart, I can move aluminum I’m sure.” So I got on a plane, I went and I talked to him. I got enthused about it, and he got enthused about it. I finally gave him a contract and he did some of the theoretical analyses [more science]. Then I built the coils, based on his calculations largely, and I learned that I could stretch [electrically conductive] materials in every dimension isometrically.

Normally when [one] makes a weld and it shrinks like that, [one] comes along with a big roller and [squeezes] it. [But] that’s only two-dimensional stretching, [and] my coil stretched it in three dimensions, thickness, crossways, and every way. Back in those days if you had an idea you were allowed to pursue it. So I put some guys on it and we developed these coils and tested [them] on the metal. Then we tested [them] on some welds, and we found out that it stretched the welds in three dimensions instead of just two. [Conventional approach is to] roll planish it—that’s what [one] calls that—it does it in two dimensions but [not three, as the “hammer” does].

So I said, “Look, I believe we can chase these meridian welds on this big bulkhead and stretch that thing.” We had a big capacitor bank. It was about as big as two [large] tables, with capacitors in there. We designed electrical stuff to charge those capacitors, then we would discharge those charged capacitors through this coil and we’d get 20,000 or 30,000 amperes of current that would last only one and a half milliseconds, one and a half thousandths of a second. But boy, it would really stretch that material.

[But] I had a selling job. Here’s this big thing sitting, and the guys are all done with it, and they’re standing there and I come in there and say, “I think I can fix that.” There was a sound and round discussion about, “You’re going to let him get on that thing with that crazy tool that’s never been used before?” They let me get on it, so we started chasing one after the other, one after the other. I never will forget, it was so surprising. We got about three quarters of the way up on the last meridian weld, and that big 33-foot diameter thing went ka-ching, and it popped right up there where it was supposed to, because we took all that [shrinkage out of the welds]. We did that on all of [the bulkheads] after that. In fact when we made the gore segments [which make up the bulkheads], if they weren’t in contour we used to hit them [too] with this magnetic hammer. The guys called it the swinghammer.

Wright: I love that. That’s great.

Schwinghamer: I got that thing patented, and they used it in the ship factories at Pascagoula, Mississippi and they used it someplace up on the northwest coast building their hydrofoils. We had all kinds of stuff going on. But that preceded Shuttle; that was during Apollo, Saturn. It’s been a great time.

My dad used to come down for vacation. Sometimes he didn’t have a very good one, because I remember during Skylab I was out [at the center] all day, and several nights; I never spent any time with him. [Earlier] I took him fishing; he liked to fish. We went down the Tennessee River to fish. On the way home I remember—I think he must have been about 76 then—he said, “It’s getting harder and harder for me to go to work every day, it’s really a job. But you seem to get up and go out there like you’re anxious.”

I said, “Man, I’d like nothing better than to get up in the morning and go to work.”

He says, “You don’t know how lucky you are.” I never forgot that. That’s one of the benefits associated with it that you can’t put a price tag on.

Wright: It must have been an exciting time to be through with ASTP [Apollo-Soyuz Test Project], knowing that Shuttle was coming along. Not as fast as maybe everybody would like, but knowing you were working through those problems and a whole new program.

Schwinghamer: We had goals, objectives, we had schedules, and we knew where we were going. That’s a big difference. I don’t know if we know where we’re going today, but we did then. We absolutely knew where we were going, and believed we could.

Wright: Looking back now, do you think the flyback booster would have been the better way to go?

Schwinghamer: I was really sorry we didn’t get the flyback booster. But again, it was money. A flyback booster, we estimated its cost at that time would have been I think around $12 billion, maybe $14 billion. They gave us $5.5 billion to do the Shuttle, so you did what you could do with $5.5 billion. But oh, it would have been so neat to have that first stage separate, fly back and land, tank up and be ready to go again two days later. I think it was within the state of the art, or was close enough [so] that it could have been developed into the state of the art. We all wanted that. But the money just wasn’t there, so as engineers you do the best you can do with what you [have].

Wright: You retired in ’99.

Schwinghamer: Yes, and then they kept calling me back. So I guess I spent most of my time on Shuttle [after I retired]. I worked on several of the problems that they were having with Shuttle. Then of course when we lost Columbia I spent a lot of time working again with them on Shuttle and that program. I spent a lot of time at Michoud with them. They got back into some foam adhesion problems on the external tank. We were losing foam in flight, so we constructed a fault tree for that and then we found some very significant improvements. A little bit still comes off, always will, but nothing of any significance anymore. I worked a long time with them on that and then several other engine problems. This year in January I decided it’s about enough, “I’m 82, I think I’ll quit.” So I haven’t done much of anything with Shuttle anymore since January.

Wright: Would you like to talk any more about the work that you did with Columbia? Were you specific on the foam or were you doing something else for them in the Columbia accident?

Schwinghamer: No, it was all aspects of the accident, what did happen. There was a fault tree for that too, but after we looked at those high speed movies it became apparent what hit the wing leading edge. That was [suspicious]. For two weeks I kept saying, “You can’t let that foam do that.” You know what they kept telling me? It was a refrain, “But Bob, it’s only foam. Gee, foam is soft enough.” I said, “Yeah, going 700 miles an hour, kinetic energy, 22,000 foot-pounds. You know what my deer rifle kinetic energy is? 3,000 foot-pounds. Now does that give you some idea the difference, how important it was for that foam hitting that leading edge?”

They finally ran some tests. “Yeah, yeah, yeah, it could do it.” It did do it. But I looked at that and I said, “Man, I don’t need a fault tree for this. Look at that.” They had the pictures. As the vehicle ascended you could see that foam come down and strike that leading edge of the wing. Then after that, how to prevent that from happening, we went through all kinds of changes in the design to preclude that the foam should come off. Then we put some materials where the foam had been and changed those around.

That went on for a year or more I guess, but [frequently] they’d call me, and I’d pack up and go. Immediately after I retired they asked me to go around to the Centers and I went around and gave a pitch on fault tree analysis, how [one] sets up to do one and what the critical elements of it are and then gave a couple of examples—one of how we did it on the turbopumps on the blades. I gave them a little help for about a year there. I hope I helped some of them understand that process better. They thought it was valuable enough to all the Centers to know how to do the fault tree.

It’s interesting where [our fault tree] came from. I was at MIT [Massachusetts Institute of Technology, Cambridge] on an [Alfred P.] Sloan [Foundation] fellowship, and while I was there I read this article called “Decision Trees for Business Decisions.” It was written by some guy from Harvard [University, Cambridge]. It was very methodical; it was impressive in how methodical it was. For example if you want to build a new Walmart [store] someplace, this is how you put a fault tree on it and see if it feasible. I thought, “You know what, I believe that would work on [failure of other] problems.”

So I brought that process back to Marshall, that was about 1968. I started using it in the materials lab for different problems that required that kind of an approach. Then finally it pretty much got universally accepted. It turns out that we weren’t unique in that. There were other people other places unknown to us that had also discovered that idea and were [also] using it. It’s pretty much an accepted process and procedure now, but it had a tough row to hoe in the beginning. People didn’t want to take the time, spend the effort, contribute their best experts to something that might run one, two, three months, [or more].

Wright: And showed accountability, that’s the scary part of it.

Schwinghamer: The [experts’] names were on the [blocks on our] trees. The technical reports supporting the blocks had to come in, there’s a name on everybody’s and then when I was in one of those we’d quit at 4:00 and then we’d have a progress report. “Okay, Block 3, who’s the blockhead?”—I called them blockheads—“Block 3, how’d it go today? Did you find anything out?” Forcing everybody else that had blocks on the fault tree to listen to him helped them solve their blocks on the fault tree [also]. We had a lot of fun with those fault trees.

Wright: Probably a great transfer of information that never would have been exchanged.

Schwinghamer: Exactly. But again, it takes a little bit of ingenuity. You have to think a little bit out of the box. Then each one [of those blocks on the tree] has to be exonerated or indicted, every one, until you’re through. That gets way down to the lower level. Somebody may have to, on [a particular task], run a fatigue test and prove that the material would stand that cyclic fatigue up to the point where [failure occurred]. It involved all the engineering techniques and analytical approaches. It was a great thing for us and we used it successfully.

Wright: Sounds like it works well. I thank you so much for coming in and sharing all this information. I really appreciate you coming in.

Schwinghamer: I owe Marshall and the Space Shuttle a whole lot, I owe them a lot. I was happy as a clam, and I couldn’t have been more pleased anyplace else. I would not have wanted to do anything other than what I did do. So any time somebody said, “Can you help with this?” even after retirement, I [was] ready.

[End of interview]