NASA Johnson Space Center Oral History Project
Edited Oral History Transcript

Christopher B. Madden
Interviewed by Jennifer Ross-Nazzal
Houston, Texas – 14 April 2014

Ross-Nazzal: Today is April 14, 2014. This interview with Chris Madden is being conducted for the JSC Oral History Project. The interviewer is Jennifer Ross-Nazzal, assisted by Sandra Johnson. Thanks again for taking time out of your day to meet with us. As we start today, I thought I’d ask you about your early ties to NASA, and your memories of the Arc Jet Facility as a kid.

Madden: I grew up here in Clear Lake, right next door to NASA, in Seabrook [Texas], since I was in third grade. One of the stories I thought was interesting—one of my buddies, his dad was a Shuttle astronaut—this was before we had the first flight, and my friend Todd would talk about Space Shuttle. I’m like, “What the heck is that?” Didn’t even know what it was. “My dad’s going to be flying on the Space Shuttle.” He was a cool kid, and a couple of times his dad would bring us to work on a Saturday. His dad would go into work, he’d bring Todd and myself, and we would come in. He would go to the astronaut gym, which coincidentally ended up being basically the same building as one of the labs that we own right now in our group, the Radiant Heat Test Facility, and then also down the street a little bit was the Arc Jet.

He used to bring us on site, and he would go to the astronaut gym and work out. Todd and I would go in there and play a little bit of racquetball. There’s a little racquetball court in there. We were, like, 9, 10 years old, I guess. We’d also go out and catch snakes in the ditch. Todd was a big snake guy. We would be messing around in the ditch; I didn’t know it at the time, but right next to the Arc Jet. It was a little bit funny that I ended up coming to work at NASA and basically using that test facility a lot, and then ultimately having that under my wings as a branch chief.

Ross-Nazzal: That’s funny. Isn’t that the way life works out sometimes?

Madden: Right.

Ross-Nazzal: Who was the astronaut?

Madden: Story Musgrave. The other half of that story is not so much related to Arc Jet. He would take us to the astronaut gym, and like I said, we would mess around in there for a little while. About eight years ago, I was here at work, in a big meeting in Building 1, the top floor, and they were talking about tearing down Building 260, which is the astronaut gym number. On the other half of the building is our Radiant Heat Test Facility; they share a wall. They were talking about tearing that down, and I’m like, “Hey, we have an engineering test lab there! We can’t tear it down!”

It was, I think, somewhat of a surprise to people because the numbering system is a little bit weird. The gym side was 260-A, and the test side was 260. I think there was confusion, “Is it the whole building that’s going to go down?” Then, that all got cleared up, and they ended up tearing down the gym side, which is 260-A, and building a new gym down the street a little bit. [Building] 260 itself survived. Building 260 has a history itself because it used to house the egress test facilities, so they had an above-ground pool in there, and they would test getting out of the Apollo capsule properly. They had a big water tank. They would put in a mock-up of the capsule, and the astronauts would practice getting out of the capsule.

Ross-Nazzal: Why don’t you give me a brief overview of your career at JSC, and how you came to be associated with Building 222.

Madden: Upon graduating high school, I went to the University of Texas [Austin]. Partway through my freshman year, my brother said, “Hey, you ought to try this co-op program.” So I went down to the Co-Op Office and applied for that, and my recollection is this real simple, one-page application. What’s your name, what are your grades, and interests? I got accepted, and so that summer after my freshman year, I did my first co-op tour here. I ended up doing three more tours as a co-op, and then, upon graduation, I got a job offer and came to work in what was, at the time, called the Thermal Branch.

Got to work, first assignment was looking at the Shuttle and the thermal response you would have if the ET [External Tank] door was cracked open a little bit. The ET doors, they’re probably about three-foot doors that are opened during ascent. That’s where the oxygen and hydrogen fuel lines come in, into the main engines, from the external tank. Once they get to orbit and they’re done with the tank, the doors close. So there’s a concern over, “Hey, are these doors going to latch all the way?” because there’s an intermediate latching point where the door’s not completely shut. They’re worried about a failure scenario where the door is cracked open a little bit. If the door is cracked open, how much hot gas are we going to get in there? Is there anything inside that’s going to melt? [If so], we’re going to have a breach. Me and another guy, Matt Adler [phonetic], did that work. Matt did the heating side and I did the thermal response side. We determined, yes, you can have the door open like a half-inch, anything more than that there’d be problems.

That was my first assignment, and then for subsequent years, it was assignments like that, a lot of what-ifs on Shuttle. If there was problems, what’s going to go wrong, and it was all related to thermal analysis. We would usually take aerothermodynamic heating rates and determine what the thermal response would be. I did that for a few years and found that really interesting. We did a lot of stuff on Shuttle—what if the tile’s damaged, how much damage can you stand? Did that, and then about, I guess, ’98 or so, this project came along called X-38. It was ultimately called X-38, it had a few names before that, but the intent was to build an in-house Crew Return Vehicle for Station in case we needed to evacuate. Even back at that time, there was some uncertainty going on with whether or not we can use Soyuz capsules and would everybody fit if we had a full crew. NASA decided to try to develop our own Emergency Crew Return Vehicle. The intent was to do all this in-house, all the engineering, all the build, all the drawings, and everything else. So I started working on that project and started as the TPS [Thermal Protection System] guy.

Then my role grew a little bit to include what we call the composite aeroshell, which is the skin of the vehicle; the design and build to that is a composite system, which was fairly new for our engineering team—to design and build composites. We had to build up the manufacturing portion all in-house, from the ground up. There was a lot of work during that. We had a real good team and did that. Later on, I became lead for all the structures work and the thermal work, so our team had to design all the structures, the TPS, and the aeroshell. It included some interesting partnerships. The project partnered with ESA [European Space Agency] for them to deliver the body flaps, which was a great engineering integration challenge. ESA also was supplying the rudders, which were on the fins of the vehicle, a leading edge on the fins, the nose cap, and the landing gears. We had a lot of European involvement for some of these major systems that went with our vehicle. That went on. We did a lot of good work there. The project built several drop-test vehicles to test parachute systems, control systems, and everything else. My team was focused on the spaceflight vehicle, which was intended to fly in Shuttle in the payload bay and return for a test flight, unmanned. We got that vehicle probably 85-90 percent complete, and by that time, it was one of the classic problems, I think the cost and the schedules got stretched out. It ended up getting canceled.

After X-38 was canceled, I asked my boss, “Hey, what’s next?” He says, “Well, why don’t you go ahead and see if you want to do deputy branch chief?” I said sure, and so I did that for five or six years, and then my boss ended up moving on to the NESC [NASA Engineering and Safety Center]— a brilliant guy, Steve [Steven L.] Rickman. Applied for the branch chief [job], and I got that job, so I’ve been branch chief for the past five years. That’s where I’m at, now.

It’s been a tough past few years in terms of the Arc Jet. There has been several campaigns to close that test facility. The old-timers have stories, “Years and years ago, teams would come through and did assessments,” and it’s like, “Hey, we need to consolidate facilities.” It always turned out, “No, we still need this one.” A more recent one, in the 2000 timeframe, was very close to shutting down. We, in fact, ended up reducing the staff to what we call the bare bones staff, ended up laying off good, talented, and skilled people. We’re running it bare bones, and the Columbia disaster [STS-107] occurred.

There was a lot of testing required, number one, to determine the cause of the accident. We did a lot of testing in terms of the investigation for Columbia, and then after the Agency decided we’re going to fly again and started getting serious with that, we did a lot of testing for the repair systems, and some more failure scenario testing. The facility was very active for the Columbia investigation and subsequent return to flight, and it remained very active up until just a few weeks ago, when we were finally ordered to shut down. The past 10-12 years, we have always had a test article waiting to be tested, 100 percent occupancy, and 100 percent utilization. [We] just always had a test demand. So it was really tough for my guys and myself, trying to come to grips with a decision like that, where we have a huge demand. The engineering community needs a facility like this, yet we’re being told we must shut down and move everything to Ames [Research Center, Moffett Field, California]. It was tough to come to grips with that.

Some of the interesting tests on the Columbia investigation, like I said, we were real involved there. In fact, this was probably a year into the investigations. Sifting through all the data, the CAIB [Columbia Accident Investigation Board] started focusing in on a scenario that we thought might have happened. We set up a test where we put a wire bundle that was a replicate of the wire bundles in the Shuttle itself, and we made a little fixture and created a hole in a piece of aluminum, to replicate the front spar of the wing. We put it in the Arc Jet, had flow going through that hole and over these wire bundles. We had continuity on the bundles, and so we can see, with the hot gas that’s blowing on to the bundles, the rate at which the signals dropped out. We’re able to correlate that drop-out rate to the drop-out rate seen in flight. That was a really key test because it was very eerie in that we think it just replicated what actually happened inside the wing. From that test, we were able to back out an assumed hole size on the RCC [Reinforced Carbon-Carbon] panel, through analysis. Just about that time, within a week or two, the team that was investigating the foam impact strikes at Southwest Research Institute [San Antonio, Texas], on full-size RCC panels, replicated the same hole size that we surmised from the Arc Jet tests. Everything came together all within a couple of weeks, saying, “We had a probably 10-12-inch hole in this wing,” and that fed into the final conclusion for the accident.

Ross-Nazzal: You testified in front of the CAIB. Were you talking about the testing you did?

Madden: Yes, early on, they had questions. They just wanted to get some background on aerothermodynamics, thermal analysis, TPS, aerodynamics, so myself, Steve [Steven G.] Labbe, and Joe [Jose M.] Caram just answered a few questions on general technical questions on re-entry and how that works, and how TPS systems work.

Ross-Nazzal: You also mentioned you guys were doing some work on the repair test. Can you talk about that a little bit?

Madden: Right, so for repair, there was two camps, and boy, these meetings, these were big meetings with hundreds of people in them. There was always this tension and friction between these two camps on how to repair the tile portion of the Shuttle. It was a little bit off-topic, if you will, because we think what really happened, we had a hole on the leading edge, which is a carbon system. The tile repair camps, there was a camp that wanted to put just an overlay. Let’s say we had a big damage on tile, and they wanted to just screw with auger-type screws an overlay, a cover that just covered the damage. Then the other camp wanted to have STA-54, which is a rubber foam-type material; they were developing a gun to apply that material, to fill up a gouge with the filler material. There’s a lot of tension, a lot of, “Hey, this won’t work, that won’t work!” There’s a lot of difficulty in either one, in executing them. It sounds easy at first, but how you brace yourself and apply that material was a little tricky. We did a lot of tests in there. We would carve out some damage in panels of tile and fill it up with varying levels of material and different formulations, so the material went through several different formulations. We did a lot of tests to try and perfect that technique.

Ross-Nazzal: What did you guys find?

Madden: One thing that was a complicating factor is they would swell up. They would heat and swell and bubble up, so now you have effectively a hump on your aerodynamic surface, so does that induce boundary-layer-transition earlier downstream? Does it cause localized heating that will further damage it? We started getting smart enough about the material response to figure out how much it would swell, so the intent was to under-fill it by a quarter-inch or so, and allow it to swell, so it doesn’t stick up as a hump.

Ross-Nazzal: Did they end up using it on orbit?

Madden: No. Before we got to the return-to-flight flight [STS-114], we spent about a year developing test articles that were prefabricated ahead of time, so that if there was some sort of damage in flight, we can try and replicate it on the ground on these test articles, test it in Arc Jet, and see whether or not we would need to repair. If we did repair, would the repair work? We had all that set up ahead of time. Then, one of the flights, [STS]-117, there is a bit of a blanket that was peeled back on the OMS [Orbital Maneuvering System] pod. It really didn’t look horribly bad. The problem was, on the OMS pod, it’s a graphite sandwich, and the graphite doesn’t do as well as aluminum in dissipating heat. If you had the blanket lifted up a little bit, you’re going to get some hot gases in there and heat up that graphite epoxy skin, and it wouldn’t dissipate that heat as well as something like skin-string or aluminum. The teams, they couldn’t really clear it analytically. This was during the mission. We didn’t have a test article set up ahead of time like we intended. We can’t build the whole vehicle ahead of time. We had a guy come in and try to replicate that area, which is tile that terminates into a blanket that bucks up to it. We decided, “Instead of testing to see if it’s going to be bad, let’s go ahead and test and see if repair’s going to work.”

We built a test article and then damaged it like it was in flight, peeled back the blanket, and then the MOD [Mission Operations Directorate] guys came up with a repair technique. What they came up with was to use actual surgical staples that they have in a dock kit, so they took in that little staple gun. We went out there and stapled the blanket back together, as they would during an EVA [Extravehicular Activity]. We put that in the facility, tested it, and showed that the repair worked. That was interesting because it was an arduous process. We were up a couple of nights in a row, trying to get the tests off, and finally get it off right before one of the morning MMTs [Mission Management Team meetings], where they had to decide whether or not they’re going to do the EVA or not. We came back from testing and says, “Hey, yes, it’s going to work.” Nancy [J.] Currie ran over to the MMT and said, “The repair is going to work, let’s do it,” and they decided to do it. Danny [John D.] Olivas went out there during the EVA and repaired it, just like his astronaut buddy did on the ground. Came back, and it looked great.

It was an interesting test because we had video of the test, and you can see the staples. We also had these pins with a ring on it, and you can see them glowing, but they held together, and it worked great. The very next mission, there was a deep damage in the tile—it was all the way down to what we called the filler bar, which is the very bottom pad between the gaps of the tile. Through the inspection system, they were able to get what we called the point cloud; it was just a set of data, X, Y, Z coordinates for the surface of the damaged area. We were able to take that data and go to a CNC [Computer Numerical Control] machine on one of our test articles, hog out, and replicate that damage just like it was in flight. That test was the corollary to the previous test: here, we’re testing, “Hey, is this damage going to be a problem?” We don’t really want to repair it, but the teams were getting ready to do the repair. We recreated the damage, went and tested it, and we probably over-tested it.

We chose a boundary-layer transition. You have to guess on when the boundary layer is going to transition. The turbulence is a whole new field, a whole separate field of science on trying to predict when boundary layers go from laminar to turbulent. It’s important for reentry vehicles, because the turbulent heating is much higher than the laminar. You have the hot energy much closer to the surface, gradients are higher, and it just pumps more heat in. It’s a higher heating rate on the surface. We tested it, and like I said, probably over-tested it because it formed what we call a wormhole in the test article. Once you start forming any sort of little cavity, what happens is what we call the view factor to space is reduced in that cavity. It’s a propagation problem, so inside of that cavity gets a little bit hotter and it gets deeper, as it gets deeper, that view factor to space gets lower and lower, and so it exasperates itself and it grows. We’ve got this wormhole where we basically melted out some tiles a good three or four inches into the downstream tile.

It looked horrible. We’re like, “Crap, what are we going to do? We’re going to have to repair.” After thinking about it a little bit more, we’re looking at the aluminum temperatures on the back side. Aluminum really didn’t get very hot; it just got a little bit hotter than it normally would if it had a virgin tile. What’s good about aluminum is if you have local damage, that extra heat you get in to that aluminum will actually conduct out. It’ll almost cool itself via conduction in the aluminum skin itself. What we decided was, “Hey, the aluminum was great. Sure, it’s ugly, but we’re not going to have a breach, and it should be good.” The program made a tough decision to go ahead and come back in without a repair. We came back, and it did not have the wormhole. Of course, the subsurface aluminum looked great. There was no problems there, so it was another real-time test we did that helped us fly the Shuttle.

Ross-Nazzal: It sounds like Columbia really changed the way the Arc Jet operated.

Madden: I don’t know about changing it—it’s certainly a ton of things to do. Like I said, we were just really busy for years and years and years and remained so. One thing it did do that we didn’t like was it didn’t leave us any time to develop better ways to test. My guys got some good ideas on better ways to do this. You’ve got to understand, this technology was developed on the fly, back in the mid-sixties, so we can help the space program itself. They didn’t have the tools we have today. They put stuff together, turned it on, and see if it’d work, and kept trying until it did, just trial and error. They’re getting to a point where, “Yes, this is working. We’re getting this hot gas. We’re able to float over a test article, and the chemistry’s somewhat, we think, similar.” There were some improvements made in the seventies on different types of heaters; there’s three or four different heater designs out there, but we think there’s probably better ways to do it.

We think we have ways to replicate the re-entry environment from Mars, which is very difficult. Coming back from Low-Earth Orbit, a very high-energy event, it’s very underrated in terms of its danger and its potential for disaster. We saw [it] on Columbia. Coming back from the Moon it’s even tougher, because you got to realize that all that kinetic energy has to be dissipated in the form of heat. Your landing velocity is basically, for argument’s sake, zero. All that kinetic energy, flying at mach 24, 26,000 feet per second, ½(mv2), gets converted into heat through the laws of thermodynamics. Low-Earth Orbit is tough, coming back from the Moon is tougher, and then coming back from Mars, you call it an order of magnitude more difficult because of the energies involved.

What happens when you come back from Mars is—and a little bit for the Moon, but for Mars, especially—those high velocities, it’s really not convection-type of heating around the spacecraft; most of the heating is a form of radiation coming off the shock itself. The shock gets so hot that it’s radiating like the Sun and just is a big heat lamp right in front of your vehicle, the whole time you’re flying through until you dissipate all your energy. These Arc Jet facilities test, and they test convector plates; it’s hot gases flowing to form a convection, whereas coming back from Mars, it’s mostly going to be radiation. We have ideas in using the high-powered lasers to replicate that radiant heat at the same time as the convective heat and try and develop ways to simulate coming from Mars, which really, there’s not much capability right now in testing the Mars return environment. We regret, and it was unfortunate we didn’t have time to develop these techniques because we were so busy testing. It was always people basically pounding on the table, “Hey, I need to get my test data; I need to get my tests,” and we didn’t have the time, [during the] past 10-12 years, to do any of that development. We, as an Agency, need to have that capability.

Ross-Nazzal: Were you primarily working Shuttle flights at that point, or were you also working on different programs and projects, looking at different materials?

Madden: Not so much material development. The flagship test for that facility back in the sixties and seventies and eighties was material development. It’s great as a materials tester. For Shuttle after Columbia, we’re testing systems. We’re testing repair systems, we’re testing gaps, we’re testing different configurations. More of what I would call system tests as opposed to material tests.

That was the other thing, we didn’t have a lot of time to do material development, nor test it. Ames was doing some of that. There was progress made within the Agency, but here at the JSC facility we got away from material development tests. The facility’s good for material testing. When I say “material testing,” it’s, “Will this material perform well as a thermal protection material? Will it survive? Will it maintain its shape? Will it insulate and protect whatever structure is underneath?” It’s well suited for that because for the most part, it recreates the different forms of energy flowing around the vehicle.

There’s thermal energy, of course, in the flow field, but there’s also chemical energy, in the fact that the flow gases are so hot that the atomic molecules in air—nitrogen and oxygen—those molecules split into atomic forms of atomic oxygen and atomic nitrogen. The atomic oxygen, it’s especially important because oxygen, as you know, likes to react with everything. Even at room temperature, burning of a flame, that’s oxidation of the wood. Oxygen wants to be involved in that reaction. At higher temperatures, it wants to react with even more things, and when it’s atomic, it really wants to react. When it reacts with something, it’s going to do damage to it, almost by definition. It’s important that for re-entry simulation, that you do at least a reasonable job in replicating the chemical environment as well—that is, the atomic oxygen. Arc jets are good in that because it gets the gases hot enough where the enthalpy is high enough where the oxygen gets disassociated. You can see if you’re going to get this atomic oxygen attack on your material.

Ross-Nazzal: Can you explain, for the reader who may not be aware, how an Arc Jet works? You’ve given some indication about gas and thermal and radiant heat and things, but can you give a summary?

Madden: You’ve got to think about your spacecraft. You’re flying through the upper atmosphere at a very high velocity, it’s hypersonic, Mach 24 is usually what you’re coming in at for Low-Earth Orbit, and you develop a shock in front of it. If you can just picture a molecule in front of the shock, it’s at the ambient temperature up in the upper atmosphere; it’s actually pretty cold, it’s probably –100 degrees, and it very rapidly gets accelerated or decelerated, depending on your vantage point, through this shock. The energy level goes up, so if you think about it as decelerating, it has a cool temperature and a huge velocity, and it suddenly decelerates through the shock, so that has to be converted into heat. The temperature of that molecule rises very high. If you were to look at just a straight MCCP [phonetic] on the temperature, it’d be tremendous, it’d be like 10,000 degrees Fahrenheit, the temperature of the Sun.

After you get so hot, it doesn’t make sense to talk about temperature. Temperature’s not a good parameter to talk about because temperature really is the average vibrational energy of a molecule, and heck, it’s not even a molecule anymore. A lot of the energy is tied up into splitting those molecules into atoms. The word “temperature” doesn’t apply anymore. Of course, you have this real high-energy gas that’s very hot, and it’s also very chemically active. It’s split into these atomic species, like I said. In Arc Jet, we want to replicate that, but we don’t do it hypersonically. Instead, we take cold air, cold nitrogen and cold oxygen, insert it into a heater, and actually pre-heat the gas in our heater. Then the gas is accelerated into the test chamber, and it’ll cross a supersonic shock, probably Mach 8 or so, rather than Mach 24. The sin, if you will, in testing on the ground is you have to pre-heat the gas and cross a mild shock, versus flight, where you have cold gas crossing a strong shock. Ultimately, behind the shock, where the gas is flowing around the test article or the vehicle, we hope that it’s similar environment when we test on the ground. We’ve done a lot of diagnostics to see how similar it is.

How do you heat the gas? We insert the oxygen and nitrogen into this long column, and it’s actually a column of copper segments that are insulated electrically between each little segment. In one end, there’s an electrode, the cathode, and the other end is an anode. The anode actually has a hole in it to allow the gases to flow through it. The gases get injected into this heater, and we strike an arc from the cathode to the anode with a very high voltage and current. It’s where the Arc Heater or Arc Jet comes from. There’s an electrical arc on the inside that heats up and excites the air as it’s inserted into the heater, so that’s how it gets preheated. Then it flows into the test chamber, crosses the shock, like I said, and flows around the test article to provide your test. It takes a lot to do that, though. We need a lot of power. Our facility’s rated at 10 megawatt. We’ve tested up even higher, up to 12-13 megawatt. The energy densities are tremendous; focusing all the energy into tiny three-or-four-inch diameter test specimens, we need a lot of power; we need a lot of cooling water.

What happens is that arc, it’s heating up the gas, but you’re heating up the heater itself and you’re heating up the walls at the test chamber. All those are copper and have to be cooled with water-cooling loops. Part of the replication is getting the pressure right, so most of the heating for re-entry spacecraft are pretty high in the atmosphere, and the density’s fairly low, the pressure’s fairly low. Our stagnation pressure, during peak heating, it’s 100 PSF [Pounds Per Square Foot] or so. In order to get that low pressure, we evacuate the test chamber with a vacuum system, keeping in mind we’re constantly adding gases to it. We constantly have to remove those gases, number one, to accelerate the flow, number two, to keep the pressure down and get them out of there. You can’t do it with a roughing pump or a turbo pump or anything else. The only way we’ve found to do it is with a steam injector system.

We need a bunch of steam, so we have a dedicated boiler in the facility, out back. That high-enthalpy steam is routed to these ejectors, which are really simply little horns inside our diffuser system, which is just a big tunnel attached to our test chamber. You blow the steam in through these horns and through motive forces and Venturi effects and momentum, it just sucks the air with it. You have to do it four different times, because each time you only get a certain amount of pressure drop. We have four stages, so the steam is injected down this tube. One hundred feet later, you got another steam ejector, and it happens four times. Now, you’ve got to start cooling that stuff too, and you have to condense the steam. We have an industrial system out back that condenses that steam, cools our diffuser, cools everything else that needs to be cooled, and it goes through a big cooling tower. We got a pond, as well.

The cooling water on the Arc Heater side has to be de-ionized and very pure in order to remove the electrical conductivity. Very pure water isn’t electrically conductive. The only reason why it’s electrically conductive in a rainstorm [is] because it’s got dirt and ions and everything else that would conduct the electricity. If it’s pure water, it’s a good insulator, so we’ve got to keep that water pure, and it’s on a closed loop. That’s one of the complicating factors, we don’t want electrical conduction from the heater to ground somewhere else because the arc will attach to that, and there’d be problems.

We had a real good video system. One of the key products, if you will, is video. Steve [Steven V.] Del Papa invented a real nice video system, where he took basically still cameras and just took pictures, 15 times a second, like you would on a cartoon. It’s real simple in terms of cost and set-up, but it results in tremendous amounts of data because you have full resolution picture of your test article, 15 times a second. You add those up, and then you start getting into lots and lots of data. Our solution to that was just to brute force it at the time. Hard drives got to be real cheap, so we just bought terabyte hard drives and kept filling them up. It made for a real nice high-def finish, and a video system on the cheap.

Ross-Nazzal: You took us on a tour of the facility, and it looks like there have been changes made since the building was first designed in 1967.

Madden: Right. The guys that have been around, like Jim [James] Milhoan, even before ’67, they did some testing with Arc Jets out in 260-something. It’s where they actually have the key shop, now, for all the keys on site. They make them in this little building. That [was where] they had the first Arc Jet, and it actually was much smaller. It exhausted into the high bay, the room itself. We got some pictures from the old times. There’s guys standing right next to it, who are turning it on—for short durations, presumably. That was back before—

Ross-Nazzal: Before OSHA [Occupational Safety and Health Administration]?

Madden: Right, before everything.

Ross-Nazzal: Testing sounds pretty complicated. How long does it take you to prepare, and how many people are involved in this whole effort?

Madden: Some of the simpler tests where you’re just taking a puck without much instrumentation—the instrumentation is what is [complicated], in terms of the hours we need [to prepare]—but if it’s a real simple puck, if you will, on just a carbon holder, the guys, they got to put it in there and they got to pack insulation around it, so that just takes a few hours. We like to take pictures before of every test article, so we do photographs. We log them in, try to keep track of every test article we’ve had. The facility itself needs to be in good shape. For any test, simple or not, the facility has to work. There’s a lot of maintenance because it is high-energy density, so any time you have a high-energy or high-energy density spacecraft, airplane, car—compare an Indy car versus a regular car. That’s why they fail so much. They’re operating at the edge of the envelope. It’s very high-energy, high horsepower; they have a high failure rate because of the energy, really.

This takes a lot of maintenance. Most of the systems are single-string, so if one of the transformers isn’t working, we don’t have electrical power; if one of the injectors isn’t working, we don’t have a vacuum; if the boiler’s out, we don’t have vacuum. If one of the water cooling pumps is out, we don’t test. There’s a lot of maintenance in terms of repairing pumps and stuff that go out, and preventative maintenance. That’s why it’s tough to run these facilities. There’s not a ton of them around, and with everything being single-string it makes the maintenance part tough. We’ve had periods of time where we just couldn’t test for months at a time, and weeks at a time, days at a time. [That] comes up periodically, and it drives the customers crazy, because they’re waiting for the test. We’re like, “Hey, we’re down, we cannot test.”

Maintenance is a big one and then preparing the test article. Our test directors will meet with the customer and figure out how they want to test. “Do you want us to focus on the heat rate or pressure or enthalpy?” That’s the parameters that they toss back and forth. “Do you want a shear-type test where it’s more of a shear flow, like in a wedge, or do you want a stagnation test?” There’s engineering work upfront, designing the test itself. Like I said, preparing the model, preparing the facility, and then installing the model into the arms or into the chamber in the test cell. That’s one thing—we always had great plans to improve our way of getting specimens in and out of the test chamber, because right now we’ve got to do a lot of hands-on work inside the chamber itself. If there’s a guy in there working, we can’t test, of course. If we had a system where he can be doing his integration outside we can be testing. Then if we get the system rolling, that test article or that assembly gets bolted in, and then he’s working on another one while they’re testing that one. We never really had a chance to perfect a system like that. It’d take some time and money to do.

Ross-Nazzal: It sounds like there’s a lot of work prior to a test. The test is probably a very small effort.

Madden: Right. You can spend a day or two preparing for a simple test, and then the test itself is usually just a few minutes long. Shuttle was an interesting spacecraft because it’s got a large wing area, so high lift to drag ratio. That’s what made it successful. It’s able to keep its lift up and fly in the upper atmosphere, modulate the energy disposition, and spread it over a longer period of time, so that the heat rates weren’t so high. What happens with more of a capsule-type shape, like Apollo or MPCV [Multi-Purpose Crew Vehicle], is it doesn’t have as much lift, and it’s diving into the atmosphere quicker and hotter, so you have much higher heat rates on a capsule like that. It’s a shorter duration, but the rates are higher, so it’s a more peaky profile on heat rate. For Shuttle, we’d do 10, 12, 15-minute tests because that’s how long the heat pulse lasted.

In fact, there were some RCC tests for mass loss. I’ll explain that in a second, but there is some tests that were hours and hours long, all in one test. If you were talking about a one-time, single-use system, on a capsule, the test is just a couple of minutes long. Those RCC mass loss tests, I was mentioning the oxygen in the flow field wants to combine with anything, so for a system like reinforced carbon-carbon, like the leading edge of the Shuttle, it’s made out of carbon, and that oxygen wants to chew it up. They had to build a protective coating with silicon carbide to protect the carbon underneath because if you didn’t have that, it would just go away in seconds from the oxidation. That coating isn’t perfect—it’s got a bunch of tiny holes in it—so over a period of time, there is actually some oxidation of the carbon underneath the protective coating on those RCC parts.

Don [Donald M.] Curry went through a big test program to quantify how much mass loss you get and then correlate that to how much strength you lose over a period of time after you’ve flown so many missions. From all that testing, they were able to develop, for each panel, how many missions it can last before it has to be replaced. Some of them, some of the hotter panels, they would only—I’m going to get my numbers wrong—they’re only good for 10 missions or so before they had to be replaced, whereas some of the cooler panels can go 100 missions without having enough mass loss to be a problem.

You had a good point—usually, it’s a few days to set up for a test campaign, and then each test is a couple of minutes long. We had a system set up, and Ames has this as well, where we can test multiple test articles in one run. The hard part is firing up the Arc Jet itself, so we try to take advantage of that and test more than one specimen in each Arc on, or each run.

Ross-Nazzal: How many people are working when there is a run, and you’ve got that control center in the building?

Madden: We have a little control room. Nowadays, there’s a power operator. He’s basically in charge of the power to the heater and dialing in the current and the voltage that’s required. The test conductor, actually, he’s the one throwing switches, turning valves on and off, inserting the test article, and putting all the commands into the control system. We have a quality engineer, usually, that’s witnessing the test.

Then we’ll have a test director, who’s the leader of the whole test. He gives all the calls, so he’s like the Flight Director. He’s saying, “Hey, go ahead and insert the arm,” and then the test conductor presses the buttons to do the things. He’s calling the shots. He’s also looking at the data coming from the test article and from the facility. We monitor pressures, we monitor enthalpies, temperatures. Most of the time, the test requestor or test article expert’s sitting right next to him, looking at the data, saying, “Hey, I don’t like what’s going on, let’s pull it out,” or, “Hey, leave it in there, it’s supposed to look like that.” We also have a data person that’s running all the data systems and making sure the data gets recorded properly. There’s a guy in the boiler house operating the boiler. He’s on the headset, but he’s doing things to the boiler and making sure it’s producing enough steam.

Ross-Nazzal: When you were working on the X-38 and working on its thermal system, were you doing testing out at the Arc Jet?

Madden: Yes. Now, we didn’t have a huge campaign because we’re using systems which we had a competence in. The Europeans tested some of their materials there, for the body flap and our rudder, then we had some tests of the gaps, and some of our discontinuities.

Ross-Nazzal: That brought up another question. Space Station is such an international effort, do the international partners have a chance, or have they had a chance to test at the Arc Jet Facility, besides ESA for the X-38? Do they ever bring stuff over?

Madden: Not very often. There’s a similar facility in Germany that ESA owns, and then Italy built a really nice Arc Jet. They’ve run into tough times, and I think it’s associated with the economy and the politics in Italy. They haven't been running for a couple of years, but they had a real nice complex. I went there one time. They’re able to test, theoretically, to high heat rates, high pressures, and pretty nice test envelope. Like I said, they’ve been down for a while. I think a lot of the Europeans tested there and in Germany, as opposed to dealing with us and our ITAR [International Traffic in Arms Regulations] policies.

Ross-Nazzal: You mentioned that there weren’t that many Arc Jet facilities, but there are other Arc Jet facilities. How is JSC’s facility different from this one in Italy and Germany, Ames, I think there was one at Langley [Research Center, Hampton, Virginia] at one point?

Madden: Right. There’s one in Russia I know very little about. In the U.S.—our facility we had here at Johnson, and then Ames has a complex where they have, I think, four test cells now, with different types of heaters and set-ups. They’ve got a nice operation. Langley has a small Arc Jet. It’s actually pretty cool because one or two guys run it. It’s just set up in a laboratory somewhere, so it’s easy to get in and out of there and fairly cheap because it’s nice and simple. It’s just that the sample size gets small once you’re talking about smaller facilities. It’s good for preliminary evaluation, but for systems, if you’re going to test the gaps in a tile system or a TPS system or test repairs, you just can’t do it. Test article size is the main thing, and even material tests, once you talk about small sizes, you get a lot of 3-D effects that you just can’t get rid of because it’s so small. Since heat shields are usually bigger, the 3-D effects aren’t very great. Most of the heat flow is one-dimensional, so you want to simulate that in your test. Let’s see, Boeing had a good one. In St. Louis [Missouri], they had their own facility, and it’s gone back and forth. It’s hard to keep up with them on how active they are. I think they’ve pretty much gotten away from that operation. In fact, one of our last tests we did was for Boeing.

There’s a big, classic difference between JSC and Ames. We’re certainly doing some things that are a little bit different. We developed a capability to test CO2. This is if you wanted to simulate the environment during the Mars entry, which can be done. It’s been done certainly without a CO2 Arc Jet in terms of developing a heat shield for Mars. It’s been, as everybody knows, many success stories. Once you start getting to larger class missions, like a manned mission, we feel you’re going to need to test in a CO2 environment to get closer to replicating a Mars entry. We developed techniques to use lasers to interrogate the flow field, and actually determine the velocity of the flow, the chemical component, the thermodynamic component, and the kinetic energy component. It just goes back to what I was telling you about the chemistry, how similar is this flow field to the flow field you get in flight. You end up going through that critical thinking and that assessment and those analyses.

Actually, in partnership with Ames, [we] developed that technique to interrogate the flow with lasers. We also have some unique capabilities, in we can test low pressure, low heat rate, so different areas. If you were to plot the flight envelope for say, MPCV [Multi-Purpose Crew Vehicle], it’s not always high heat and what you might call mid pressure. There’s also low pressure/high heat, there’s high pressure/low heat, there’s different corners. We’re able to do some work to get these unique corners of the envelope. Those were some of the, “Hey, Ames, if y’all can do these things, then we’re not so unique,” right? They’re tasked with developing these different techniques so that the Agency would retain the capability. People will tell you that Ames facilities are much higher power. Their power system is rated to a higher power, but we’re able to hit some of these high heat rate, high pressures. Not quite as high as them, but we’re pretty competitive with them in terms of the high-energy, high-pressure test points.

Their heater is a little bit different in its design, that was the other big thing. They use compressed air as opposed to we manufacture our air, in that we insert nitrogen and oxygen separately. What that allows us to do, is it protects our cathodes, our electrodes, so we don’t insert the oxygen close to the electrodes. Otherwise, they would burn up, so we shield them with the nitrogen. Since Ames is injecting air, they needed to shield their electrodes with argon, which is an inert gas, but there’s always been an argument, “How much does it affect the results?” That’s one reason they took a lot of our heater parts. We gave them a heater a year and a half ago, I think, so they can have the capability to test with simulated air, rather than compressed air, and get rid of the argon.

Ross-Nazzal: What role has the Arc Jet played in testing of the MPCV Orion capsule?

Madden: It’s been huge. You got to go back to Apollo for that lunar return. They developed a material called AVCO, which is an ablative system, the TPS system, used on Apollo. After Apollo, all the focus was on Shuttle, of course, and developing a reusable system, which at this time was a great innovation, in that you don’t have to replace the thermal protection system or the heat shield every flight. There was a lot of energy spent in developing these reusable systems, and it was a great success story. This is before my time, of course, but they had to drop ablative systems. That’s where NASA’s going—we’re building a Shuttle. We’re going back and forth 100 times. They developed these reusable systems, dropped the ablative systems. The company that made it put the formula away and started working on something else.

Actually, a little bit before MPCV came along, some of our guys recognized, “Hey, we can’t lose this capability forever. What are we going to do if we want to build a capsule and explore outer space?” They were able to convince [NASA] Headquarters [Washington, DC] to form this program called ADP [Advanced Development Project], really for the sole purpose of retaining the skills and retaining the capability in the country to build ablative systems. They put a call out to industry and had people respond with their best ablators. Since the Apollo days, [the company] who used to make AVCO—I think the company was Avco—traded hands a few times and I think it even traded since the ADP days to Textron. There was also PICA, which is Phenolic Impregnated Carbon Ablator.

The ADP did a lot of good tests and a lot of good analysis to develop these materials and get them back. MPCV started firing it up and had to pick a system, and initially, they base-lined PICA. Then there was problems discovered with building PICA on a large scale because it’s built in blocks, if you will, so now you got to dice it up. The MPCV program went through a re-down-select, and actually changed the system to Avcoat. Textron had to fire up their production floor to build it on a larger scale, and they ultimately did. The guys got EFT [Exploration Flight Test]-1, the heat shield built. It’s ready to go. That was a long process to get that capability back up in the United States. It’s still a bit tenuous because now Textron has got to wait around a little bit before EM [Exploration Mission]-1, and they just can’t have guys sitting around. It’s a tricky thing, building these spaceships one at a time, spread apart. A huge program like Shuttle is so big, it’s there to keep everybody active all the time, whereas a smaller MPCV, it’s a little bit tougher.

The Arc Jet did a lot of work for developing the Avcoat, doing tests on PICA. We have to do a lot of testing to get the properties. It went through different formulations. To get the properties and everything else so that we can do analytical predictions on how it’s going to behave, those predictions are impossible without having Arc Jet testing. There’s a lot of testing, just straight on the material itself, determining its properties, that we need to do our ablation analysis. The other interesting part of the Avcoat story is when Textron went to go make it for the first time, during the ADP, we tested it and it did horribly. It just crumbled apart. They had the recipe back from the Apollo days; they followed it. It’s just one of those weird things; it’s like cooking: the recipe says to do this but you really need to use a cast-iron skillet as opposed to an aluminum skillet. It makes a difference; it’s just not written in the recipe, or there might even be something in there that you throw in and it’s not written down. They did a really good job, but they struggled a little bit in recreating that material because it’s just not as simple as putting parts A and B together and mixing it up. There’s a lot of nuances that they had to re-perfect to get it to work right.

Ross-Nazzal: How funny. I can imagine. I would also think that materials changed just slightly over time.

Madden: Right, and there’s materials that get outlawed. They did have to make some substitutes due to, I forget what laws, but some of the materials you just can’t use anymore in the industry.

Ross-Nazzal: Do you guys keep track of how many tests you guys have done over the years at the Arc Jet facility?

Madden: Yes. We probably ought to take the time one day to go count them. We test about 200-300 a year, has been our clip recently. I think one year we did 400-and-something.

Ross-Nazzal: How many people work at the facility?

Madden: Now, there’s zero.

Ross-Nazzal: How many people worked at the facility?

Madden: It’s not super clean, but on order of 20. For a while, we were running two shifts during Return to Flight. Before Columbia, we’re down to bare bones, and then we had to build back up. That process wasn’t immediate—it was a good three, four years before we really felt a lot of confidence, and we’re blowing and going and had some good inertia and some continuity. For our first few years, we just clunked along; things would break. Our PM [Project Management] plan wasn’t very good, and we’re just basically inefficient. It took a long time to get into a mode where we’re really humming along nicely. Probably two years ago, we’re doing great. Just real good reliability, we’d be turning that Arc on any time we wanted, and morale was really high. Guys were doing real good. The shut-down process killed that. That was another thing that was disappointing. It’s like, “Man, we got this place running how we always wanted, it’s really doing the best it’s ever done, and here we are, shutting it down.” That was tough to take.

Ross-Nazzal: I can imagine. What are your memories of that last test? Were you there?

Madden: No, I missed the very last test. I was with my father, he’s been sick, so I missed the last one. Matt and some of the long-time customers came and thanked the team for their dedication. I had written a little note that I had Ron read for me, but I wasn’t able to be there. Then, that afternoon, they tested a test for the Safety Office, testing a system that if you do have a breach in your heat shield and you’re able to turn on some sort of fluid or compressed gas to pressurize the compartment between the pressure vessel and the heat shield—there’s two different pieces of structure, usually, in the capsule like that. There’s a pressure vessel where the crew is, and then outside that is a heat shield. If you’re able to pressurize that, you can actually make air—let’s just say it’s nitrogen—if the pressure’s great enough, it would actually flow out of the breach, so you have cold gas flowing out rather than hot gas coming in to your spacecraft. You wouldn’t need to repair it—you’ve just got this air flowing out. No hot air is coming in, and it’s entirely feasible because, like I said, the heating usually is at a high altitude and the pressure’s pretty low. You really don’t need that much mass flow or pressure to get the flow to go out. They built a little rig to test the system like that, and that was the last test we did. It worked out great, evidently.

Ross-Nazzal: A bittersweet moment.

Madden: Yes.

Ross-Nazzal: Would you talk a little bit about the camaraderie out there at the facility?

Madden: Yes, I always liked going out there because my seat wasn’t there. Just a real good atmosphere, everybody’s super friendly and always willing to do whatever it took to get things to happen. They share food all the time and they have a Christmas dinner every year, so it’s a really good group.

Ross-Nazzal: I saw you guys have a logo or a patch, when we were out there, doing some of the taping.

Madden: When I became deputy branch chief, that was one of my assignments, to be lab manager. I got—it was Lockheed, at the time—a Lockheed guy to design us a patch, so we made a little patch and went out and bought some shirts. It ended up being a pretty cool patch.

Ross-Nazzal: Would you tell us about the patch?

Madden: It had the name on the outside, like a mission patch, a little icon for a shockwave, and a little lightning bolt arc heating it up. It said, “JSC Arc Jet” on it.

Ross-Nazzal: Earlier, you had mentioned one of the tests you did was for Boeing. Was that for their commercial space venture?

Madden: Right.

Ross-Nazzal: Can you talk about that?

Madden: This country’s developing four re-entry spacecraft right now. We have MCPV, and then Boeing, Sierra Nevada, and SpaceX, vying to be the next Low-Earth Orbit guys. Right now, there’s four, and those guys need to test. That’s one thing I’ve been trying to point out. Boeing approached us to do testing for their vehicle. They’re going through different materials in their selection, and they have an in-house material they’re trying to see if they are confident enough in. They approached us to do some testing, and we signed the Space Act. We have a similar one with Sierra Nevada to do radiant testing. Boeing was real anxious to get in. In fact, we delayed, if you will, our closing so that we can meet that commitment because we had already signed the Space Act. It was basically the last set of tests we did before that one for the Safety Office.

Ross-Nazzal: Did you do any testing for SpaceX?

Madden: No, did not. They’ve tested a little bit at Ames, but we’ve never tested for SpaceX. Their heat shields are doing good, so you got to hand it to them.

Ross-Nazzal: Have you done any work for any universities, perhaps, or other Centers out at the Arc Jet?

Madden: Yes, we’ve done work for other Centers. We did a little bit of things for the Air Force, Air Force Research Lab. I’m trying to think. I can’t recall too much else. Our bread and butter has been the big programs: Apollo, Shuttle, X-38, MPCV. The other interesting thing is, Shuttle—we were testing up until two weeks before the last mission. Arc Jet testing isn’t just for developing the material. We found that we tested every year for the Shuttle in its existence, through development, through qualification, certification, first flight, last flight. Through the whole life of the program, the Shuttle, at least, required Arc Jet testing. It’s a very big player, the whole time.

Ross-Nazzal: You’ve talked several times about the CO2 landing for Mars. Why’d did you come up with that concept? What drove you to come up with the idea?

Madden: We just thought, “We can do it,” and the guys did it in their spare time, on the side. Manned mission to Mars and big missions to Mars has been talked back and forth forever. Sometimes, our Agency says, “Hey, don’t worry about it; don’t talk about it,” the other time it says, “Hey, that’s what you should be thinking.” We just felt if we attained the capability, we would probably have a customer some day to use it.

Ross-Nazzal: You’ve shared that with Ames, since they’ll be the only Arc Jet Facility?

Madden: They haven’t developed it yet, but they intend to, by the end of the fiscal year, develop that capability.

Ross-Nazzal: Tell me a little bit about the deactivation process that the Arc Jet Facility has undergone since the decision was made to close the facility.

Madden: The process has been long, long in terms of actually closing. There’s basically an order given, “You need to shut down,” and our Center Director at the time, [Michael L.] Coats, wrote back a letter, said, “Okay, we’re going to shut down, but we’ve got to do these things first. Ames needs to develop the capabilities that we have that they don’t have, so the Agency doesn’t lose those capabilities.” We had a list of tests we needed to do, and Ames had to develop these capabilities. At first, each of them were projected to be six months away, and then we both kept marching in time. They really marched together, coincidentally. So it wasn’t until probably a year after that, [that] we originally intended to close, before we finally finished up what we needed to finish up. Ames has come real close to finishing their capabilities, so we did our last test. That shutdown process has been basically: drain the systems, de-energize everything, safe it out, and walk away.

We got a set of contractors running it because nobody’s giving us any money to do any sort of real tearing it apart. Our guys did the last test, we spent two weeks turning off things and de-energizing it, then Ames came out last week and pulled all the Arc Heater parts that they felt they could use, packed them up, and shipped them out to Ames. That’s where we’re at now. We got a building full of Arc Jetty type pieces of equipment and pumps and valves and tools and copper tubing and all sorts of crazy things, that we’re going to have to figure out a way to disposition. The shutdown was simple; it was turn everything off and leave.

Ross-Nazzal: It’s slated to be demolished sometime this year?

Madden: No, 2017.

Ross-Nazzal: I hadn’t realized it was going to sit for a while. That seems like a shame, to sit.

Madden: Yes.

Ross-Nazzal: I wanted to ask sort of an, I don’t know, emotional question: what do you think that the Arc Jet Facility has meant to you, since you worked there for so many years on different projects?

Madden: It means a lot to me, and it means a lot to the guys because they want to do a good job. We want to build thermal protection systems that are going to work, and we lost a ship during the re-entry. It’s very frustrating for my guys and for me, because all we want to do is do a good job. We want to do it right. You got to have Arc Jets to do it right. You cannot analytically predict how a material, if you don’t know anything about it, is going to behave in an environment like a re-entry environment. There’s no codes that do that. We have codes that will predict a known material, but that’s after a long test campaign in an Arc Jet. The analogy I give, it’s like trying to do a stress analysis or a structural analysis without the stiffness matrix—which is what the material is—and without the strength. You can’t do the analysis. We can’t do this predictive analysis without testing. Testing is all there is.

We’re real disappointed and just didn’t understand it. It’s been a struggle because there’s a lot of love for the facility. We think it’s important. That’s why there’s been a lot of emotion in that. We’re sitting here, saying, “Hey, you trusted us to be the engineers for these systems and we’re telling you we need this facility,” and [we’re] largely ignored. We’re like, “What are we going to do?” Their answer is, “We’ll test everything at Ames.” It’s been argued that that’s feasible, but you are going to miss some throughput. More importantly, you’re going to miss—it wouldn’t be fair to call it healthy competition all the time, but just a second set of eyes, the diversity of having another group do this. You have a risk of a monoculture and not being challenged, in terms of the way to do things, with the single facility. You also have a reliability risk. Like I said, take a race car versus regular car, take a re-entry spacecraft versus a bicycle. The energy density in the facility is tremendous, and the reliability is poor. They can lose their boiler, they can lose their steam injector system, and then the TPS testing is gone, or it’ll be suspended for a long time. That’s where a lot of emotion has come in.

Ross-Nazzal: What do you think has been the significance of the Arc Jet facility here at JSC?

Madden: I think it’s been a key player in all of our spaceships that we’ve produced. It’s got to be up there.

Ross-Nazzal: Do you think there’s anything else that you would like to talk about, about the Arc Jet Facility? I think we’ve talked about most of the material that I wanted to discuss.

Madden: I think that’s it.

Ross-Nazzal: I thank you for coming in today and sharing your thoughts with me.

Madden: No problem.

[End of interview]

Return to JSC Oral History Website