NASA Johnson Space Center Oral History Project
Edited Oral History Transcript

Jonathan L. Homan
Interviewed by Jennifer Ross-Nazzal
Houston, Texas – 28 June 2018

Ross-Nazzal: Today is June 28, 2018. This interview with Jonathan Homan is being conducted at the Johnson Space Center for the JSC Oral History Project. The interviewer is Jennifer Ross-Nazzal. Thanks again for taking some time to come over today, especially in the heat. It’s pretty warm out there today.

Homan: You’re very welcome.

Ross-Nazzal: I wanted to start by asking how you became associated with the Jim [James] Webb [Space] Telescope [JWST].

Homan: I’ve been supporting the Johnson Space Center since about 1990, mostly—about 15 years—with Lockheed and Lockheed Martin, but always supporting the Crew and Thermal Systems Division in some way or form. Eventually started working full-time supporting [organizational code] EC4, the Systems Test Branch.

In about 1999 the Next Generation Space Telescope was trying to make decisions on where they would test, and I think they were looking at large facilities. We put together a proposal for the Johnson Space Center, and they decided not to come to the Johnson Space Center.

In 2004, in August, a team from the James Webb Space Telescope—I think it was transitioning names at that time from the Next Generation to James Webb—came down and looked at our facility, gave us kind of a set of requirements. They were having some problems with the facility they decided on, and we put together a proposal for a review team at [NASA] Headquarters [Washington, DC] to look at and make a decision. And pretty much kicked off about that time, [I] started working almost full-time supporting James Webb.

The first years were a lot of just understanding what their testing was going to require, how we would respond with requirements on our facility and the chamber. Officially in 2006 [NASA Administrator] Mike [Michael D.] Griffin signed—I think there was still some back-and-forth of where this test would take place—that [the JWST] will be tested here at the Johnson Space Center. That really kicked off a lot of momentum to start modifying the chamber.

Ross-Nazzal: Would you give us a little more detail on your background? You mentioned that you worked with Crew Systems, but what sort of things were you testing in chambers before? Obviously it wasn’t telescopes.

Homan: I’m a mechanical engineer by degree. With Lockheed I started doing some project management work, a lot more project engineering work. Fortunately, it was great to get over and work in the large facilities, [JSC Building 32] Chambers A and B.

We—through the ’90s—did a lot of preparations for the [International] Space Station, did a lot of testing of what we called the human-rated thermal vacuum testing, where we ran a lot of things in Chamber B primarily, where there was high-risk EVA [extravehicular activity] activities.

My role was typically to improve the facility. We actually expanded the facility for some changes on the EMU [extravehicular mobility unit]—and to expand the astronaut’s work envelope when in the chamber, change the way the EMU was being supported, improve the safety of astronauts getting in and out of the chamber. Of course, whichever EVA we were looking at, we were working with Boeing or one of their contractors to get their hardware into the facility and make test stands so that we could simulate the EVA activity. We weren’t doing training. We were always doing probably certification—really high temperatures, really low temperatures. It wasn’t a training like you would get at the NBL [Neutral Buoyancy Laboratory]. It was more, “Let’s take this hardware to its extremes and see if they can perform the task.” So that’s a lot of my background.

We did of course the thermal systems. Being in the big chambers, I got to do plummet testing, which was the impingement of the [Space] Shuttle docking with the Space Station. We did a lot of testing in both Chamber A and B for that. Thermal systems for the Space Station were all done in Chamber A.

One of the most fun tests we got to do was the Beagle 2 [Mars] lander, which was ESA’s [European Space Agency]. Mostly the British space team had that lander. We provided, in Chamber A, a system that would accelerate them to their—and created a Mars-simulated environment that they would hit and land [on to] see if their landing system would work. That was a lot of fun.

Other things in B, the Wake Shield [Facility], a lot of Shuttle experiments, and the Hubble Space Telescope tools are things I’ve been involved with over the years. Did some more in some other of the facilities in Buildings 7 and 33 as well. Probably the big things in 32 are a lot of the highlights of my career, I would say.

Ross-Nazzal: It’s an impressive facility. You said in 2006 the administrator signed the agreement that they were going to use the SESL [Space Environment Simulation Laboratory]. That is a historic property, it’s one of two National Historic Landmarks that we have here onsite. Were there any concerns about changing out that facility?

Homan: Mary [A.] Halligan was our facility manager and civil servant in EC4, and she really worked with the Texas [Historical Commission] and the [National Park Service]. Essentially we just had to document how things were and show how we were modifying it and why it was good for the future. We didn’t have any issues with, “Hey, you can’t modify the chamber.” They realized the chamber was built for the Apollo service/command module.

A lot of stuff was in there from the original design that hadn’t been used and probably would never be used again. James Webb had a completely new set of requirements from a testing environment standpoint. Apollo was low-Earth orbit and lunar, so it had this really high/low temperature, but its low temperature was probably about 100 degrees Kelvin, where James Webb is looking [into] deep space.

We were only really testing a sunshield simulator and the telescope—which is all facing deep space—and never seeing the Moon, Sun, or the Earth, which are the brightest objects in the sky that would have caused some thermal issues. We had to redesign the facility to be able to get that cold, dark, deep-space environment for them to be able to do their infrared optical testing. Hopefully that answers your question.

Ross-Nazzal: I wanted to just ask that, because I know it’s a very historic property. Would you talk about the changes that were made? Obviously you’ve been working on that for, gosh, [how many years]?

Homan: Sent a note to the program when I changed roles in February and said, “Wow, I think I actually had hit 14 years of work almost exclusively for James Webb.”

Ross-Nazzal: Yes, I didn’t want to do the math there.

Homan: It was close to, yes, 14 years. The chamber was designed for Apollo and low-Earth orbit, which had no extreme low-end temperatures, didn’t have the contamination requirements, had no vibration requirements. Heavily designed to save human lives, but not necessarily heavily designed to protect critical flight hardware.

That makes sense, and now we got a new set of requirements of a deep-space, large object that has to go in there. So we had to start taking all the design requirements [into consideration]: vibration isolation, contamination control, longer-duration testing, and much more extreme cryogenic temperatures.

The floor of the chamber used to rotate. You had solar on what we call the north side of the chamber, and this would radiate a lot of the simulated Sun’s energy on the Apollo spacecraft and tested the thermal systems. The floor rotated back and forth and allowed it to simulate its flight to the Moon and back. It actually had a couple issues. It had seals that used an oil that was not liked by the contamination team. We, at one time, thought about having some optical instruments where some of the hub of the floor was. We had to pull all that rotating equipment out, seal that off.

All the solar penetrations—and I think there was 152 on the north side of the chamber and like 36 on the top of the chamber—were removed. We repurposed the north side of the chamber and turned it into platforms and feedthroughs that were occupied by either our new cryogenic lines, or a lot of them were used by the program for a lot of their data handling and fluid lines. Those were some major modifications.

The chamber also had diffusion pumps, which is a great technology for high vacuum, but it’s an oil-based pump. It tends to have a little bit of backstreaming that was a direct requirement, so we had to remove those. It required a demolition [team] taking those off. These things are 14,000-, 15,000-pound pumps. They’re massive. This is including a large valve system that operates them. We replaced them with gate valves and cryogenic pumps and turbomolecular pumps. Those are what we did on the high-vacuum system.

On the cryogenic system, all that solar stuff was removed. They used to have liquid nitrogen-cooled walls, but those areas didn’t have liquid nitrogen. So we had to go in and fill those walls in, and they’re still significant. One area was about 30 feet by 80 feet that we had to put in paneling, and at the top of the chamber was about a 30-foot-diameter disk.

To give you a perspective on the chamber, the chamber itself is like 65 feet in diameter and about 120 feet tall from top to bottom. The liquid nitrogen shroud is about 55 feet in diameter and about 94 feet from top to bottom. It’s a massive shroud. Even though we’re only replacing sections of them, these sections are probably bigger than a lot of other facilities in the world.

Of course the bigger one was a brand-new helium shroud that went inside the LN [liquid nitrogen] 2 shroud. That actually got us from around 100 Kelvin down to about 20 Kelvin. We actually ran it all the way down to 11 Kelvin, 13-ish type of range on average, just to prove our capability. The James Webb wanted 20, so that’s what we actually ran for the test. That was a brand-new shroud. It’s 45 feet in diameter and around 66, 68 feet tall. That’s where James Webb resided for the test, and that created the really dark, closed-out environment to help it simulate deep space. With that, of course, the chamber had to act as a clean room. Apollo—I’m not sure why—had an actually pretty good contamination control system, but it was a bottom-up HEPA [high-efficiency particulate air] filtration system.

We did lots of studies. Part of our design did airflow studies, and it just never worked, because it would always be depositing [contaminants on the mirrors]. You’re pushing contaminants up, and they eventually fall onto the mirror.

Ross-Nazzal: Not a good thing.

Homan: Everybody else designs clean rooms where the air is flowing down and around, and you’re sucking it out from the sides and bottom. We actually reversed our airflow and put in a brand-new filtration system. And so now the chamber—even though it’s a deep-space chamber as a vacuum chamber—when it’s not in vacuum, we run the air in such a way that it actually meets high-performance clean room standards. It’s not a clean room because the shrouds actually have a high emissivity paint on them, so it doesn’t meet the requirements of the smooth walls and all the things that you have to have to be a clean room. But it is as clean as a clean room, and it maintains itself as clean as a clean room, with some smart personnel procedures and hardware procedures.

Let’s see what else. Of course the clean room—if you’ve been over to 32, for most of its 50-, 60-year history it was a large empty bay between Chamber B and Chamber A. We filled that entire bay up with a clean room, with a gowning area and what we call an equipment lock, which was part of the clean room, but it was also an area that we designed so we could transfer hardware to Chamber B from the rollup door.

It takes advantage of the Apollo design of the building with the truck lock and the rollup doors and the cranes. But, now you open it up, you can do some precleaning and stage it so that when we enter hardware into the clean room we can essentially over-pressurize the clean room, move hardware in at a fairly clean state, and get it extremely clean. Of course, that was a large modification that has been added to the Johnson Space Center.

We talked about the shrouds. With the shrouds we also needed the thermal systems, the systems that actually operate. On the liquid nitrogen system, we completely gutted and redid the process. It used to be what was called a forced flow process. If you’ve driven around the Johnson Space Center and you see those large liquid nitrogen tanks on 32, we would take liquid from there and it would run through systems of pumps, pressurize it, and flow through the shrouds. We would recover some of that and create a loop going that way [demonstrates]. With that pressure, it required a lot of power. The pumps were not overly reliable. It’s not very efficient from a thermodynamic standpoint.

We changed all that to a thermosiphon system. We actually take that liquid from those tanks still. We go through a subcooling heat exchanger, and what we call a recovery heat exchanger where we actually subcool the liquid coming in there. Right now the liquid takes a state where it is in a two-phase, where it can be both liquid and vapor. We subcool it so it would take a rise in temperature to get it to vaporize at all. We do that, and we use pressure to push it up through a thermosiphon.

Actually we just have large tanks at the top of the chamber that were all new, part of this, and feed it into the zones, and it has a natural gravity loop that feeds it. It cut down on our use of liquid nitrogen by about three- to fourfold and removed all the heavy electrical equipment that it used to use. Now it’s all on uninterruptable power. That was part of our safety systems too, which I can talk about a little bit.

The other big modification with the environment was we had to install a new helium refrigerator. The liquid nitrogen gets you down to around 100 K [Kelvin]. You’re just buying the liquid and using its already cold properties to provide the refrigeration.

Helium, it’s so expensive you actually have a refrigerator, kind of like Freon in your home refrigerator. You take the helium, you compress it, you precool it, and then you expand it. You actually have these large turbines that expand it and reduce its temperature and entropy so that you can use its refrigeration properties to pick up the heat load in the chamber. This refrigerator that we added has about a 1,700-horsepower electric motor for its compressor, to give people a size. This isn’t like a small refrigerator. It can draw up to 1.3 megawatts. The nice thing about it, again it’s highly efficient. Typically, during the test we’re running between 350 to 450 kilowatts of electricity. Still a lot, but when you look at your balance of refrigeration at the temperature and the energy input, for a while there we were probably the most efficient cryogenic refrigerator in the world. I think the Department of Energy has created a couple newer ones since then. That had to be installed.

Then we did a lot of modifications to piping all over the chamber, and replaced our emergency generator with a higher output, higher-performing emergency generator. Tied in more systems to the emergency power, which I can talk as we get more into the emergency systems.

That gives you a quick breadth. I know I’m forgetting a few things, but it was a pretty massive Agency-level [effort]. Eventually it became a Construction of Facility-type managed project because the Agency wanted to make sure that if the program had an issue, the Agency wasn’t going to let the facility not complete at least getting back to a functional state.

Ross-Nazzal: It doesn’t sound like much was left. You really gutted things and left just that structure.

Homan: Yes, the structure. Most of the liquid nitrogen piping we were able to repurpose; the massive rough vacuum system was reused. We did refurbish all the pumps as time went on through the James Webb testing, but we didn’t have to replace that system at all. A lot of massive pumps from the ’60s we spent time just improving—spending time refurbishing the equipment so it ran like new and got warranties back on the stuff. We knew if we had an issue we could have it maintained real-time, real fast.

But yes, it really was a massive [project]. When you realize, “Oh wow, we’re cutting all the solar stuff out and we’re cutting all these [systems]”—you’re just creating all these holes in this chamber, and it’s unusable for years until you start getting the new hardware in and start plugging those holes.

Ross-Nazzal: It brings to mind so many questions. First of all, you said this was an Agency-wide effort. Can you talk a little bit about that? Who was involved besides JSC? Obviously it’s a JSC facility.

Homan: The administrator signed for it to come here, but the space telescope is of course managed out of [NASA] Goddard [Space Flight Center, Greenbelt, Maryland]. They’ve been our customer here at the Johnson Space Center, giving us our requirements for all the testing and really even through the C of F [Construction of Facilities] portion they were the direct funders and managers to verify that we were meeting goals.

We actually had some congressional milestones on completing certain parts on time and running our functional tests, verifying the chamber was back to a functional state by late 2012, and we were able to meet all that. I think it was—yes, late 2012. We did that. There were milestones the program was able to check off and take credit for as well.

Over the course of this [NASA] JPL [Jet Propulsion Laboratory, Pasadena, California] was highly involved with one of the instruments. Because James Webb is large and has so many other contractors throughout the country and international partners with ESA, we’ve been able to interface with some of those folks and their requirements and their needs as well. It was pretty neat doing all that.

Ross-Nazzal: You mentioned all these different systems, you mentioned doing studies. Were you traveling around looking at other facilities and seeing what was cutting-edge and what was possible for JSC before you started cutting pipe and thinking about, “Well, is this what we’re going to do?” How did you come up with the plan that you eventually came up with, basically refurbishing the entire facility?

Homan: A little bit of my history—I was involved in the Johnson Space Center procuring some new helium systems and helium refrigeration-type stuff and learned a lot on that. Also, I’d gotten involved with the [International] Cryogenic Engineering Conference. Through that—and I was going to talk about this a little bit later—we eventually formed a collaboration between NASA and the Department of Energy using the cryogenic lab at Jefferson Labs there in Newport News, Virginia. That was tremendous.

I think they were in a little bit [of] a lull, and they had been doing a lot of cutting-edge technology on large cryogenic systems. They were going to help us on the helium refrigeration. When we started having some cost and schedule problems on all the LN2 modifications, which we were using more of a general contractor for, I had asked for some studies. I had some struggles myself with seeing that they were really understanding the problems and taking care of it and eventually took it back to the Department of Energy. They were able to quickly analyze it and come up with some different solutions. Like the helium, we actually pulled it back. We managed most of that in house. NASA maintained most of the technical control and used our engineering support contractor Jacobs [Engineering Group] for a lot of the implementation instead of hiring an outside contractor. That way we were able to have NASA and the DOE [Department of Energy] have overall project management and technical authority, while we were able to use the skills and the resources of Jacobs to really procure a lot of the stuff and create all the subcontracts and integrate. They had a lot more people than just me and somebody else. When you’re doing all these massive things that was really good, that really helped.

We did a lot of studies on what would be the best pumps. A lot of it was driven by what met requirements and—we were buying so much—what kind of pricing we could get, because you’re talking a little bit of pricing change on some of these things, we’re saving millions of dollars. It was essentially just, “Hey, this is a really good technology, may not be the best.” Like our cryoabsorption pumps, we’re using probably standard Gifford-McMahon cycle. There’s some pulse tube technologies that are better for vibration, but their performance long-term—their cost is really high and their reliability is really low. So we were like, “Okay, we can manage the vibration slightly different and save a lot of money, and improve our reliability,” which was very important for us on this modification.

Probably 2006 through 2008 were primarily getting through the PDRs [preliminary design reviews] and CDRs [critical design reviews] at the program and Center levels to say, “Yes, we’re ready to really start letting contracts go and procuring.” We had already started doing the cutting just because the schedule required us to, but we did a lot of studies to try to make sure we were doing the right things.

I would say the helium system and the LN2 system were both major changes in technology, and the helium system was a cutting-edge refrigeration box. That was neat to do something like that, because there’s probably no other refrigerator that looks like ours and acts like ours and has the performance like ours. It’s really great because we’re not limited to doing thermal vacuum testing for just James Webb.

If Orion [Multi-Purpose Crew Vehicle] came here, we could probably meet or exceed any of their thermal requirements. Any commercial crew-type vehicle, any large type of thing. If we wanted to do planetary-type testing, the chamber could do low-Earth orbit, lunar, deep space, planetary. That’s the range that the thermal systems have now that they probably didn’t have before and the flexibility. They’re extremely reliable and really controllable. I don’t think there’s another facility out there that could have the range and performance that Chamber A right now has, especially when you start thinking about, “Oh, it’s got contamination control.” Not that I think people will probably use our vibration isolation system, but we’ve got all these other really neat things that have been left over from James Webb that are more specific for a satellite that requires a stable environment and a contamination-controlled environment. Most of the stuff we use here at the Johnson Space Center isn’t as contamination-sensitive since we’re dealing with human space-type stuff and doesn’t have the stability requirements because you’re not holding optics stable for long periods of time.

Ross-Nazzal: When you were working on those studies and kicking off this project, were you thinking, “We need to be looking beyond the James Webb Telescope. We need to be thinking about the future here at JSC?” Was [JSC] 2.0 on your mind? I guess that was way before 2.0, but thinking that?

Homan: I think because in 2006 Constellation [Program] was so big here at the Johnson Space Center, I couldn’t keep that out of my mind. All I kept thinking is, “We need something either to test a vehicle or to test a planetary hab [habitation] module.” Those were on our mind.

Interestingly enough, because the system is so robust and capable, we used all that to test James Webb. Especially on some of the pathfinder testing, which I know you had a question about testing things like that. If they had done those tests at Goddard, they probably would have been like 200-day tests to be able to achieve the temperatures and perform anything. We did them in 30 days, because we have the ability to take on a higher heat load and have a better thermal, stable environment.

People were going, “We could actually create a Martian atmosphere and test things at really cold.”

I’m like, “We absolutely can.” This chamber is not limited to creating a 20 K deep-space environment for James Webb. If we wanted to do polar simulation of Mars or one of its moons, we could create that in that chamber.

So that was on our mind, to get back to your original question. Yes, I always want to make sure we’re doing things that are flexible and looking to the future. I really thought at the time Orion and Constellation would be doing some type of testing in the facilities post-James Webb, but unfortunately they’re not.

Ross-Nazzal: We’ll see what happens, we’ve still got Orion and some other programs going.

Homan: Yes. They’re actually getting ready to test in another facility.

Ross-Nazzal: Oh, that’s too bad.

Homan: Yes.

Ross-Nazzal: I wanted to just clarify. You mentioned Jacobs was doing this. They were in charge of actually building most of what you put in there?

Homan: We used the engineering contract here at Johnson and Jacobs to really provide the ability to get the contracts let and do the implementation and integration project management for the chamber modification. It was a huge thing for them. For a lot of the systems, myself, and people from the Department of Energy were writing all the requirements for them or reviewing and essentially worked hand-in-hand trying to maintain.

I had a certain amount of budget with some stuff that of course the Jacobs people didn’t see. You’re trying to see their budget and how they’re managing it, and hopefully you’re not using your reserve for their problems just yet. But they did the heavy lifting, I would always say. At times we had 200 people working modifying the chamber. The chamber is filled with scaffolding; the building is filled. We had built a tent behind us, and we put air-conditioning in because we had so many people. We needed a place for people to eat lunch and take breaks, because we had welders and machinists and pipe fabricators.

There were, at times, I would say five or six different contractors working. Cryogenic piping companies that were out of Minnesota, two or three local pipe fitters that were from the Houston area. We had the shroud companies that were from the Boston area that were all working and had very specialized fields, companies from Ohio that really dealt with our vacuum systems and pumps. We had to coordinate that and make sure people weren’t working on top of each other.

We helped with that, but really I think Jacobs was the one who let the contracts out for all the subcontracts. But even I would review the proposals and verify that it met the requirements that I had placed down and out.

Ross-Nazzal: Was there anything made in-house that you used?

Homan: Oh, yes. A couple things, the helium shroud, we let one contract out, and we had some reservations. Quickly that company started coming back with extreme new cost and large schedule delays. So Jacobs, we asked them, and they did—they pulled the contract away from that company, then we split the contract up. Two companies up in the Boston area, and one we’ve used JSC Manufacturing Services. The actual floor was all done there. Then Jacobs, with some other subcontractors managed the final fabrication and installation, and we used JSC riggers. We used JSC riggers for the whole thing. They’re actually under JA [Center Operations Directorate] and not under EA [Engineering Directorate].

We just funded JA and tried to look at contracts and say, “Hey, you’re not going to need rigging services. We can provide that for you. We just need to know what you need. Our riggers need to go out and rent stuff.” That worked; [it was] a very efficient way to do the business. Also I think the rigging team here loved it because we had extreme challenges. They got to work with all different types of people and they eventually got to work quite a bit with the program installing a lot of the GSE [ground support equipment].

Hopefully that answered your question. But yes, we had a team here at Johnson where you had a project manager and one person who was more the electrical, data systems, because we did major modifications of the data systems and instrumentation. Russ [Russell E.] Bachtel, he’s in this building now. John [D.] Speed handled a lot of the pressure systems and the vessel modifications and the structural-type stuff. I did a lot of the vacuum and thermal systems. Helped architect the overall modifications, but then I stayed there, because that’s where I felt like I needed to make sure I had my hand in there. It was one of the bigger [efforts], both cost and overall requirements for the overall modification. Hopefully that answered something.

Ross-Nazzal: Oh, this is all great. When we interview people nowadays one of the big challenges that we have is NASA is really good at promoting social media but there’s not a lot of history out there. You’ll find the same quotes over and over again just being repeated, but you don’t get a lot of the details about what happened. This is really interesting for us.

Homan: When you say about the history, it made me think—when you talked about Tom [Thomas R.] Scorse. Those guys like Gary [W.] Matthews and Tom Scorse, they were probably some of the first people that I met with the guys from Goddard. It was really interesting. Gary probably told you—he was originally with [Eastman] Kodak [Remote Sensing Systems].

Ross-Nazzal: Yes, I think he mentioned it.

Homan: Then they changed badges, now they’re Harris [Corporation]. We would go quite a bit to their facility up in Rochester [New York] to try to understand what they were doing and how that was going to integrate into the chamber, because we had to make sure that we created structure for them to be able to mount their hardware. They needed tie-ins, needed helium refrigeration for something, so we needed to make sure we put in lines and had interface points.

Requirements and defining the interfaces took several years, and even after that it still took a lot more years, because as the test better defined itself—more and more details come out as you start building hardware, and you realize, “Oh, this isn’t as easy to build. We’re going to have to modify something.” They modify that, then we had to modify something in our chamber.

I think back about that—we definitely worked probably mostly with the team in Rochester and the team at Goddard to really define the requirements on the chamber. Since the guys at Rochester were doing the GSE for testing the telescope, we had to work with them to integrate that into the facility.

Ross-Nazzal: Were there major changes that you can remember as a result of those meetings?

Homan: I feel like I was heavily involved in putting together the project plans. You put together project plans, and you present it to the management at Goddard and the program. It’s always too expensive, so you try to figure out what’s palatable.

I think of our helium system. I think originally we settled on 14 zones, with 7 zones being dedicated to running the shrouds and 7 zones being dedicated to running GSE thermal systems. We had proposed doing more. I think we ended up putting in two spares because I was nervous about not having enough spare zones. We ended up building probably about two or three more zones past the spares.

I try to tell people, “Wow, doing this post, it becomes more expensive and more challenging,” because it’s not integrated. Now we’ve got systems clean, and we’re trying to do welding in a clean chamber. So we had to add some new zones, that was a challenge.

We had to come up with new safety systems. We had proposed putting the roughing system on emergency power at the beginning, again from a cost standpoint. Understandably—I think it actually worked out very well for the program from a cost and schedule standpoint, so I thought it was good project management decisions—we were told no. Then, as we completed most of the stuff, I think that risk actually got elevated all the way to the top of the Agency level. If we have a hurricane, not having a roughing system—we had all these other things backed up on emergency power, but not being able to either get the chamber down to a rough vacuum or handle a large off-gassing event—like when you first warm up, sometimes those shrouds are so cold they’ve got liquefied air on them. The pressure gets so high you need a larger vacuum system; they require more power. So it could thermally cause problems for the spacecraft.

We eventually installed that, and that was a big last-minute change. Significant amount of new dollars to the Johnson Space Center to be able to install a system. We did it with a temporary generator, but it was new transfer switch. So if we ever have another high-risk test, all we have to do is rent a generator for that system to be ready again. We’re not maintaining a large generator, because there’s some rules about that and it takes up footprint on the Center as well.

The backup systems we talked about. The clean room, same thing. We had suggested having a little bit more robust emergency system on the clean room, and at the time I think everything was cost-controlled. We built an excellent clean room that was very sensitive to any changes in the utilities here at the Johnson Space Center, and that created some problems. When the Center would have, “Oh, we got water issues in this building” or something like that, we could see it affecting the clean room. It was extremely sensitive to any changes in the chilled water system coming from the large utilities building at the Space Center.

We ended up putting in an entire backup system after the fact. It was rental equipment, and we rented it for a period of years. Which was really great, because it got upgraded through the time. I think from a cost standpoint and schedule standpoint it probably actually worked out better than we had [thought]. It was also a redundant system, so if we had an emergency like a hurricane, we had a full-up backup system.

Those are a few things. There’s probably other major modifications we’ve done over the years, as I think about it.

Ross-Nazzal: You mentioned budget being an issue, and during that time we had sequestration. Then—I think it was in 2013 when Ellen [Ochoa] became Center Director—there was a significant cut midway through that fiscal year. Did any of that impact what you were working on?

Homan: No, once we got established—I would say our budgets in the first years were always iffy. Once the Agency tried to make a C of F—the dollars to really make the modifications, primarily for the chamber—we took some of the emergency systems off the table to focus on just the main requirements getting the chamber ready, and then used program funds to put these things in after the fact. But those dollars were pretty well-secured for us.

The James Webb funding didn’t seem to be as affected by some of the other funding that we had seen. We always tended to have enough money to make our requirements. Once we started the testing, it seemed like there was never an issue. We started the facility testing in ’12 and ’13, and then ’14 and ’15 we did mostly chamber and GSE commissioning testing. Then ’15 and ’16 we did the pathfinder testing where we actually had the engineering unit, the James Webb, here at the Johnson Space Center for a good two-plus years. Integrated that and did a series of smaller tests that all were components of the large flight test, and then of course the flight test was in 2017.

Ross-Nazzal: That’s amazing. Unless you have anything more to talk about—about the mods [modifications], I know you had mentioned safety and contamination control. I don’t know if there’s anything we didn’t hit on there, but I thought we might turn to testing. I had no idea the testing was so lengthy and so elaborate. I don’t know if you want to walk through those various years that you talked about, 2012 through ’13, and the iterations and what had to be done.

Homan: Sure. Like I said, in ’12 the clean room wasn’t even built, but we had to prove the chamber was back to being a functional state. And we did. We ran a test in, I think, August and September and exceeded all our requirements except for the chamber leak rate. We had removed 200-some penetrations, cut a lot of holes, and patched everything back up. So we found where the leaks were, which was part of the test to go find out what needs to be retightened up.

The next test in 2013 in June we were able to really correct a lot of the leak rate. We were able to at least test the systems, “Is this thermosiphon system going to work?” There were people who were doubting it. We were like, “It ought to work, the physics tells us it’s going to work.” I understand why there was a lot of doubt. Of course that makes you nervous when there were some academia folks who were telling us, “Don’t do it,” and it wasn’t going to work. We were able to show that it actually did work. The helium system—we had just finished installing it, commissioning the piece parts. “Does it work as a system and integrate with the shroud to control everything?” And it did; it worked extremely well. So it verified all our design work upfront, and thoughts upfront actually did come together.

We still had bugs to work out. We didn’t have all our procedures written. You’re learning and writing procedures as you’re going. You wanted to have pretty well-defined procedures by the time you got to the flight test. Really it was an iterative process.

The emergency systems—we actually had the Center, over a weekend, kill all the feeds that went into Building 32 and simulate a major power disruption like a hurricane. Or like what we had when the guy ran into the [utility] pole and took out both feeds when the thing went down. I’m not sure when that was, but whatever year that happened. I think it was like 2010.

Interesting story there, because we were actually talking with the program management saying, “We’ve never had a serious power loss.” And then the lights went out. You’re like, “That is so—” You jinxed yourself, type of thing. Then it was out for days until they got that feed fixed.

Ross-Nazzal: That’s funny.

Homan: Yes. It wasn’t, but it was.

Ross-Nazzal: Looking back it probably is, not at the time.

Homan: Which was good. In some ways it let us find some other issues around the facility when it happened that we realized, “Wow, when the power goes down, you lose air.” When you lose air, you lose control to your control valves, you lose some of your water flow to things that you need. It had a domino effect. It really helped us, at the last minute, look at some things and solve some problems before we got into our first test.

In ’12 we did the major power outage, and everything worked extremely well. We did find some areas didn’t come on. Mostly it was more like a computer we thought was on either uninterruptable or emergency power. Some of the circuits just weren’t labeled right or something like that. It was mostly minor things. We did that in ’12, and repeated the test again in ’13 after we learned our lessons and tried to improve on that. So that let us know the basic chamber performance was there.

Of course then in ’13 we started with the clean room construction and also a lot of the major GSE integration. That’s when, “Okay, now that the chamber is there, let’s start integrating the hardware from Harris.” We had the CoCOA. Probably Gary Matthews talked about that, the Center of Curvature Optical Assembly.

Ross-Nazzal: He mentioned that, yes.

Homan: The USF [Upper Support Frame], the vibration isolation systems were installed. The systems that connected the vibration isolation system down to the HOSS [Hardpoint/Offloader Support Structure]. The neat thing about the HOSS—that was a huge piece of GSE designed by the guys up in Rochester, but a lot of their hardware was built here at the Johnson Space Center using either Building 10 or local contractors to help assemble it.

As we integrated that hardware, we did tests for both chamber performance and integration of the GSE to prove that the hardware that was supposed to test the telescope would work, and that was what we focused on mostly through ’14. One of those two tests were structural, where we cryo-proof tested everything. We took everything through really hard thermal extremes to see if any of the weld joints broke on anything that was in the critical path of the telescope.

We also baked the chamber out. We ran the chamber for two weeks at elevated temperatures and just collected everything that came off of anything that was in the chamber. It’s amazing how much stuff will off gas at these elevated temperatures and pressures. That was a great way for us to really drop the contamination levels at vacuum. Because when you’re not at a vacuum you have to really worry about fingerprint moving and depositing stuff on the mirror or oils moving. When you bake it out, all that stuff is driven off. Then we had scavenger panels that we would actually collect, do an analysis on, and clean that off. That way you drove everything off. You could watch it. You could watch the molecular off-gassing peak, and then when it starts to tail off you know that most everything that’s volatile has been driven off in the chamber. Then it tends to stay clean.

We did that test, then the last big test we did in ’14 was a long test called the chamber and GSE commissioning. The first ones were more of cryo-proofload and bakeout. The last one was about a two-month test where we actually really verified that we could start doing testing, all the test equipment worked, and the chamber could do any of the profiles that they had asked us to do. And the GSE could of course do all it was supposed to do, with the distance measurement with all the lasers and trackers and neat stuff that we had in there to monitor what was going on with the telescope.

Soon after that, the pathfinder unit came to the Johnson Space Center. That was an engineering unit of the telescope. It came with a flight-spare secondary mirror and three primary mirrors. The James Webb has 18, it only had 3. Only one of the three was actually gold-coated, but they were all polished. There’s three different prescriptions on James Webb, so I think each one was one of the three different prescriptions.

I think we ended up doing, including the flight test, 11 vacuum tests on James Webb. I didn’t think about this, but when you were writing this—2015, if you remember, the Memorial Day floods.

Ross-Nazzal: Oh, yes.

Homan: Right before the Memorial Day floods that affected downtown Houston, about a week or so before, we had some pretty severe weather in Clear Lake, and it blew one of our main transformers out. We had a lightning strike actually hit something on the Center that blew out a transformer that was required for the test. Our Center Operations did a great job of bringing in an emergency generator, and we were able to get right into the testing. We had a couple other incidents show up. We were doing some cryogenic runs, and we had lost a little bit of control on some things. It showed some vulnerabilities and just made [us] more aware of the potential [risks].

If the telescope is in the clean room and we have a hurricane, it’s a whole lot safer in the chamber. And it’d be a whole lot safer in the chamber under vacuum, because now you don’t have the environment changing around it. You don’t have the potential of air carrying contaminants or anything like that. So that drove the, “Hey, you really need to put your roughing system on emergency power.” That was a last-minute, high-dollar, schedule-driven task that we ended up having to do.

It was a good thing that we ended up having that. We had people around the clock at that time, and it was probably around midnight when that lightning strike hit and all these domino effect of little things happened. You track those lessons learned, and we came up with solutions and implemented those. It was good that that happened in ’15, and we were able to get all the solutions implemented before the flight test of ’17. That was a blessing in disguise in some ways. I think even the administrator found out about some of the problems we had during that ’15 lightning strike. So it gave a little push to say, “Hey, now we know what the problems are. Go ahead and come up with a solution and implement it.”

Ross-Nazzal: You mentioned that academics were a little, “You shouldn’t do what you’re about to do.” The National Academy [of Sciences] or the National Research Council, were they monitoring what you were doing?

Homan: No, when we had decided to move away from just refurbishing the liquid nitrogen system—using the same process, and just improving all the equipment—we realized it was just going to really cost a lot. The system they designed for Apollo was overly complex, not as safe as it should have probably been. It was pretty safe for handling the astronauts, not safe for the people operating it. A lot of new rules have come out from the ’60s to the 2000s in terms of how you handle cryogenics and things like that.

Ross-Nazzal: With OSHA [Occupational Safety and Health Administration]?

Homan: We had to implement a lot of changes, and we decided, “Hey, you know what. We can get away with a lot safer, a lot more reliable, a lot less expensive system, changing the overall thermodynamic process from a forced system pressurized to a thermosiphon.”

But the chamber wasn’t designed [for that]. Most chambers that are designed for thermosiphon are designed in such a way that you have real smooth piping and everything goes straight up and down. Because we had all these different loops—and I’m not sure why they did this, maybe they wanted more thermal control, or more thermal variation when they were designing Apollo testing. I don’t think they implemented it, looking back at the history, but I think maybe at the time they overengineered things and then realized they didn’t need everything they designed. We were stuck with an overly complex system. People were like, “With all the bends in the tubing, all the loops, there’s a potential that this won’t work.”

Because of our involvement with the Cryogenic Engineering Conference, there were some folks that had been there, heard that we were making changes, and were voicing some strong concerns. “Hey, I’m a professor at such and such tier-one university.” We actually wrote a paper and presented, and they were on our side as we were doing the implementation. When it worked, they actually gave us some praise. I think they look back and go, “That’s the right thing to do.”

We even said that it would be great if we had the money to transform the other large chamber here at the Center, Chamber B, to a similar type of thermodynamic approach. It just increases the reliability, saves a lot of energy, and much easier from a control system standpoint, too. But it does require a fairly massive overhaul of some of the infrastructure around the chamber.

Ross-Nazzal: You mentioned history. Did you go back and look at some of those building documents or some of the history of the chamber itself as you were working?

Homan: Oh, yes. That was one of the first things we did was grab all the volumes. We had them electronically scanned in. Put them on different databases, so we were able to go back and really find out what they did, how they designed things, how they figured out the heat load back then, and why they designed certain things the way they did.

Unfortunately, the chamber was never built with a 3D [three-dimensional] model. That was part of the whole chamber modifications. We spent a lot of money going back and taking all those drawings and creating a high-fidelity 3D model that we could hand over to the program, use with our new designs, and do our airflow, dynamic analysis, vibration analysis, thermal analysis.

You had to go through all those old documents and understand how they did everything and how things were built. They had a lot of simple drawings—a significant amount—but a lot of the details were missing. We had to go in and try to really find those details and build a highly accurate model at some point—that we now have—and were able to show the chamber and the telescope.

So we could integrate and say, “Oh, how is it going to perform when the air is moving at a certain velocity? Is it going to stay clean? How many people can we put around? Oh, how are things going to work thermally? How well do the mirrors react when the shrouds cool?” We did a lot of that analysis and worked with the program in integrating it.

Ross-Nazzal: Where did you do that sort of analysis? Obviously it sounds like you were using computers. Were you doing that here or Harris?

Homan: EC did most of the thermal analysis on the chamber, a lot of it was done using Jacobs. The vibration analysis was mostly done at Goddard, and Harris did a lot of vibration analysis. A lot of thermal analysis we were passing back and forth.

I should say the thermal analysis for the chamber systems was all done here at Johnson, and we would collaborate the models between Goddard and Harris. And a little bit with Northrop Grumman [Corp.], because Northrop maintained a higher-fidelity flight article model. It was quite overly large and cumbersome to really get results from. It was good sometimes, once we understood here, to let them run the higher-fidelity one to see if it had any minor changes on the details of the telescope.

So it was a group effort that we just sometimes had to meet together as a thermal team. The Goddard, the Johnson, and the Harris [people] would all get together and pass STEP [Standard for the Exchange of Product model data] files back and forth and see who was working on what, so that we had the right answers going into a test or even going into design sometimes.

Ross-Nazzal: That’s interesting. Totally different from what the guys would do in Apollo. You basically had to do a test during Apollo, but here you could run through things beforehand to get an idea of what might happen, correct?

Homan: Oh yes, when you think about the flight test, the flight test was really to prove out all the optical requirements. And to prove out all the thermal requirements, since it’s going to be sitting there looking at deep space and running at cryogenic temperatures. It was really critical that we understood what we expected it to do and then ran the test.

Whenever there was a variance, a flag would come up. [We would] try to understand, “What’s causing this variance,” and “Oh, okay.” We didn’t have anything, I think, major on the thermal systems. We did have one on the optical during the flight test, that was one of the reasons why the test lasted quite a bit longer. It ended up just being what we would call an “offloading point.” Something that tried to simulate the mirrors at zero g [gravity] was actually interfacing sometimes, so it was actually deflecting a mirror at certain times and temperatures.

You would notice, “Hey, every so many minutes”—when I say the mirrors are deflecting, it’s probably deflecting a thousandth of a millimeter, very small. But when we had it on mapping, it looks very distinctive. “Oh, the mirror is almost bubbling,” and then coming down. That large bubble on the computer graphics you would never be able to see with your eye, but it was enough that it was taking it out of range of its allowable focal point.

It was just an interference. We had two interferences. One was a bolt holding a piece of thermal insulation [that] started reacting with one of the mirrors and didn’t let it have the freedom it needed to move quite right. Another was this gravity offloader. Once we figured out that’s what it was, Building 10 made the modifications and we were able to run a quick little simulation and show it went away. So that was the problem. It had nothing to do with the flight design, it was all the test support hardware. It looked like it was a flight design issue when we first discovered it, like, “Oh no, what is going on.” Because it looked like this one flight heater would come on and it would affect the mirrors. It turned out that it was the flight heater that was affecting a gravity offloading bolt. When you change the temperature on something, it just expands a tiny [bit] and that was all it took for these things that are measuring nanometers to see that things were off.

Ross-Nazzal: That’s pretty impressive. Do you want to talk more about the pathfinder test?

Homan: The first two pathfinder tests were optical. It was called OGSE -1 and 2, Optical GSE 1 and 2. The first one was just really to prove out, “Would the CoCOA work and the photogrammetry system work with it?” It was, “Can we phase primary mirrors?”

If you’re familiar with James Webb, it’s got 18 segments. When they launch, they’re on the launch lock and they’re pointed, so they’re not in focal position when they deploy. They actually have to deploy and then deploy again. There’s an algorithm called the “wavefront sensing and control” that helps them all look at something in space, like a single star, and work together to focus as a monolith.

That was of course a huge thing for us to test in the chamber, and the first thing we did in pathfinder was to see, “Does that work with the pathfinder unit?” The CoCOA is an interferometer. I’m not sure if Gary told you a lot about it, but it sends a laser light, bounces it off the mirror, and then it comes back. It’s shifted just a little bit, and you see a fringe pattern on the mirrors. If you don’t see much, if it just looks something grayed out or blacked out, you know you’re not in focus. When you start getting nice zebra patterns you know that it’s actually reflecting light the right way, because you’re interfering with yourself. We were able to prove that one out in the test.

The second one was a much more complex test called the OGSE-2. We actually got the flight AOS, which was the very center portion of the telescope. It was called the Aft Optics System. It looks like a big black cone if you ever saw the telescope; it was a large black cone with a shutter on the top. The primary mirror focuses everything to the secondary mirror and then that focuses it through this Aft Optics System. It has a flight tertiary mirror, a fine guidance steering mirror, and the pick-off mirror.

It’s sending information to the science instruments. We didn’t have the science instruments of course, because it was an engineering unit. But we had a thing called the Beam Image Analyzer we installed that had a little positioning system that could move to each of the light-gathering locations of each of the science instruments. We could verify that we could bounce light, and it found the path and would be sensed correctly at the science instrument that we were expecting it to work.

When you think about it, our test light bounces off the primary mirror—oh, no. We bounce it off the secondary mirror. It bounces off the primary mirror, goes up to a series of autocollimating flats—which are just super flat, controlled mirrors that act as infinity of space—and then reflect it back down. So now it’s passing back off the primary mirror, secondary mirror, into the Aft Optics System, hitting all those other mirrors, and hopefully going to one of the science instruments.

You’ve got to remember, the Aft Optics System and everything, this is all in the chamber. If there’s any vibration and you’re looking at wavelengths of light, it could ruin your test. Everything had to be extremely stable, extremely precise. That’s what that second test was, and it really proved out that we could do the more complex optical testing, like the pass-and-a-half, and really prove out our photogrammetry system.

I think we may have had the Digital Measurement System in there as well, which really was great too. It ended up being a fantastic tool. It’s just a laser that would look at different things. You could watch small vibrations because it was giving high hertz [frequency unit], a lot of recordings over a second, of positions of where the mirrors are, and if anything was changing on the shape of the mirror. It was a good test.

The last pathfinder test was called the thermal pathfinder. We actually spent a lot of time putting in flight simulator blankets for all the center section. We didn’t do the wings, but we did the center section of the primary mirror. Put in all very high-fidelity, not flight blankets, but flight-like blankets all around, and tested mostly how the thermal performance of the spacecraft would work. Were we going to get the results that we expected, and could they build a thermal model that would work?

We knew the differences, because it wasn’t flight-like, but it was well understood. “Can we model this and understand it?” Then when we go into the flight test, we expect the flight model to correlate well with what we actually see in the test. That was the last longer test of probably 2016.

We probably spent close to a year just getting ready for that thermal test, because it had a lot more integration to get ready for it. The other ones, we had an open architecture. We just needed the mirrors in place. We still had to get the temperature, but we didn’t need all the flight-like arrangement of everything else that the thermal guys were really worried about.

That was it, and then we of course sent the pathfinder back to Goddard. The Aft Optics System was removed after the OGSE-2 test, and we flew that back to Goddard. Just a simulator was installed for the thermal pathfinder.

Then of course in May of 2017—that was last year—the beginning of May is when the actual flight unit arrived at the Johnson Space Center. We prepared it for test, were ready for test in—I think we started the test the second week of July, and we didn’t open the door and roll out until the weekend before Thanksgiving. It was a long, long test.

Ross-Nazzal: Yes, long test. I did want to ask you. Several times you’ve mentioned power issues. We’ve interviewed folks who work up at what used to be [NASA] Lewis [Research Center], now Glenn [Research Center, Cleveland, Ohio]. They mentioned how sometimes they’d have to reach out to the power companies and say, “We’re going to run this test,” and they would have to provide additional power to that area. Was that ever an issue?

Homan: Good thing is we have the two feeds on the Center. We do tell our Center Operations team. We call them, we say, “Hey, we’re getting ready to start our large vacuum pumps,” because they do draw a lot of power. Our helium refrigerator—when it starts, it draws a lot of power. So we let them know that that is going to happen.

One of the nice things, back in the design phase of all this, is we worked with Center Operations because we had to modify parts of the building and needed new power feeds coming in for hardware. The new refrigerator—they actually gave us a separate feed that was only used for the arc jet facility and the large acoustic facility. They weren’t used that often, and they knew the arc jet was shutting down, so we had an independent large feed for the refrigerator so that it wasn’t dependent on the rest of the building power and provided some redundancy that way. We never had a problem. If anything, we probably, for the size of the equipment, used quite a bit less electricity—when you look at what our refrigeration loads were and what we were trying to accomplish in terms of creating a deep-space environment—compared to a lot of other facilities. It’s a lot more efficient and a lot more reliable because of the design that went into it.

We did tie a lot of things in to the emergency power. We had to improve our emergency generator. Then once we knew we wanted to have backup air systems, we were able to come up with a diesel unit. It didn’t require electricity, but if the Center air system went down, it picked up immediately. We had additional emergency generators for our roughing system and for all the environmental control during ambient-type operations in the clean room or the chamber.

But we didn’t need additional power. If anything, I think when you look at how it used to run, we actually have more low-end refrigeration capability and we consume a lot less power than what it used to consume. A lot of it is just the design of electrical motors has become more efficient.

Ross-Nazzal: I know we’ll talk about [Hurricane] Harvey later, but was that ever a concern for people at Harris and Goddard, bringing this piece of flight equipment down to an area so prone to hurricanes and flooding?

Homan: I think it was always a concern from the very beginning. I think it was concern with the administrators as well. It seemed like we’d always plan our tests to happen in the winter, and they’d always slide into the summer.

When we designed the original modification to the chamber, we did ask them. We said, “What is our safe state?” Our safe state that we defined was around 80 to 100 Kelvin for 10 days at high vacuum. That was what we tested with that first testing in 2012 and ’13. “Hey, if we kill all the power, what state do we leave the chamber in? Do we have enough emergency power to make sure that it stays there? Do we have enough fuel storage and cryogens to last out the time required?”

That was the going-in state. Of course, as you got closer to the flight hardware, requirements changed. They wanted more time, more sensitivity. That was one of the reasons why we added more and more things. And that’s why we ended up—if you ever went behind 32 back in 2016 and ’17, we had two or three large diesel tanks. We bought a lot of diesel and had it stored so that if we had to run the generators they could run for probably weeks at a time without needing to be refilled.

There was the chance that maybe we all had to evacuate. Worst case, for safety reasons, we had to walk away. I think we did some analysis where this would allow the telescope to stay there and warm up on its own over a slow period of time. Then eventually if we drained all our diesel—or if, God forbid, we were actually gone for a month or something like that—the telescope would have safely gone through its transitions without damaging anything or contaminating itself.

That was always on the mind of everyone from the very beginning. A year or two before each of the tests it seemed like we would have the test scheduled to start like, “Oh, start it in November and run it through March or April.” It would always slide to June, July, several times August, September. Like I said, the flight test started in July and went through the middle of November, right through the heart [of storm season].

We’d shown the charts to everyone. Linda [M.] Spuler with the [JSC] Emergency Management, she gave really great presentations that we asked her to give to Goddard and all their contractors to prepare them. “What is it like for a hurricane?” You see, late August and through September is the peak season. So that was always in the minds [of our folks], and I think we did a really good job of preparing for it.

From a personal standpoint, we signed agreements with the Center management—especially the heads of engineering and JA—to make sure that we had the ability to ride the tests out with a limited crew. Not everybody could stay, but people who passed certain physicals and could handle certain stress levels could possibly stay. We were in that agreement, always under the authority of the Center commander. So if they told us, “Hey, it’s too dangerous,” we would pack up and leave with them. We never had to experience that, but we had those things in place. We had stuff in the building in place. We actually provided food, water, bedding, and beds—all in lockers, so that when we had a situation we could always have emergency supplies. Of course flashlights.

We actually rented a port-a-potty that we put in a shelter. It was only for emergency case, but we had all the kind of things like, “Hey, what do you do if we lose air-conditioning, water running, and general electricity?” We know we’ve got our electricity, and we have our backup air and water systems. That helps the chamber run, but it really doesn’t help go to the bathroom. That was one of the things that they talked about. “If the water is down, please don’t use the restrooms, because it’ll be really bad really fast.” We were like, “Well, we got to think about all this stuff and make preparations,” and so we did. Some of them came into use for Harvey.

Ross-Nazzal: I’m sure. Was all of that in place when you started doing testing, starting 2012? Or was that later?

Homan: No, no. In ’12, of course the risk—because we had no real delicate hardware in the chamber—was low. We were pretty much just trying to prove that our emergency generator would run the emergency systems as expected.

We really started adding more and more as the flight hardware started getting installed. Then when we really looked at the cost of things, like the GSE, we were like, “Man, even if we’re not in test, if we lost some of the hardware that’s in the chamber, like the CoCOA or something like this, you would create huge delays to the program.” Then we started thinking, “All right, what do we need to be doing to make sure that we’re ready,” even in all situations.

We wouldn’t require people to ride things out, but at least we had procedures in place and systems in place so we could say, “Here’s how we can walk away.” Everybody’s got their little thing, “you put the cover over your computer.” Well, we had a lot bigger checklists.

We would actually probably close the door to the chamber, backfill certain things with nitrogen, and power certain things on that needed to be powered, cool certain things on that we all had on emergency power. Then just walk away. Let data systems try to run and operate, so that if we came back you could see what actually happened in the environment. For most of the things, until we had more flight hardware, we didn’t require people always having to be in-place. For the flight test, yes, we stayed and we tested.

Ross-Nazzal: Before that test, I understand you had your control center in the building. Were you part of that decision and how that was designed?

Homan: The control room in the building that we control the chamber from had been in existence for a long time. Ended up involved with that from the mechanical engineering side and I also acted as a test director through the test as well. On the facility side we all gave our inputs, but it was really designed probably more to do with the human-rated testing in B, the facility control room.

We ended up having three areas of control for the flight test.

Ross-Nazzal: Talk about that.

Homan: We had the facility area where we do all the thermal systems for the chamber, the vacuum systems for the chamber, monitor the general overall safety of those systems and the safety of the test at a higher level.

The old Chamber A control room we completely gutted, and it became the James Webb Telescope room. This one probably only had somewhere between one to five people. During the flight test, we actually had the MIRI [Mid-Infrared Instrument] team in there, too. You’d have a few other people in the facility control room.

In the James Webb control room, they had their quality, their safety, their test director who was doing their operations on the telescope. They had a thermal team, they had a contamination team. When I say thermal team, I would say there were probably eight people on station. The optical team probably had at least one person for each of the optical telescope instruments. I’m not talking about the science instruments, I’m talking about the primary optical systems. Most of that is Goddard, and it’s contractors.

Then the science instruments, because they were European and the optics were American—mostly Northrop Grumman and Harris, and they have a lot of ITAR [International Traffic in Arms] restrictions—we had to keep them separated. The science instrument team—which monitored the flight science instruments and their operations, and turned them on and turned them off—they were in their own little area. They sometimes worked with the facility control room because we had the foreign access control thing.

There wasn’t anything that they couldn’t see operating the chamber. It was really just mostly the primary optics on the telescope that Northrop Grumman or Harris or some other contractor wanted to keep proprietary or was ITAR-restricted because of the designs. So yes, we actually had two primary control rooms with a third instrument control room.

Then we had a large room in Building 30 right next to the old Apollo Mission Control [Center]. We pumped all the data there, and we had a team through that test that sat there and they analyzed everything. We also sent data to Goddard and to Rochester to be real-time analyzed so that we could say, “Oh, yes.”

We collected the data, turned it up there, and then they would crunch everything and say, “Hey, we got a question. Re-run this test point again, we didn’t understand how this looked.” Or, “Yes, everything looked good.” You would usually get a quick look from them every eight hours. Every eight hours we’d typically have some type of handover where we’d all meet, talk about what happened the time before. What was successful, what wasn’t successful, any issues that we had, any problems we needed to be taking care of. And then run the next eight hours of testing and make whatever changes needed. Run whatever instruments needed to be running or just monitor. A separate group over there was doing a lot of the analysis, a lot of thermal and optical analysis.

The photogrammetry took thousands and thousands of pictures. There’s, I think, a computer up in Rochester that actually took all that stuff and said, “Oh, over that period of time this thing moved 50 nano [nanometers].” It was probably 50 microns [micrometers]. It probably saw things down to a thousandth of an inch, which is pretty amazing that it’s a camera. If you’ve ever seen the chamber, we have these four huge booms, and we have cameras that swing around and can take tons and tons of pictures. I’m not super familiar with photogrammetry, but I know the technology, and that was one of the main ways that we were able to see how things shifted and moved with temperature over time.

We could adjust things on the mirrors. If we adjusted the mirrors, did they go where we thought they went? You could run all that stuff from the control room, but it had to be analyzed by a large computer up somewhere else.

Ross-Nazzal: How many people were working in the control room, and then Goddard and Harris when you guys were running these tests, and even sims [simulations]?

Homan: Ooh, wow. I know because I was trying really hard—during Harvey, I was trying to get the numbers down to around 30, and I don’t think we ever did. I think they ended up staying around 40 on a shift.

Ross-Nazzal: On one shift?

Homan: On each shift, but we did switch. During Harvey, especially during that Sunday through Wednesday, maybe even Thursday, we told everyone, “No more eight-hour shifts. We’re working 12-hour shifts,” just to reduce time on the road. “If you’re not actually operating your equipment and you think it’s safe, you don’t need to come in.” Most of the time people wanted to come in, because they were more nervous to verify at least if it’s running, that at least the safety systems look like they’re all there. Yes, we ended up having a lot of people staying through the shifts. I can talk more about Harvey in a little bit if you want, but it definitely was the unexpected.

I think we had prepared a lot for a hurricane, how we were going to handle [the situation]. “Oh, the Center went from level five, four, three”—what we were going to do. Well, Harvey didn’t have five, four, three. It was essentially like, “Yes, it’s going to be a rain event, no worries. No worries, no worries.” Then Saturday night came, and we were like, “Oh, crud.” I’m on the phone all night long. I was actually at home and didn’t get in for several days. Those folks that were there stayed. Some of them were not planning to stay, but they couldn’t leave. We had some folks who were trying to get in, and we had at least one person whose car flooded. They had to wait 24 hours at a convenient mart until they were rescued by boat, and we were like, “Ahh.” A lot of people just said, “There’s no way I’m leaving my house.” We were like, “No, go find safety.”

We’re safe here in the building. Fortunately we have a shower in the building, and we had all the food, we had all the supplies. We had refrigerators and microwaves. Fortunately we didn’t lose primary power, and we had enough backup power that the refrigerator and stuff like that were going to be on, and emergency power anyway. We would have had food, and we did. So it worked out okay. It still was extremely high stress over the next few days.

Our biggest technical issue was Linde [Group (Linde AG)] was our provider for liquid nitrogen. On an average day, we need about two to three tankerfuls of liquid. We knew storms were coming, so we tried to make sure all the tanks were pretty full. That was part of our goal going into the test. “If we’re going to be testing in August and September, let’s always keep the tanks full, because that gives us close to between 5 to 10 days of full runtime.” If we make some modifications, even get some longer runtime.

“Hey, none of the liquid nitrogen trucks showed up.” Day one goes, we start making calls, and they’re sending us to somewhere else, New Jersey or Pennsylvania. They’re like, “No, that plant is closed. It’s flooded; it’s shut down.” We’re like, “We need liquid.” I think they didn’t deliver either on Friday or Saturday. I think they were having some problems with the storm already, so we were already getting a little bit behind, and we didn’t get delivery for several days.

[NASA] Kennedy [Space Center, Florida] was great. We called the Kennedy Space Center. Between Mary Halligan and myself and Kennedy, we opened an emergency contract with two different companies. One was called GenOx [Transportation] and the other was Praxair. Praxair could produce the liquid nitrogen here locally and had a clear path—their plant wasn’t shut down—and Genox would go pick it up and deliver it for us.

That came on, and about the same time that started happening, Linde, who was our prime contractor for the liquid nitrogen, was able to get ahold of a driver who had been flooded out but had a truck with him. The plant had plenty of liquid, it just wasn’t running. He had a clear path, so he started delivering. It was actually the CEO [chief executive officer] of the company who made the calls and made it happen. I think they realized that this was a big deal for NASA, and they took it very serious. It took a little bit of time, but it was scary. I would say it got stressful when you go one day, two day, three.

At a certain point we would have had to make a decision to stop testing, and we didn’t want to go there. Because it took the chamber probably only 20-some days to get cold, but it probably took the spacecraft 45 days to really start meeting their thermal requirements. It was right when the telescope got to its flight-like temperatures that Harvey hit, and you’re like, “Uh.”

I would say we were even testing 10 days. This is like 60 days, we’ve already spent close to two months to get to this point. Yes, because we started pretty early in July. Maybe 45 days we were there. We really wanted to continue testing, and we were able to. Between the contract that Kennedy set up and Linde coming back with a driver, we were able to maintain our liquid level for a period of time, essentially getting one delivery from two things.

Then about five or six days later Linde got their plant back up and operating, and they were able to fill our tanks back up to full capacity. We didn’t get to a point where we got so low that we ever had to make a decision of stopping the test, but that was probably one of the bigger stressors.

I think the personnel stress—for me, it bothered me when I knew certain people were there and stuck there and they didn’t plan to. I knew some of the folks, their houses were flooded, and they didn’t expect to be separated from their kids and family. Now they are, and their spouse is having to deal with the safety of their family while they’re required to maintain the safety of what we’re doing at work. That tugs at you a little bit.

Ross-Nazzal: I imagine that must have been challenging. Plus you probably wanted to be here and you were at your place, couldn’t come in.

Homan: Yes. I came in I think—probably was like Tuesday before I actually made an effort to get in, and that was partly because of flooding around where I lived. Definitely some people had been there a little too long already. It was getting stressful.

Maybe you’ve heard—the chamber did great, and all the systems did great. The telescope never knew the hurricane was going on, but the building had several significant leaks. Even our data system, they were leaks all around. We had planned, with metal and plastic, to buy supplies to protect everything. And we did. The techs [technicians] worked, mopping stuff up, creating barriers.

The control room, the James Webb guys—we created this huge essentially tent structure that kept the ceiling tiles and all the rain off of them when they were sitting there. The Europeans, they took this one picture where they actually taped the stuff to the wall and just laid it over them. So it looks like they’re working underneath tape, and they are. They’re working, and their computer screens are underneath this huge tarp. It doesn’t let any of the water that was leaking in come in.

The Center Operations did a great job that night of finding where the water was coming in and deflecting it and capturing it and mapping the roof. We had spent years mapping the roof finding where leaks were over those times. Even in 2015 we had several major roof leaks, and every time we had a major storm we took it as an opportunity to look around and note, “Hey, does anybody see any leaks anywhere?” We would let the Center Operations know and they would come and fix those areas of the roof.

I think because of the quantity of water and the wind, certain things were not designed to handle certain loads, and it breached the roof at certain areas. They were able to fix that immediately after. They did get the building back to a pretty safe state and verified that there wasn’t asbestos that came down or caused any problems. I think some did, but it was liquid, because it was wet. It wasn’t airborne that you were going to breathe, it was usually just things disintegrating coming from the roof. They were able to get their hazmat [hazardous materials] teams in here and clean it up and take readings and verify that the team was safe.

Of course, like I said, we had a lot of Europeans here. We walked through all this with all of them as our safety briefing for the testing, and even before the testing. Gave them a hurricane briefing. I think they’re still—

Ross-Nazzal: Are they scarred?

Homan: Yes. “Oh my goodness, this is crazy what’s going on here in Houston.”

Ross-Nazzal: How long did it take you to return back to what you would consider normal operations for the test?

Homan: The testing never slowed down. We were in the middle of optical testing. Actually the testing during the storm was excellent. We actually understood some of the issues and were able to keep moving forward on checking off where we thought we were having some problems on some of the major requirements. We were able to verify, “Oh, yes. No, those did happen correctly,” or “We understand the problem. Here’s what we need to do to make the change.”

The nice thing about James Webb is it’s not like Hubble that’s a monolith. If you needed to make adjustments, it’s got that ability. And that was what they were trying to learn. “Okay, they made those adjustments,” and were able to move on. It was very successful optical testing during the storm.

The biggest problem—people are always coming and going. A lot of them got stuck because the airports around here got shut down. A huge amount of folks, I think somewhere around 30 to 40—we always had probably about 100 to 120 folks between Europe and the overall James Webb team that were on travel staying in hotels here. Probably most weeks about 30 or so were exchanging. Some of them ended up not being able to exchange, not being able to get back to Europe, not being able to get back to Washington, DC area or wherever they were from. I would say it took probably a little more than a week to fully get things back to normal. The nice thing was I think the Agency or Goddard provided a jet that they landed at Ellington [Airport, Houston, Texas], and we were able to get the Center to use our buses to pick people up.

That at least let a lot of folks that lived in the DC area go home, and new folks to come. We told everybody to come prepared. We would actually tell people on their shifts to bring an extra change of clothes just in case, bring some basic toiletries just in case.

As the storm was coming, we definitely told everybody, “Bring spare clothes, bring some extra snacks and food that you might want and might be willing to share. And bring some basic toiletries, just in case something changes and you get stuck on shift.” It was good because a lot of people stayed in hotels so they just grabbed all the little hotel bottles, and they used the shower there when they couldn’t get back at Building 32.

That was an experience. It took a while. The heavy rains really were Saturday night and Sunday. I would say by the following Sunday, I think, they got that plane in here. Maybe it was Friday or Saturday they got the plane in here. That really changed a lot of the attitude. Some people had been stuck there four or five days and hadn’t left. When you’re there 24 hours a day and you’re sleeping there, it was getting stressful.

Ross-Nazzal: Yes. It’s kind of dark in there at least. I guess you could go out to the areas where there are some windows, but it’s kind of cave-like.

Homan: Yes. A lot of them were, I will say, extremely dedicated and focused on just the testing. But I think it’s nice to, at the end of your shift, leave and go somewhere and know that there’s restaurants you could go to. Now there’s not much you can do in the area. A lot of times people just ended up hanging around the building.

Ross-Nazzal: I imagine you spent a lot of your time on phone conversations with people at Goddard and Harris, giving them updates, and talking with people from 32.

Homan: Ken [Kenneth J.] Anderle was on shift, and he was our local incident controller for the period of time. He was Jacobs’s manager supporting EC, and supporting us on the test from the Johnson Space Center. He was a West Point [U.S. Military Academy, West Point, New York] grad, military guy. Great incident commander, nothing is too scary for him type of thing. I was on the phone quite a bit with him, just trying to find out what was going on, how people were handling things, and making sure what security was going on.

I was contacting Security too a little bit here. What were their thoughts, “Are we going to be able to get people on or off?” Sometimes they actually locked the gates, saying, “The waters are coming up. Please don’t tell people to come, or tell your people that are on shift, ‘Hey, make sure everybody knows. Don’t get in your cars right now. You might want to go out and check your car and move it if it looks like the water is rising. Find a high spot. But don’t leave the Center because it’s flooding by the main gate, and that’s the only gate that’s open right now.’” That was that.

Definitely was in contact with the program management, trying to let them know what our decisions were, trying to get their buy-in. Sometimes I got a little pushback from their test team to say, “Oh no, everybody can stay,” and I’m saying, “No, not everybody needs to stay. I really want to limit the numbers in case we do have a real emergency and something does happen.” They wanted 70 people at the building, and I’m like, “We’re limiting it to 35,” and we settled at 40. I know they were trying to accomplish things. I know everybody wants to babysit their hardware, but they’re safe in their hotel rooms. The ones that are there, keep them there. Then when we have the ability to change people out [they can come in].

It really wasn’t too bad for a lot of people. Once the rain subsided and a lot of the water cleared, the paths around Clear Lake were a lot better than of course what you saw right around—

Ross-Nazzal: West Houston.

Homan: Yes, West Houston, or Dickinson, or Friendswood. Once you got outside this Clear Lake bubble, the waters were a lot worse for a longer period of time. Really saw the people that we had to help and deal with the flooded homes. The local hotels and the paths to JSC were not too bad for most of the people that were here on travel.

So yes, I was just in communication with them. Once we realized the LN2 situation, we were able to get in contact with the procurement team at Kennedy. They worked really well between Goddard, Kennedy, and us to get a contract, get it funded, and get some emergency things in place.

Tried to let people [know] what was going on with my branch and division, directorate, and even let Ellen and Mark [S. Geyer] know what was going on as much as we could. They were checking in on us as well.

Ross-Nazzal: I understand from Conrad [Wells] that after the storm you pitched in and helped out a lot of people with their homes.

Homan: Yes. The James Webb people that were on travel, a lot of them were like, “We can help out during our off time.” The first thing we did was try to give them a list. EA put together a list of, “Here’s people in EA that are affected. We’re meeting at the Gilruth [Center].” We found three or four local churches had some really great organized teams and said, “Hey, go.”

When they were like, “Hey, we got too many people supporting the NASA families,” we were like, “Here’s a couple of churches.” They just went over to these churches and they were like, “That was great.” They were like, “Oh, here’s a wheelbarrow, shovel. You’re going to this house.” They just pitched in and went to houses of people they didn’t know. Looked at it and started cleaning up the debris or whatever they needed to do, and I think felt like they were part of the Houston Strong recovery team. That was nice that a lot of them chose to participate that way.

Ross-Nazzal: Yes, something positive came out of it.

Homan: Yes, it was. I think it helped deal with the stress that had just happened previously, so I think it was a nice way to help the recovery. At the time I think there was a lot of stress. Everybody’s focused on the test. I’m usually a very technical person, and now I’m really worried about everybody’s safety and for something possibly going wrong. “I know you guys aren’t seeing it from my perspective.”

It took me a little bit calling Goddard management saying, “I need you guys to help your team there. We’re not going to stop testing, but maybe we can slow some things down. Just to get the numbers down and to make it feel like if we have a bigger emergency we have enough things to handle the situation. Because I think we have too many people in the building and something does go wrong, it’s just a lot worse to handle logistically.”

Like I said, it just seemed like the stress levels got really high for a period of time. I think after the recovery and working some of that stuff, it helped set everyone’s attitude a little bit more. Especially when you go out to somebody’s house that you don’t even know who they are. “Here, show up at this church,” and this church has got groups going, “Here, we can handle seven people at this house. Can you go? There’s bleach, wheelbarrow. Go get it done.” “Okay.” They did, that was good.

Ross-Nazzal: I’m going to let you go here, I just had one other question. We’ll have to have you come back, but I did want to ask—was there any way of perhaps controlling the test remotely, doing a few things remotely, if you guys had had to evacuate?

Homan: The chamber systems are all automated, all the alarms are automated, but you have to not just be within the JSC firewall, you have to be within the test firewall as well. Essentially you have to be onsite on a dedicated computer to make changes.

But what we did, and what worked really well, is anybody who has access could at least remote in and see what was going on and monitor the health [of the spacecraft]. I would get text alerts. “Oh, a temperature was changing that we didn’t expect,” or something like that.

Ross-Nazzal: From the hardware or from somebody who was there in the control [room]?

Homan: No, from the hardware on our chamber system, our helium refrigeration or our liquid nitrogen. “Oh, something’s not quite right,” the flow rate of something is not quite right. It was all automated, and it would send it to the engineers or the technicians that needed to know about it. They could start checking it, even if they were on-shift or not on-shift.

A lot of them too, they would just have it up at their houses. During the storm people were able to monitor the systems and look and make some suggestions. “Here’s some things we can do to conserve some liquid nitrogen,” and do some things while we’re holding but not changing the environment. We did not have the ability to change control settings. You could leave it in a safe state and monitor, but you couldn’t change parameters, if that makes sense.

Ross-Nazzal: Yes, just curious about that. Thank you so much for coming in today.

[End of interview]