NASA Johnson Space Center Oral History Project
Edited Oral History Transcript

Conrad Wells
Interviewed by Jennifer Ross-Nazzal
Houston, Texas – 14 August 2018

Ross-Nazzal: Today is August 14th, 2018. This interview with Conrad Wells is being conducted in Houston, Texas, for the JSC Oral History Project. The interviewer is Jennifer Ross-Nazzal, assisted by Sandra Johnson. Thanks again for coming over and walking over in the heat. Appreciate it. We ended last time reaching the pathfinder OGSE [Optical Ground Support Equipment]-1 and 2 tests, but we did not talk about them. We said we’d talk about them today. I thought we’d talk about your role in those tests.

Wells: Yes. I think I’d been talking about how these tests helped to reduce risk for the final flight test. They were learning experiences for the facility, for the optical staff, for the thermal staff, for the hardware, frankly, itself to get ready for the flight test and to reduce the risk for the flight test. That was the smartest thing that was ever done. We learned so much in each one of those tests.

During one of them we had a water main break during the middle of the test down in the cryopumping room. There was a 10-inch water line main burst, and they had to turn the whole system off. Luckily that was during a warm-up, and we were at a safe temperature. So everything was fine. If something like that were to happen on the flight unit, that would have certainly been more of a concern.

We learned lots of things in all these tests, especially about the facility and thermal and optical. The pathfinder consisted of 2 primary mirror segments out of the 18, a secondary mirror. In the second test we had the flight Aft Optics System, which is the tertiary mirror and fold mirror that then directs lights into the instruments. From an optical testing standpoint, we needed all those practice runs to get our software ready for the flight test. Processing the interferometer data we knew would be challenging, and it was always more challenging than we thought it would be. In each test, we learned a little bit more and continued to progress in our processing abilities.

I think the first pathfinder test was either 45 or 60 days, the second one was 60 days, and the third one I think was maybe 90 days. They were long. They were grueling tests for the people that worked it. One of the unique things about those tests—because it wasn’t a flight unit, everybody had to take a lot more responsibility. The team was not as big. I got to be test director for a number of third shifts, because that’s where they always needed people to fill in. Frankly, they were a lot of fun. It would be during warm-up or cooldown, and things always go wrong in the middle of the night. You always have challenges or need to make some decisions in the middle of the night. It was a lot of fun to be there to make those decisions, to make those calls. It was a great experience for all of us there doing that, making these [tough] decisions.

These tests always present things that you don’t expect, and practicing solving those problems is very important. There were times that we weren’t warming up quick enough, or we weren’t cooling down quick enough. You started to get a feel for how the chamber worked. It’s a three-quarter-of-a-megawatt helium compressor. We get a handful of trucks of liquid nitrogen a day. Dialing those temperatures and making it work thermally was a great experience.

Keeping the hardware safe was really a thermal concern. Very often in those early tests you were learning how to tweak things, how to make things work, to stay within the safe range for all the hardware. Your goal was to stay just in the—you’ve got a red zone that you never want to get to. You’ve got a yellow zone, which you can go into if you need to. Then you’ve got the safe zone. You’re always trying to ride the yellow region, just outside the yellow, because that would be a faster test, the quicker you can go. So we were always riding through it, “Maybe can we get permission to go a degree over, a half a degree over, into the yellow zone? It’ll save 8 hours in the test.” You’d get approvals to do that, because a good portion of the test, more than half the test, is the warm-up and the cooldown. For the flight test, it took 30 days to cool down and 30 days to warm up. You only had about 30 days in between, 30 or 40 days. So most of the test is the warm-up and the cooldown.

Our goal was to align those two primary mirror segments to each other and phase them, to turn them from two individual mirrors, to bring them to be perfectly flat and parallel to each other was our goal. That first test certainly we struggled trying to get that done with the software that we had at the time. In the months that it took to get ready for the next test, we worked on our software.

We did a better job in OGSE-2. We still struggled; it was still challenging. One of the problems is with only 2 mirrors it actually perhaps is a little harder than if you had all 18. Getting the data to process correctly so that we could converge to a phased mirror was again more challenging than we thought. We got close probably in the second test, but we didn’t have it nailed yet.

I think partly as a result of that, the third test, which was the thermal pathfinder, they decided to make that an optical test as well. There was a pretty long period of time between those two tests. They had to bring the telescope out of the chamber and make a lot of thermal modifications. Basically the goal of that test was to have a simulator of the flight telescope thermally inside the chamber so that you could practice the thermal profiles of the flight unit, you could practice those on something that was thermally similar to the flight unit.

That took months. I wouldn’t be surprised if it was close to a year of work to get all that done, certainly six months at a minimum. That gave us a lot of time to process the data. There was a team at NASA, a gentleman by the name of Kong [Q.] Ha worked really hard with Joe [Joseph] Cosentino and Gene Olczak, the engineers from Harris [Corporation]. The software was really developed by Gene. He was the real brains behind the concepts, and Joe Cosentino was the guy that got it done. He made the software user-friendly. He made the software give you answers that an engineer could understand.

One of my goals was to be able to get as much diagnostic information as we could out of the software to try and help us make decisions. Joe was always ready to help us do that. We ended up with a final product that gave us probably maybe more information than we needed, but at times we needed portions of that information. It gave it to us, so that when we got to the flight test it all worked. It all worked; it all worked really well. That was really a testament to the team as a whole but then Joe getting the details of the software, giving us diagnostic information of individual mirror tilts and mirror pistons for a bunch of different entrance conditions to help the engineer be able to make the right decision. It really isn’t a process that can completely be done by a computer. It really takes a knowledgeable engineer to look at the data and understand what’s going on. We ended up getting the information that we needed to get that done.

Those tests really were critical in executing a successful flight test. OGSE-1, OGSE-2, and thermal pathfinder, those three campaigns really helped us get ready for the flight test.

Ross-Nazzal: You mentioned that you were a test conductor during that third shift, when things might go wrong, if they were going to go wrong. Are there a couple of examples that you might share that you were surprised by?

Wells: One of the goals of program management was always to get these tests done as quickly as possible. How much could you get done in how short of a period of time? I know one of the evenings that I was working, I thought, “Well, heck, I don’t see why [we can’t go faster].” We were told during the daytime to go a degree an hour—or half a degree an hour—and I started looking at the data and I said, “I don’t see any reason why we can’t go,” I don’t know if it was 30 percent faster or whatever it was. I said, “This is what we’re doing. We’re going to do that. It all looks safe.” The thermal people bought onto it. The flight hardware safety people bought onto it, and we did it. When the leaders came in the next day, they were very happy that we had done that. We’d made the right decision. That was a good one.

I’m trying to think of one that wasn’t. Obviously the broken water main was a huge event. It teaches you that as hard as you can to try and find problems with your system, who would have thought that the water supply, just a water supply, was the problem. We follow these risk management procedures to look at all the different aspects and what could go wrong. To say, “Okay, if the cryo system went wrong what would go wrong?” We were thinking it would be the power or the air pressure or something like that. It turned out to be the water main.

I can’t right at the moment come up with one. I know there were a couple. I know we called Jon [Jonathan L.] Homan in the middle of the night one night because of something happening, and I can’t quite remember what it was.

Ross-Nazzal: It’s okay, memories fade. Obviously [it] didn’t impact things too greatly. You continued on, and it led to a successful test. Were you there when Webb finally arrived here in Houston?

Wells: When the telescope arrived? Yes, I wasn’t there when it flew into the airport. I was there when they loaded the shipping crate. The shipping crate would come into a truck lock, a very dirty truck lock. It’s interesting. You’re trying to keep everything very clean. You’re opening these doors and 10 feet over is this very dirty area.

You manage for these sorts of things. You’re always going to get a little dust on things. The telescope comes in in a clean shipping container. They air barge it in, and then they start the cleaning. Before it comes in they clean the whole container really well. It was just piles and piles of rags. It’s not really rags, they’re these clean cloths that are, I don’t know, probably $5 or $10 apiece. There’s just piles of them, because they only use them once, and they’re not very dirty. I’ve taken some of those home, and they’re great for washing your car. They last forever. They wash it really well, they clean it, they bring it in.

I was there when they first lifted it out of the shipping container and placed it onto the lift and turnover fixture. I was there when they first lifted it up, and I have pictures from the viewing area of me being reflected off the mirror. When the telescope first got there, there were lots of moments of awe of looking at it. I must have given probably 100 people tours by getting friends who work for NASA, we would all come together and get visitor passes. I gave two Boy Scout troops a tour, I gave two large groups from my church a tour, and that was really great to help people to see it.

We’ll get there, but after a while you almost become, “Okay, there’s the telescope. There’s this $8 billion telescope sitting in the clean room.” You got used to it. The weeks before it was leaving, I will admit it was a very sad time, to think my time on the program was coming to an end. I probably wouldn’t see the telescope again. I suited up and went into the clean room and got permission and basically was right underneath the telescope. They said, “Well, while you’re there, can you do some inspections?”

I found some things on the insulation that were of note, and a couple of them they had seen already, but I did find something that they hadn’t noticed. I pretty much just sat in there for a whole afternoon just looking at it and being with my telescope. It was really quite nice.

Ross-Nazzal: What did you find on the insulation that they hadn’t found?

Wells: The insulation is a black Kapton. It’s a Kapton sheet with a very black coating. One of the things in orbit is that any corner, I guess the electrons flying through the space will basically cause an edge to glow, so you can’t have any exposed edges. Those pieces of insulation are actually tied together with a clean room string tie. That string tie also has to not be exposed, so it has these things called Part C, which are basically these little folded things that are about a third of an inch by an inch that cover all that up. One of those seemed to be too loose. It was one of the things that they would have found anyway. They spent a lot of time inspecting it since the test. It was a nice time, and now it’s in California sitting for an extra year now. It launches in 2023 as we now know, and another $1 billion later.

Moving to the flight test, I talked last time about our goals of aligning the primary mirror to make sure that we had enough actuator range on orbit. Our goal was to take these 18 mirrors, which had never been aligned together at the same time, and to turn them into 1 mirror. The piston and the tilt between all these mirrors would be the same to tolerances of numbers like 100 nanometers and 100 nanoradians, probably more like 50 nanoradians. Our ability to measure the mirror would be about one-ten-thousandth of a human hair or so. We could measure to about 10 nanometers the flatness of these mirrors, and your hair is about 125 microns. If I did the math right, it’s about one-ten-thousandth. We have a special interferometer that uses three different lasers to measure the surface of the mirror with three different wavelengths, two at a time. By combining the data from those two wavelengths you’re able to measure across two mirrors. Interferometry, which is what we use, is very good at measuring a flat continuous surface, a flat mirror or curved mirror that’s one mirror.

When you start having two mirrors it gets confusing to the interferometer. What happens at that interface between the two mirrors, this interferometer measures the surface of the mirror with two wavelengths 10 microseconds apart. Laser one fires and measures it, and laser two fires and measures it 10 microseconds apart. You combine those two sets of data to understand both the surface figure of the individual mirrors as well as the height difference between the mirrors.

I think I mentioned last time we had a lot of challenges with those lasers. Those lasers, it was difficult for the supplier, 4D Technology, to find a combination of wavelengths that they needed to solve this measurement problem. The ones that they ended up using from New Focus had lifetime problems. We bought some spares at the beginning, [but] we didn’t buy enough. We basically had to refurbish the lasers. When we designed the system, we made sure that it would be serviceable. The interferometer was inside the chamber, the head with the optics, but the lasers are fiber-fed from outside. The lowest reliability item we kept outside the chamber. Reliability in being able to conduct this test successfully was very important. Keeping the things that could fail, the highest chance of failing, and the things that did fail, outside the chamber. We had two source modules, and each one failed once, but we always had a spare, so we were always okay.

When one of them broke, we sent one back to be refurbished. New Focus had stopped making these lasers and basically said, “I’m sorry. We don’t have any more. We can’t make them.” Luckily, NASA has a lot of pull. NASA does a lot of business, they buy a lot of optics, they buy a lot of lasers. New Focus agreed to do custom builds and make more of these lasers. For the James Webb Space Telescope, companies and individuals will do what it takes to get the job done, and that’s an example of a case where they got the job done. We always had a spare to use, and we needed them.

We struggled a lot in the early tests trying to understand the performance of the lasers, and by the time we got to thermal pathfinder we certainly knew what we were doing. For flight, they behaved very well until probably the last [few days]. At the end of the test we did some ambient work, and they started getting a little flaky then. It did allow us to align these mirrors. We did a coarse alignment and a fine alignment and then a fine phasing to bring all these mirrors into a single mirror.

We had the luxury during cooldown, during these 30 days, there wasn’t a whole lot else going on, so we could basically practice. We hadn’t done it with 18 mirrors yet. We’d done it with two mirrors. We hadn’t done it with 18. We practiced for probably at least two weeks before we had to do anything for real, and that was really important. I requested that time. We needed the support of people to actually move the mirrors. It was really important to us to get that done, and it really helped us again, the practice, so that when it was time to do it for real we could do it, and we could do it successfully, and we could do it in a reasonable timeframe.

It probably took us a week, two shifts a day, to get all the mirrors into the interferometer. You need to get all the light through about a 1-millimeter hole, or so. You’re shining all this light through the telescope. It comes back, and you’ve got to get them all tilted into about a 1-millimeter hole, 16 meters away. It was more challenging than we thought it would be, once again. Once we had done that, it became much more routine.

The testing, we had to get ready for. One of the important things for the telescope is how it performs when the temperature of the telescope changes. On orbit, when it aims from one star to another, the sunshield angle to the sun will change, so the temperature of the telescope changes. The predictions say it’s about 0.15 degrees, so a little over a tenth of a degree. Our goal was to check how much error in the wavefront was introduced by that 0.15 degrees. That number is too small. It’s an unmeasurable amount, so we did it over a 3-degree range. We multiplied it by 20x. We went from 0.15 to 3 degrees and measured for—I think it was maybe two or three days, 24 hours a day, every 3 hours we would measure the mirror. Then they would look at the progression of those measurements over time.

We had done that in the earlier tests, and we always had challenges processing the data until thermal pathfinder. The other two, we really had challenges processing the data. In this last test it worked really well. The first test, they had a prediction. They had a measurement, and the lines basically were right on top of each other, so that was great. We were working 24 hours a day at that point. When you’re working first shift, second shift, or third shift, it’s hard to be at the top of your game all the time, and that’s what’s needed. You’re a little stressed. You’re working long days, so it’s challenging. It’s mentally challenging, but we got through it, and that test worked out really well.

Then we had a couple days to phase the mirrors again, and some other people did some testing. For about two days or so other people were doing testing. We didn’t have to be here, and that was really nice. For the thermal team it was 100 days [on], and they staffed 24 hours a day 7 days a week for 100 days. For us it was periods where we would be on and then periods where we wouldn’t be on. We had a smaller team as a result, so it was very specialized. The people that could support us had to have been in one of the previous tests. They wouldn’t have understood the system and how it worked.

We had James [B.] Hadaway from the University of Arizona; myself; Joe Cosentino, an engineer from Rochester; Gene Olczak, an engineer, the architect of the software from Rochester; Mike [Michael] Zarella, a young engineer from Harris in Rochester; Randal Telfer, who was at the Space Telescope Science Institute at Johns Hopkins [University, Baltimore, Maryland]; he was a really big help to the team, and Dave Chaney from Ball Aerospace [& Technologies]. I guess we had seven people, and we had a technician, Mark Connelly. The eight of us had to staff 24 hours a day seven days a week while that was going on.

We did then another thermal distortion test—so we had a first thermal distortion test and then the second thermal distortion test. The data in that test did not look right. The straight line went up and down really badly. The team paying attention to that—and it wasn’t that I wasn’t paying attention, but we had our focus, and our focus was to continue to learn how to phase that mirror better. There were other people who had the bandwidth to think about these things, and that’s when they started looking into what would become some real concerns in the test, this instability.

They were seeing wavefront instability 10 or 100 times, probably 100 times larger than we were supposed to see. As a result, teams of people from the Space Telescope Science Institute—the thermal teams, there were optical teams, mechanical teams. People all across the program were starting to look into this.

Our team was, even though we did the measurements, a little bit oblivious to it because we were busy trying to get the mirror aligned for the next test. That thermal distortion test ended up not succeeding, and I’ll get to the causes later. In between these tests, when we would phase the mirror, there was another team that was putting light through the whole telescope. We had simulated stars down near the primary mirror that would bounce through the telescope and bounce off of three mirrors. Harris built three one-and-a-half-meter autocollimating flats, basically three large flat mirrors up at the top of the chamber that were round. Those would allow light to go through the telescope, reflect off those, and then come back through the telescope and get imaged right on the science instruments.

One of the goals of building a telescope is to test it as you’re going to use it. Originally we were going to put an interferometer in the back of the telescope and shine that interferometer through and come back. That wouldn’t have tested the instruments themselves. This concept that people came up with during the replanning of the testing that we discussed last time, that’s actually going to continue to be used on future observatories.

We were putting light through the telescope, and the people looking at that data, the pass-and-a-half team it’s called, because light passes through the telescope and then comes back, they started seeing funny things in the data as well. At one point they made a movie. You could see kind of a sinusoidal, an upward and downward change in where the spot was, where the image was. People started understanding that there was a variation every 40 minutes in some of the data, and then at times there were these large changes. During the thermal distortion test there was this very large change.

We then worked to get our final phasing done. It was the last week of August. On the horizon was a tropical storm. There are three shift reports a day, and each one of those shift reports would report on whether or not there was any weather in the Gulf or in the Atlantic. These reports of a storm that ended up becoming Hurricane Harvey became apparent that that storm could potentially hit us. There were test planning teams getting ready for that, facility teams getting ready for that.

Our job was to quickly get that mirror aligned. We needed to get that mirror aligned. We worked probably for about at that point three days maybe to get the final alignment done that concluded I think on Friday the 26th of [August]. That was the Friday that Harvey arrived actually is when we finished the alignment, the first alignment at stable cryo[genic temperatures].

We had been aligning the telescope, but it was while it was cooling down. It would be changing. Each time you would measure, there would be a little bit of change [in the mirrors because they were still cooling down]. Early on it was almost impossible to align, in the first part of that two weeks. Towards the end of that two weeks now here we are after the second thermal distortion test. We were very close to stable conditions, stability being defined as 0.1 Kelvin per hour, so about 2.4 degrees in 24 hours was what we would say, “Okay, we’ve started stable cryo.” At that point then the mirror is quite stable. You’re at the 50 Kelvin temperature or 40 Kelvin temperature. When we would align the mirrors they would stay in place [because the temperature was stable enough]. Now it was quite satisfying to see the convergence [in alignment].

The mirrors took a long time to move. One of the reasons it took us so long, it might take us 5 minutes to take data, it took us about 15 minutes to transfer it, and about maybe half an hour to process it. Then you think about it for a little while. Between half an hour and an hour you’d have an answer for what move you wanted to make. The moves took sometimes as long as 8 hours. If you were making a really long move it could take [at least] 4 hours. Eight hours probably was some of the longest moves. One to 4 hours was very common. Then there was a lot of sitting and waiting around [for the mirrors to move].

Fortunately and unfortunately, there are lots of checks. When the mirrors are being moved, it’s constantly checking that a command has gone out and the command has come back and that the mirror has moved that amount. There’s a command going out, there’s a command coming back saying it did it, and then there’s a measurement that said it moved that much.

If any one of those three things has a hiccup in the software, the whole move stops and they have to piece together exactly what happened. Was it the command going out? Was it the command coming back? Was it resolved? They actually count the number of turns in the motor. So by counting the number of turns in the motor they can understand how many times the motor has moved and then they have something called a LVDT, a linear [variable differential] transducer. Basically as it moves the voltage changes, and they can measure that [change in voltage]. If that gets an anomalous reading it stops.

Quite often we would come in and there’d be a mirror [move error]—not quite often. Once a day it wasn’t surprising for that to happen, and that would really slow us down. It would give us some time to analyze the data, but that would really slow us down.

We started this fine phasing process, and I think during that there were a couple mirror move errors over those let’s say two or three days. Then we started doing the final alignments. Early on in the tests we were moving the mirrors in piston and up and down. We were tilting them, but the mirrors can also be decentered and they can be clocked. They can be rotated about themselves. There’s six degrees of freedom on each mirror. There’s six actuators on each mirror that lie in a hexapod arrangement that give you those degrees of freedom. There’s one radius of curvature actuator that helps bend the mirror to give them all the same radius of curvature.

When we started doing the decenter moves, some of those were half-a-millimeter-to-millimeter moves, quite large. When we’re doing a fine move, it’s nanometers. We might be moving 100 nanometers or 1 micron. Some of these though were half a millimeter, 500 microns, or a millimeter move. Those would take 4 hours. So you’d go out, you’d get dinner. You’d wait around and hope there wasn’t a mirror move error. There was a lot of hurry up and wait. You’re under a lot of pressure once you’ve taken the data to get an answer. Then you wait.

The mirror phasing was going quite well, and Harvey was coming. My kids were out of school on that Friday, so I actually wasn’t there the final day that they did the final alignment. That was the day Harvey arrived. They did that final alignment. We had another setting in the software that we’re really never used and didn’t know that we needed to use. We had that set wrong in the software, and it ended up [mis]-aligning the mirror in decenter by about half a millimeter.

We knew it, because we had this amazing—I think I talked about it last time. In order to measure where the mirrors were externally, from a big picture, we had a photogrammetry system which took 1,100 pictures and could tell you where the mirrors were to about 100 microns. You’d know in general where they were. That system takes about 4 hours to take data and about 8 hours to process it. You would take a measurement, and the next shift would get the answer.

At that next shift we found out that the mirror had been decentered. I was locked in my house, because we couldn’t go anywhere. We didn’t flood. We didn’t lose our electricity. We had just enough food. We had water. We were some of the lucky ones. People here [at JSC], they went to two shifts a day, so that people would only have to drive to hotels twice a day. Food became a problem for people staying at hotels. Restaurants didn’t have food. That became some of the major concerns of the people staying in hotels. Some of the hotels, they had water problems.

People were ferrying back and forth and sleeping [in Building 32]. Joe Cosentino got stuck here that Friday night. He didn’t get back to his hotel. Saturday morning they did some diagnostics on the mirrors, taking some measurements. He got a ride back to his hotel. Unfortunately, he left his computer here. So the poor guy was stuck in his hotel room for [the next] five days without his computer. He couldn’t do analysis. The poor guy, at least I was at home with my family.

The team continued to do testing. We stopped testing with the interferometer, but the other team that was using the simulated starlight continued testing during all of Harvey and actually got a lot done. That was I think one of the times that they noticed—it became very obvious that there was some sort of instability in the telescope that was much larger than the required value.

When Harvey was done on about August 31st we started coming back. At that point, we had to find out why the telescope had this instability. That was the goal really for that portion of the test. People were still doing functional tests of the instruments and doing other diagnostic tests that were part of the test plan [but we had to figure out what was causing the instability].

One of the things in the test planning is you plan for more tests than you’re going to do, and you’re ready to do any test hopefully at any time. When there was an [extra] 4-hour block here [or there you wanted to use the time efficiently]. Mark Waldman, a former Harris and Kodak engineer, and Tony [L.] Whitman, my boss, the chief systems engineer from Harris, were the test conductors. It was their job to figure out given this shift and the following shift what test they could squeeze in where and how.

Our job became at that point to start measuring this mirror and figure out what’s going on. We started taking data. Every hour for 8 hours we did, and then we started taking data every 2 minutes. Then I realized that there would be a way to take data basically continuously. So then we took data was every 30 seconds or so, and then we figured out we could tailor it. We were able to take data at any periodicity we wanted for almost as long as we wanted. We started taking data sets that had 7,000 frames of data when normally we did 100 or 150.

It would take an entire shift to process that data. It was that data that helped us see this 40-minute [periodic behavior of instability]. Lo and behold, when we measured the primary mirror we saw this 40-minute profile. The thermal team said, “Well, what goes at a 40-minute profile?” They couldn’t find anything that did except they found a warm electronics compartment that actually had that [time] period [on its temperature]. There were heaters going on and off with that exact period. There were four sets of heaters. This was electronics for the science instruments. There were four of them. One for the NIRCam [Near Infrared Camera], one for the Mid-Infrared [Imager (MIRI)], one for the [NIRSpec (Near Infrared Spectrometer)], and one for the Fine Guidance Sensor. It turned out that that electronics compartment was set to 1-degree variation, and they just went on and off. We reduced that down to an eighth of a degree, and it smoothed out that variation to a great extent. The variation went down to like a 5-minute period and the peak-to-valley went way down. The data looked a lot better. They came up with concepts for how to deal with that. It was good to solve that.

[End of interview]