Johnson Space Center
Return to Johnson Space Center home page Return to Johnson Space Center home page


NASA Johnson Space Center Oral History Project
Commercial Crew & Cargo Program Office
Edited Oral History Transcript

Michael J. Horkachuck
Interviewed by Rebecca Wright
Houston, TX – 6 November 2012

Wright: Today is November 6, 2012. This oral history interview is being conducted with Mike Horkachuck at the NASA Johnson Space Center [JSC] in Houston, Texas for the Commercial Crew & Cargo Program Office History Project. Interviewer is Rebecca Wright, assisted by Rebecca Hackler. Thank you again for taking time out of your busy schedule, we appreciate it.

Horkachuck: You’re welcome.

Wright: You’ve been involved with the COTS [Commercial Orbital Transportation Services] program since September of 2006. Share with us how you first got involved with NASA’s commercial efforts.

Horkachuck: I guess going all the way back, I was working in the [International Space] Station [ISS] Payloads Office. At one point we were looking for ways to bring samples back from the Space Station more frequently than was planned with the Space Shuttle. I had looked at a bunch of small reentry vehicles over the years, and I think I even wrote a short white paper for [William H.] Gerstenmaier [ISS Program Manager] looking at some options that the ESA [European Space Agency] folks had been working on in combination with the Russians. At the time we thought we just needed a small reentry capsule to bring some critical science samples down on a more frequent basis than we were getting.

That went into the ether for a little while, and then I got called to help these guys that were working on this COTS proposal to help write the requirements for the initial RFP [Request For Proposal]. I rolled a lot of the science support requirements into the very top level requirements that they were developing for COTS. At that time there was a requirements document, top-level goals and objectives of the COTS vehicles that were going to be built. It didn’t get down to a lot of real specifics. It was how much total mass do you need, what was some of the top-level functionality that we wanted out of the vehicle, without dictating what the design was. I worked with them over in the procurement bunker for a little while developing the requirements.

At about that same time the Constellation Program was ramping up, so I went off to a job in test and verification over in the Constellation Program [Ares rocket and Orion Crew Exploration Vehicle]. I was one of the managers for test and verification for all the Level II requirements. Then after being there about a year I got a phone call saying, “Well, we down selected to some COTS providers. Would you like to come over and manage one of the COTS companies’ development programs?”

That seemed to be a pretty good fit for me because I always liked hardware development, I tended to do that and enjoy that more than the ongoing operations and sustain functions. And I’d had a lot of projects in the past that were more partnerships than strictly NASA contractor relationships. I worked a lot with the European Space Agency developing a bunch of the freezers that are on Space Station right now, and we worked as a partnership.

Technically they were building them for me, but since it was another space agency it wasn’t like NASA was contracting with [The] Boeing [Company] or some other company to build it strictly for us and we could really dictate exactly how we wanted it built. We had to work in cooperation with their constraints and budgets, and some of their other political constraints, and jointly come up with the best solution.

It seemed like the COTS program was going to be very similar in that respect, where it was a partnership instead of a formal contract. I had a lot of that experience, and I had also had a lot of experience when I was in the Payloads Office. Not only did I manage the whole fleet of the freezers and some of that laboratory support equipment that was used by all the science community, but I was the manager of all the payload engineering and integration. By that I mean I owned all the requirements for how the payloads would interface with the Space Station—both the basic requirements and the verification of those requirements—as well as the integrated stage analysis that we’d do to make sure that when we plugged in that complement of payloads to the Space Station, the integrated spacecraft was within its limits. We’d do a whole module level power balance, data, thermal, acoustics. A bunch of different analyses, so I had a whole team of different discipline engineers working for me.

I did sort of the same thing in Constellation. Although we were trying to use other NASA centers a lot more in the Constellation Program, so I’d have people reporting to me from all different NASA centers with different technical expertise. That fit really well into the concept they were trying to roll out for COTS and having these COTS Advisory Team members, CATs. Effectively, it was a matrixed organization of people that we could go reach into, pulling their technical expertise when we needed help on a particular area or problem. Then they’d go back to their normal day job when we didn’t need them anymore. They weren’t around full time, and that had some pros and cons to it, but in a lot of respects it was similar to a lot of the things I’d done in the past.

Wright: Since you’re talking about the similarities, what were the non-similarities? What was so different now about this new way of doing business with industry partners?

Horkachuck: Well, it wasn’t a contract. It was a Space Act [Agreement], which is NASA’s Other Transaction Authority. We’ve got some unique authority under the original authorization to create NASA [National Aeronautics and Space Act of 1958] that lets us do these Space Acts. That has a big benefit in that you can get things done a lot faster than you can in a normal contract.

Quite frankly, I think our normal FAR [Federal Acquisition Regulation]-based contracting system is broken. It’s so out of date and so slow and antiquated that it’s, I think, a big part of the reason that not just NASA, but all federal agencies have such massive cost overruns and complete screwups on contracts. It’s just so antiquated, it doesn’t work anymore. There have been cases where we’ve tried to make a change in requirements on a contract, and it’s taken the pricing and proposal and contracts folks a year to get the proposal back on making a change. By then it’s too late. You’ve wasted a year of your development program dicking around with pricing and negotiations of the cost.

It’s just so slow that it becomes a major burden. Your cost overruns go through the roof because you’re still designing it to your old baseline, which you know is wrong, but you’re still wasting money doing that because you haven’t been able to authorize what you know is right, because they haven’t gotten around to doing the pricing for you.

Wright: With today’s technology you have new materials as well.

Horkachuck: Yes. If anything comes up new, you can’t implement it because you don’t have it on contract.

Wright: So you found not using the FAR system to be a more liberating way of doing business?

Horkachuck: Oh yes. Since it was a partnership, they went in with their eyes open knowing that we’re going to do our best to determine what the basic top level requirements were. We weren’t going to go down really deep into the requirements, but they knew what the end goal was, so if there were changes in things that had to be done, if we could convince them that technically it was the right thing to do, then they just went and did it. There was some technical convincing that would have to happen sometimes. You’d have to have a discussion on the merits and why a certain change needed to be implemented, but once you got through the technical convincing, they just went and did it. It wasn’t a one-year monotonous, gory fight over “It’s going to cost this much,” “No, it’s not going to cost that much,” “It’s going to cost this much.”

That was part of what their risk was going into the partnership. We came in with a fixed amount of money, and they had to come up with the solution that met the top level requirements, and how they got there was their business. We’d help them along the way and give them as much technical expertise as we could, but in the end it was their responsibility. They knew that either they were going to come up with a solution that was going to work, or they weren’t going to get the follow-on contract.

Wright: As we walk through your first year, talk about when you came on as part of this team. You became the [NASA] Project Executive for SpaceX [Space Exploration Technologies Corp.]—some of those first encounters with them, and how you started laying out that foundation to move forward with this new way of doing business.

Horkachuck: They were really small back then. In fact they were in small buildings in El Segundo [California]. They were in multiple different buildings. They had some of their manufacturing and avionics in a different building, and some of their engineering in a different building. It was spread out. They’d still only been developing a Falcon 1 rocket, which was a single-engine small rocket that was supposed to take maybe 1,000 pounds to orbit.

We wanted a much larger class rocket in order to take the capsule to orbit, so they had to develop a whole new rocket, and the whole capsule. They’d been launching out of Kwajalein [Marshall Islands], which is way out in the middle of the Pacific, Reagan Test Range [Ronald Reagan Ballistic Missile Defense Test Site], a series of small islands and an atoll. The Air Force wouldn’t let them launch out of Vandenberg [Air Force Base, California] for risk that their rocket wouldn’t work and it would destroy some other national asset, some other [launch] pad for an Atlas or Delta [rocket].

It was a big leap for them in those early days. They had a lot of really good smart people that were very enthusiastic, but they had a lot of work to go. They had some very early concepts, and they seemed to do a really good job of coming up with concepts and developing the concepts a little beyond what we would normally expect at a PDR [Preliminary Design Review] level. They actually would start building the hardware and testing some of the components even at the PDR level, then they’d roll into a CDR [Critical Design Review] without the level of maturity that we’d sometimes normally see.

They wouldn’t have all the drawings released, although they were trying to move into more of the automation and electronic data and not do as much paper drawings and design. That worked to some degree. I think they are learning that when you’re still manufacturing parts there still needs to be some amount of paper trail there. They’re finding where that balance is today—how much can they do purely electronically, and how much do they still need to go from the 3D CAD [Computer-Aided Design] models to actually making drawings for the individual parts that they’re building.

The advantages—they were a lot like the Lockheed [Martin] Skunk Works [Advanced Development Programs]. It was a separate brand-new company, and it didn’t have a lot of the baggage that a lot of the bigger existing companies have on processes and how to do business. So they could innovate in what processes and systems would work for them—which were lean and efficient and would help them get things done quickly, and which ones were burdensome.

They’re still, I think, learning some of that. They’ve made a lot of progress today, they’ve come a lot closer to what we’re used to than where they were when they first started. Having that clean Skunk Works organization gives you a lot of ability to innovate and do things more quickly and more efficiently.

Wright: How were you involved in those first months with them? Were you in residence with them helping them along the way?

Horkachuck: Yes. We’d have milestone meetings on a fairly regular basis in the early couple of years. Then, in between those, I eventually set up weekly meetings with their lead engineers for the rocket and for the capsule. We’d have regular tag ups to see what progress they were making, what areas they were doing well in, how they were progressing to schedule. In fact, I spent a long time trying to get them to actually build a schedule, because they had really no idea how to lay out a big project schedule when we started. They’ve learned a lot over the years.

Wright: You mentioned being able to share NASA’s expertise with them as they were formulating those plans in those first months. Can you give us some examples about whether you and/or your advisory groups—some examples about how NASA was involved in those days. Did you help with actual development of the spacecraft, or things other than helping them develop schedules? What are other examples of what you did?

Horkachuck: One thing I did was I tried to coach all the technical experts. Early on I tried to hand select a lot of the folks that we were using as advisory team members because I’d worked with a lot of them in the past, whether it was on the Space Station Program or on Constellation. I tried to pick folks that I knew really knew their subsystem and technical area, and were reasonable to work with.

What I didn’t want to do was get a bunch of people that were strictly by the book. “The requirement says this, and you have to do it this way because that’s what the requirement says” types. I’m one of those people that asks, “Why is the requirement like that? What is the end goal that we’re trying to get to with this requirement? Isn’t there another way to get there if we need to?” The point of most requirements is to make sure that the system works the way that you want it to. The way we sometimes word requirements is a combination of a lot of history and “That’s the way we’ve always done.” That doesn’t go very well with innovation, so I tried to get people that really knew their stuff and their system, and then I’d coach them and let them know that this is different.

It’s not the typical NASA-contractor relationship where you can’t tell the contractor how to do business for fear that you’re violating some contractual requirement. “Giving a constructive change” is the parlance I think in contracts, where if you tell them, “You have to go do this,” they may go do it, and then the government is liable for the cost and takes on not just the cost liability, but also the technical liability if it doesn’t work.

I would tell the NASA people, “This was a partnership. You can tell them what you see the issues are, and explain to them some of the things that you learned in the past, what’s worked in space and what hasn’t worked in space and why. You can even suggest how they may want to do it, and they may take that and they may not.”

I’d say, “They’re going to make some internal corporate cost trades and tradeoffs on what’s the best thing to do is for them. That may not necessarily fit with the way that you’d recommend, but they’re trying to do things a lot more cost effectively than we have in the past. Quite frankly, cheaper. They may come up with a different way of doing it than you suggested, but they’re going to listen and they’re very receptive to understanding from our history and what’s worked and what hasn’t worked in the past.”

I tried to coach them into a more open and friendly dialogue from the start, when I got them on board the teams. Sometimes that worked, sometimes it didn’t. There were some CAT members that I had in the early design reviews that didn’t work out. I still had to use some methods that we’ve done in the past. We call them RIDs, Review Item Discrepancy. You’d write up a little discussion, on a form, of what you see that may be a problem with the design and why you think it’s a problem. And then we’d have to go through that issue, understand it and how to mitigate it as we did the rest of the review. The big ones went up through the board and were reconciled.

When I started getting a bunch of RIDs in from somebody that were just picking at the paperwork and the wording of something, I’d try to coach them that we weren’t trying to get to that. We were trying to get down to the meat of the technical areas and we weren’t so much worried about “Is the paperwork perfect?” If they continued down that route, I didn’t invite them to the next review. So there was some turnover in the CAT teams over the years. Some good ones moved on to different projects, but some of them just weren’t a good match.

Wright: Was their participation voluntary in the sense that they did it as part of their normal job? Or did they become a COTS team member that charged to a code to come work as part of COTS?

Horkachuck: We had a charge code.

Wright: Was it that if you asked them they could, but if they didn’t want to that was okay too? There was no mandate that they needed to have to do that?

Horkachuck: Yes, we didn’t have that big a stick. At the time that we started, there were a lot of major projects still going on at the Center. The Shuttle was still flying, so the Shuttle employed a lot of people. The Space Station was still going full-bore in assembly and had a lot of people dedicated to it. Constellation was ramping up and still had a lot of people. So it was really hard to get people. It was hard to get people at JSC to give us the time of day.

I think the only reason early on that we did get some of the experts that we did was because I knew them from the past and they had worked with me on previous projects, and they found this interesting and challenging. But otherwise they were dedicated full-time to other programs. The JSC Center didn’t give us any allocation of people, so we were more or less forced to go to other NASA Centers for a lot of the help. That’s good and bad.

There are a lot of smart people throughout the Agency, but there’s a unique skill set in manned spaceflight that you don’t have in say some other research or unmanned centers. It just that that’s not where their primary expertise lies. If I got a recommendation about an acceptable material from somebody at an unmanned center, it could be wrong because they don’t know the toxicology requirements or the flammability requirements of manned spacecraft. So a lot of times, I’d have to temper the responses I got from some of the other organizations. That’s why it was really critical to be able to pull out some key folks at JSC and at [NASA] Marshall [Space Flight Center, Huntsville, Alabama].

Wright: I’m curious, was there a type of broadcast system—you knew what experts to tap into that you sent a request for help, and then got responses? How did you tap the people at the other centers?

Horkachuck: We compiled a list of names on a spreadsheet. First we looked at each of the reviews. Myself and my deputy Warren [P. Ruemmele] and one of our contractor support people would sit down and look at what review was coming up, what the major topics of the review might be, and what kind of technical discipline support we had. We’d get a feel for what documents we were going to be getting in from SpaceX for that review, then we’d map out each one of the documents to a set of skills that we wanted it to be reviewed by: structures; thermal; power; avionics folks; guidance, navigation and control; some operations folks; maybe some ground ops [operations] folks. We’d map the review to the types of technical disciplines that we wanted to have for the review. We assigned each one of the documents to at least one, if not more than one, of those technical disciplines, so we knew we were getting coverage on each one of the documents submitted.

We had acquired over time this list of people that we could call on and I’d send out an email to them en masse asking if they’d be available for a review during this period of time that was coming up and get responses back. If we got enough people in each one of the technical areas then we were okay. If not then we’d go out to some of the other centers and ask for additional help in specific areas.

It took a while, but we eventually got a main point of contact at each one of the different NASA Centers. They’d be able to go out to their engineering directorates and pull some of the technical support that we needed. But that took a little while to get going. It’s a well-greased wheel now, but it wasn’t in the beginning.

Wright: Just identifying people had to be a task in itself, trying to find the right people at the right place, then having them carve out time to help you.

Horkachuck: A lot of times they were in the middle of another review that their management said was more critical. There was a major design review for Orion going on, and they didn’t have time to support us because that was where the Agency’s focus was. Flying out the Shuttle safely, continuing assembly of the Space Station, and then building up the Constellation Program, Ares and Orion, were the Agency’s focus. This was the back burner project. It was the side bet, literally, is what it was. I think at one point I heard that the [NASA] Administrator [Michael D.] Griffin was quoted as saying, “We’ll put in $500 million—no more, no less—to this COTS commercialization idea and see how it goes.”

Wright: Look what happened. When do you think it didn’t fit into that term anymore, being a back burner project or being a side bet? What was the tipping point that moved you out of that spot?

Horkachuck: Continual schedule slips of Constellation, Ares and Orion delays. Kept on slipping further and further out to the point where it wasn’t going to make it to Space Station. Shuttle’s retirement was becoming more and more imminent and was more of a firm date. The things on the outside moved around us and we were still tracking.

Granted, we originally were going to be a three-year development program, and it almost took twice as long to get the flight off to Space Station as what SpaceX originally proposed. I told you they had some difficulties with scheduling early on. They were very optimistic on their schedules, very aggressive in what they thought they could get done in a certain amount of time, and had no schedule margin built into any of their schedules when they built them.

I think, when we even signed the agreements, we knew that their schedules were ridiculously optimistic, and assumed it was going to take quite a bit longer than what they had proposed. But that was okay. As long as they were making good technical progress I think we were happy, because it wasn’t going to cost NASA any more.

On a typical government contract, if the contractor is late and the schedule for a development program drags on by a year, you’re paying a lot more money. It’s not just that the rocket takes longer to build; you’re paying for a standing army of people that are on that project for another year. So you’re paying for maybe 10,000 people at $200,000 per year. That’s some cost overrun compared to what you originally thought.

In this case, NASA wasn’t liable for any of those cost overruns. We had a fixed milestone-based agreement. When they met a technical milestone, we reviewed it. If they met those technical criteria, then we paid them the amount that we agreed on for that milestone. If it took them longer to get to that milestone, we didn’t pay them any more. It was just the amount that we had agreed on. Any delays were costs that they had to incur on their side, it was their contribution to the project.

Wright: Things were running pretty much on schedule with your reviews when you first started, is that correct?

Horkachuck: Early on. The first couple years I think were pretty much dead on.

Wright: When it got to the point where [milestone deadlines] started to slip, can you share with us—not specifics of the conversation, but the conversation as a whole—how to get them on a track to move forward? Were there concerns on the NASA side that since they had begun to slip that it’d continue slipping, maybe not reach fruition?

Horkachuck: Well, SpaceX was really scared that we would terminate them if they missed a date in the early years, because the agreement said, “You’re going to have this milestone in this month in this year.” It didn’t give you a lot of flexibility; it said you had to do it on that schedule. We laid out a schedule for the milestones partially so that we knew what the funding profile would be for our internal budgeting, just so that you had some kind of general guideline for what we were doing.

There wasn’t a lot of language in the Space Act that said if you miss a milestone what happens. I think what it said was if you missed a milestone, NASA had the rights to terminate the agreement. So they were really scared that if they didn’t meet a milestone on a given date we were going to just end the agreement with them. I don’t think even on the NASA side we knew exactly how to deal with some of that early on, the first schedule delay.

We terminated Rocketplane Kistler mostly because they were making no progress. It wasn’t that they had missed some milestones. I think they even rewrote the Space Act at one point and let them combine some milestones to try and help them, but they just weren’t making progress. In the case of SpaceX, they were making progress. They had made a tremendous amount of progress through the early design reviews.

But when you get to the hardware build and assembly and test phase, things happen. It’s just the nature of actually designing new hardware and building new hardware and software. When you start integrating all those things together, some things work that you thought were going to work, and some things don’t work that you thought were going to work. Then you have to go redesign it and try again. That leads to schedule delays.

We were getting to that phase of the program, but they were still making a tremendous amount of progress. I had lots of insight into how they were doing technically on all the different subsystems, but they weren’t exactly on the schedule that we had laid out in the beginning. So we had some long conversations here with our folks and legal to make sure that if they were showing good technical progress, was it okay to let them miss a milestone date? Then if they met the technical content of that milestone at a later date, was it okay to still pay them for the milestone? The first couple times there was some wringing of hands. We wrestled with that quite a bit.

Wright: Were the eyes of the Agency beginning to look at what you were doing at that time period? Or were you still on the back burner when you were starting to make those decisions of letting them stretch that date out?

Horkachuck: At that point we still weren’t too far in the limelight, so I think they weren’t all that worried about us. They had much bigger problems to worry about.

Wright: Were you given the authority, based on your position, where you could make some of those decisions to allow them to have leeway one way or the other? Or was part of your responsibility to come back and get that authority from here within your program?

Horkachuck: Alan [J. Lindenmoyer, Commercial Crew and Cargo Program Manager] had the real authority, although I think he needed to still check with his boss and with the rest of the NASA community, legal and other folks, to make sure it was okay. I’d make recommendations based on whether they had met the technical content of what the milestone was, did I think they were making technical progress. I was basically making recommendations on where we should go with things. I didn’t have the ultimate decision making authority, Alan had the final sign off on those things.

Wright: We’re learning more about how the partnerships work and how traditional protocol was set aside on some things, but it sounds like on this it still went up the ladder.

Horkachuck: In a lot of cases we still followed a lot of the protocols that were done in the past. Even the proposal and procurement process, although it was streamlined. It was done a lot more efficiently than a normal procurement. They followed a lot of the same procedures and protocols and did a lot of the things that you’d do in a normal FAR-based procurement, even though it was a Space Act Agreement and they didn’t necessarily need to follow all those rules. Just to make it clean and not show even an inkling of favoritism or being unfair in the selection process. They tried to follow the typical process in a lot of respects so that it, by its nature, made sure we were fair and equitable.

Wright: As the process began for the development phases, how was the safety community within NASA involved? Or even the safety folks from the FAA [Federal Aviation Administration]. Can you share with us on how they were brought in, or not brought in, and how they impacted the vehicles’ development?

Horkachuck: Right at the beginning of the project, we had a Chief Safety [and Mission Assurance] Officer that was resident with us. In fact, Mark [D. Erminger] lives right across the hall from me, so I’d have him coming to all our reviews, giving me his observations as we were going through different design reviews and understanding where they were and what the risks were. He’d sometimes call directly to the contractor and talk to their safety folks, get their Failure Mode and Effects Analysis. At some point I think he did a Probabilistic Risk Assessment, or got the contractor’s data on that.

He was really helpful, and he was reasonable. He understood that there were trades that had to be made in order to actually get anything done, so he was aware of what those trades were. I tried to explain behind the scenes what was going on technically, and why there may be certain issues where it wasn’t strictly the way we would have done things in the past. He’d weigh that and balance it with what his understanding of the overall agency’s safety philosophy, and come back with recommendations in some cases, and just agreement in other cases.

Wright: Did the influence or impact stop here on his level? Was the NASA Headquarters [Washington, DC] safety involved, or the FAA?

Horkachuck: As time went on we started getting help from Headquarters. They requested to participate in a lot of our reviews. It was okay, they were mostly in listen mode. I didn’t see a lot of influence that put big roadblocks in the development program. They were interested to see how we were doing things, interested to know that Mark was an active participant. He was reporting directly up to the Headquarters [Chief] of Safety [And Mission Assurance] at the time, Bryan [D.] O’Connor, so he was passing on a lot of the information.

They were more worried if we were going to go implement a crew program, because early in the project we were both cargo and crew. The idea was we were going to start out, see how they do with cargo, and if they did a pretty good job with cargo then we’d turn them on to do commercial crew as well. In fact the SpaceX Space Act Agreement has an Option D in it that would have let us turn on the crew program, the crew Dragon [capsule]. We just hadn’t exercised it at the time. The outside organizations were much more interested, if we were going to try to fly crew on board the vehicles.

Then, as part of normal safety, as we started to have missions that were going to go to Space Station, the Space Station safety process started to take over the normal phased safety review process. Phase 0, 1, 2, 3 reviews, hazard reports and going to the Safety [Review] Panel. A lot of those safety requirements were Space Station requirements for any vehicle that was going to rendezvous and attach to the Space Station. There’s a certain number of requirements that you just have to have in order to make sure the vehicle and the subsystems on that vehicle are safe, and not going to damage the Space Station or any crew member on board the Space Station.

Wright: Were you the main liaison between SpaceX and ISS to make sure that that was covered, or were there other people involved with that?

Horkachuck: I did some of it. Originally our office had a position for that, but Station really wanted to be the primary point of contact for most of that, so I just worked with their folks. A lot of those processes were standing processes in place, so once I got the SpaceX folks to learn how to play into them I didn’t need to be directly involved. There were too many different things going on all at the same time for one person to be a bottleneck of information. There’s no way I could have been the liaison in the middle of all that, as well as making sure the development was going right, and on schedule, and we were getting payments to them.

Wright: It’s another nontraditional way of doing business to allow the flow and not have to go through one central point

Horkachuck: To actually trust people to know what they’re doing. They always knew they could call me if something didn’t seem right and they were getting into a problem that they couldn’t get out of. Then I could talk to both the NASA side, whoever was raising the concern, and understand, mediate our way through it. It wasn’t all that frequent, quite frankly. Once in a while we’d have to do that.

Wright: If I understand correctly, going to the Station was not mandated. That was something that SpaceX could choose to do, is that correct?

Horkachuck: I think that was the way it was originally written, although the whole point was to be able to take cargo to the Space Station, so not going to the Space Station—I guess you could not do that, but I don’t know what the point would be.

Wright: So far as you understood, that was always their intent, to bring cargo to the Station, from the beginning of their development phase.

Horkachuck: We’d given them information on what size packages they were going to take and be able to accommodate, and how much power you’d need to give to powered payloads, and the kind of cooling you’d need to give to powered payloads. They were all directly from our requirements that we had from EXPRESS [Expedite the Processing of Experiments to the Space Station] racks that were on the Space Station, and cargo bags that we’d normally fly in MPLMs [Multi-Purpose Logistics Modules]. It was all directly logistics for Space Station. They could not go into Space Station I guess, but they would have never gotten the CRS [Commercial Resupply Services] contract.

SpaceX had three flights in the basic agreement. The first flight was just an orbital flight. Prove that the rocket works, prove that the capsule can get to orbit, can maneuver in orbit, communicate in space. Prove that their thrusters worked to make it point at a target, and that it could reenter, that the heat shield worked and the parachute system worked.

Quite frankly that was the one that worried me the most, the first flight. They had one flight of the [Falcon 9] rocket before that COTS 1 [demonstration] mission, then the flight with the Dragon on board that was actually going to maneuver in orbit. Did their thrusters that they built themselves work, did they communicate with the TDRSS [Tracking and Data Relay Satellite System] satellites correctly? Then did the heat shield work through reentry, and did the parachutes work?

I influenced them to do some more parachute testing than they had planned in the past. They had tried to leverage on the Orion parachutes, buying them from the same company that was building the parachutes for Orion. Then the Orion schedule dragged out so long that they never finished certifying and qualifying parachutes. So now SpaceX was left out in front all of a sudden, without the budget to do the full qualification program.

We suggested they may want to do some things like some more drop tests, some more parachute tests. They eventually did a helicopter drop test. We learned from some of the things that the Orion folks did, dropping out of the back of airplanes, and how some of the systems to stabilize after you come out of the airplane cause more problems than the actual test itself.

We tried to keep it a little simpler. Some folks would say that didn’t prove as much as you necessarily could from some other tests, but it showed the basic systems worked. They tried to keep it fairly simple. Simple release of drogue parachutes, and then have the drogues pull out the mains and make sure they were in a good packaging, would inflate well. The capsule comes down really slow if all three mains are inflated; it’s a very soft ride. Even if one or two of the mains is out, it’s still coming down slower than Apollo or Orion.

Wright: That’s interesting. Share with us what it was like on that test, the whole launch aspect. You said you were concerned for that first one because there was so much that had to go right. You really had put in a lot of investment, your technical expertise and your time with them. If you can just share with us what it was like to be there, and know that things were launching and working the way you wanted them to.

Horkachuck: Well, obviously it was nerve racking. The uphill ride is always really worrisome. Getting a rocket into space—those first nine minutes or ten minutes really worries you, because lots of things are going very fast. There’s not time for something to go wrong and still have a very good outcome. So that was nerve racking. We did a lot of things to “buy insurance” is probably the way to put it.

Warren and I scoured the Agency—and other agencies—to find assets to help us have some additional insight into the demo [demonstration] missions, like bringing up some of the Shuttle radar systems. Some of the debris radar that they had implemented after Columbia [STS-107 accident] to look for foam coming off the Shuttle, we train those same radar on Falcon 9 for the ascent phase. If something goes wrong, we would have very high quality images of what might have been coming off the vehicle, so we’d be able to do a reasonable investigation if we had to. Luckily we didn’t have to.

We sent the SRB [Solid Rocket Booster] ships up the coast, and out farther than they’d gone before on a Shuttle mission, so that we could optimally deploy them for staging events and other events. We looked at the history or rocket launches with The Aerospace Corporation. Where do you typically have failures of rockets? A lot of the events, when things are changing, is when things don’t seem to work right. So we tried to deploy assets, both optics and radar, to be able to see those specific events.

Then we briefed all the way up to the [NASA] Associate Administrator [for the Exploration Systems Mission Directorate] what our plan was. We’d even worked with a group out of [NASA] Langley [Research Center, Hampton, Virginia] called HYTHIRM [Hypersonic Thermodynamic Infrared Measurements] that has used some Navy P-3 Orions [aircraft] with some optics on them to look for Shuttle reentries. They started looking at what happens if some of the tiles aren’t in place right, or some of the gap fillers are sticking out.

We used a lot of the same assets that they had to watch the reentry of the Dragon capsule, so that if something went wrong during reentry we at least had a chance of figuring out what the heck happened. You’re in a data blackout during reentry, so you’re not going to be getting telemetry back. You might get lucky and be able to recover some data recorder out in the middle of the Pacific, but in order to have good insight to what was going on we deployed a plane to go look and watch the reentry, especially during the max [maximum] heating phase. Luckily things went really well.

Wright: Doesn’t sound like too much luck. Sounds like a whole lot of preparation, a lot of analysis.

Horkachuck: We did a lot of analysis, and we did a lot of work with NASA experts. But we tended on this project not to go as deep into any given technical problem as maybe we would on a manned mission. We didn’t look at all the corner cases to the absolute extreme; we didn’t do a bunch of ground-based testing of every single thing. They did a bunch of arc jet testing of the basic material. [NASA] Ames [Research Center, Moffett Field, California] had actually developed the main heat shield material, something called PICA [Phenolic Impregnated Carbon Ablator], for some unmanned missions. They worked with SpaceX to design the heat shield, and used some of the Ames code to figure out what the heating rates would be and how thick the material would need to be.

SpaceX built a lot of margin into their design early on, so that if they didn’t do some calculation just right they had some robustness in the design. Granted, it added weight to the vehicle, but it gave you some flexibility for changes that inevitably happened. Or finding out that maybe they hadn’t designed to the right requirement for some structural margin for Space Station, or some load case that they hadn’t thought of before.

They had enough robustness in their design that they didn’t have to completely scrap the design and start all over again. They could redo the analysis and show that they still had margin. That saves a lot of money in the project development cycle because you’re not scrapping out the hardware design and starting all over again.

You don’t completely optimize performance, but I like their general philosophy where you retain some margin early in the first couple flights, and then you maybe redesign later to gain more performance. As opposed to trying to optimize everything right from the start, and you spend a lot of time and a lot of analysis trying to optimize, and then something from outside your influence throws you a curveball and you have to go redesign the whole thing anyway because you have no margin, you designed right to the limits.

NASA tends to do that a lot. Not sure it’s necessarily all that efficient. Certainly builds you a spacecraft that is a Ferrari as opposed to a Volkswagen, but if all you need to do is have the function of a Volkswagen—

Wright: You’re giving credit for that decision to SpaceX, so that was their decision to not fully optimize and retain that margin with their design?

Horkachuck: It’s their general design philosophy. They tended a little bit more towards the way the Russians do business. They design something, and test it, and then see how it worked, and then make tweaks, and build it again and test it again. They used a lot of state-of-the-art computer code to design the hardware, don’t get me wrong, but they left some margin in there and didn’t try to overoptimize right off the bat, which I think is good.

You have to have a booster that’s going to be able to take that extra mass to orbit, and I think we’re paying for a little of that right now. They’re working on the next generation booster that’s going to give them a lot more performance to orbit. Depends on where you want to get to and if you’ve got the flexibility to be able to continue development and make changes.

There used to be somebody at Headquarters that was trying to do what they called spiral development. [Rear] Admiral [Craig E.] Steidle had proposed that in the Agency. I don’t think most of the Agency knew what he was talking about, where you design a little, test a little, fly a little, redesign, test, fly. You keep on iterating the design and evolving it. Not trying to come up with the optimal design right off the bat, which is virtually impossible to do. But we try to do it on almost every damn project, and it shows in how some of the projects go.

Wright: After that first test flight, were you involved and invited to sit down with them to hear their debriefing and their evaluation of what they felt went right and went wrong?

Horkachuck: Oh yes. The flight itself was a milestone—successful demo flight. We had determined a set of mission objectives for that demo flight. We jointly agreed on them and had a document that defined what those top level objectives were. It was short, maybe a couple pages. The major elements of the goals for that flight I think were on one page. In order to get payment they had to show that they had met those major objectives. So we had a debriefing two days after the flight, where they showed proof that they had met each one of those objectives.

Then a couple months later, when they had gotten all the data back and processed all the data, we had a much more detailed debrief for both the rocket and the capsule. We went through what systems worked and what systems didn’t work, and how they were going to fix the ones that didn’t work for the next flight. Or if they were making such significant changes in the design from the first demo flight to the second demo flight vehicle that it didn’t matter anymore. The vehicle changed significantly from the first flight vehicle, primarily because it had different mission requirements.

Wright: Which leads to a question—what measures did you have to take to maintain the level of confidentiality that NASA and SpaceX have agreed to on the level of this partnership? We understand there’s so much of what they do that you’re privy to, but you don’t discuss outside of your NASA counterparts. I’d like for you to give us your definition of how the proprietary information, or confidentiality of their technical expertise, stays within their own environment. It’s not a traditional role of NASA when they develop spacecraft.

Horkachuck: Yes and no. In a typical contract, NASA is paying for the design and basically buying the design and the vehicle. So in that case we more or less own the design. We’re paying a contractor to do it, but we own the majority of the rights to it. There are some areas that are still sometimes corporate proprietary. If they have come up with a certain process or material through their corporate IR&D [Internal Research and Development] funding, or some other method that is unique to them, that you can’t share with anybody else.

Typically the government would own the design because we paid for it. In this agreement, SpaceX retained a lot of the rights because they were putting in a lot of their own money into the development. As part of the agreement, it lets them keep a lot of the rights to it unless they for some reason default.

We tried to keep it fairly tight, and remind anybody that was working on the project that they can’t share it with other contractors. NASA has so much contractor support, especially on big programs, Space Station and Shuttle Program. We flew a couple [SpaceX] DTOs [Detailed Test Objectives] on the Shuttle and some other hardware on the Shuttle.

You have to work with some of those contractors during the integration process, so we had to get Non-Disclosure Agreements signed with those individuals or their corporate to make sure that they’re not using any knowledge they have on the SpaceX design for some other vehicle that maybe they were designing.

In some cases they were really worried about Boeing, because Boeing was a potential competitor on the crew vehicle. Boeing is the prime integrator for the Space Station, so we had to firewall off some of those people from working on any proposals for a crew vehicle for Boeing. That was all done with the contracts folks.

In general we tried to keep the data in a secure repository that had very limited access to just people that we had cleared for the project, had a right and a need to know and had signed agreements either themselves or with their corporation on that particular contract that had the nondisclosures in there.

The other aspect that we had to deal with is ITAR [International Traffic in Arms Regulation]. It’s a launch vehicle, so got to make sure that we’re only dealing with U.S. persons, because you can’t share ITAR information with foreigners in any sense. So we had both of them coming at us like a double whammy to make sure that we kept things close to the vest.

It made it difficult because it was hard to get data transferred between people when you needed it to be transferred to certain people, trying to follow the limitations of how they’re supposed to handle that type of data.

Wright: I can’t even imagine. In December 2008, SpaceX was awarded one of the two Commercial Resupply Services contracts. What role was yours in that process? Did you have one?

Horkachuck: I didn’t have a role in the process. I was just trying to make sure that we built and demonstrated the basic capability. They had a whole separate group doing procurement for the services phase of the contract. I knew that ours was the demonstration phase, and if they did the demonstration phase successfully there was a follow-on services contract. I was just making sure that it worked, then I figured the rest of it was going to take care of itself.

Wright: From what we understand it did work. There were other pieces of the demonstration phases. Would you like to talk about the other flights and the roles that you played with moving those forward?

Horkachuck: There were supposed to be two other [demonstration] flights, a total of three flights. The second flight was supposed to take the capsule to orbit, be a more capable capsule. It would be in space for nine to ten days, and really let a lot of the systems start to see how well they worked in space over a long period of time, through lots of thermal cycles and orbits around the Earth and day-night cycles. Just see how well they tolerated the radiation environment.

The surface objective was to do a flyby of Station and close the comm [communications] link with Space Station. There was no comm system on board Space Station that would talk to a visiting vehicle other than the Japanese [Japan Aerospace Exploration Agency] system that they had built for the HTV [H-II Transfer Vehicle].

The ESA ATV system came up through the Russian side of the ISS vehicle. We had thought about that briefly, but then you were berthing to the Russian side of the vehicle. We didn’t know if the Russians were going to charge us a docking fee, and you had to go through the much smaller hatch that was available on the Russian side. Ruled that out early in the program.

I guess I influenced SpaceX quite a bit. I had worked with the Japanese quite a bit on Space Station, and although they’re very good technically, they are very slow and methodical in their decision making process. I knew that working with the Japanese team probably would clash with the SpaceX culture and way of doing business. SpaceX wouldn’t understand why it would take three or four weeks to make a decision on some simple question. Whereas the Japanese culture would normally go back and have some team huddle offline before they came back and gave you a real yes or no.

Quite frankly, I thought that the basic premise of COTS was to develop a U.S. capability to take cargo to orbit. SpaceX was building a vehicle completely in the United States. They were building their own engines, their own tanks. The rocket was a U.S.-built vehicle, their capsule was all built in the United States. I found it disconcerting to think that we were going to build all that capability, and then have the last mile getting to Space Station dependent on a foreign government.

The idea behind COTS was to make sure that we weren’t dependent on foreign governments anymore. We had the Shuttle, which gave us this huge capability and was a tremendous asset. Then the only other way to take cargo to Station was going to be HTV and [ESA] ATV [Automated Transfer Vehicle], foreign governments or the Russians. I think the idea behind COTS was to build a capability in the U.S. I didn’t want us to be held hostage for the last part of the mission because we didn’t have a communication system.

I started working with the Space Station Program and the comm folks. It seemed to be too much to go put a new antenna outside Space Station at the time. Space Station wasn’t paying that much attention to the COTS program yet, and having us do an EVA [Extravehicular Activity] to go install an external antenna, and either finding penetration through the pressure shell or making a new penetration just didn’t seem like it was in the cards.

So we worked with the comm folks and decided that we could use the UHF [Ultra-High Frequency] system and antennas that were already there to talk to the spacesuits, and build a comm system that would be able to talk to the visiting vehicle. It had a lot of drawbacks because it was UHF, and it was a shared frequency band so we had to deal with the FCC [Federal Communications Commission] and the NTIA [National Telecommunications and Information Administration] for regulations on how much power you could communicate out of the Space Station.

It was originally installed as just being for communications to spacesuits and very close to the Space Station, very limited power. Whereas we had a vehicle coming up from a far distance that we wanted to be communicating with at least 20 miles away, so you had to turn the power up. There were lots of complications and negotiations with the powers that be—both the frequency managers here at JSC, and at Headquarters, and then the whole rest of the government regulatory process—to be able to use the comm frequencies.

But in the end it gave us a capability to have a comm system that wasn’t a foreign government’s comm system. We ended up launching two boxes that SpaceX developed in middeck lockers. I built a bunch of middeck lockers for all the payloads back in my days with Space Station. So I loaned them a couple middeck lockers to install their comm system into, because it got us away from a bunch of structural analysis and safety hassles.

It integrated easily into some EXPRESS racks that were already on orbit. Then we hooked up the wiring. You need cable harnesses, so the crew had to go lay in some new cabling to tie into the existing UHF system. We ended up with two boxes on board Space Station that talked between the Space Station and the upcoming SpaceX vehicle.

The objective of the second mission was to close the communication link with that new system on board Space Station, make sure it really worked, before we came in a lot closer to Space Station. Part of that system had a bunch of switches on a switch panel for the crew that if it looked like the vehicle wasn’t following the right approach corridor to the Space Station, they could hit an abort and it would send a command to the upcoming SpaceX vehicle to abort, fire its thrusters to back away from the Space Station.

So it was very important that that comm link worked. It gave us data on range and range rate, information about the vehicle approaching to make sure it was still okay to continue the approach. That was what the big ticket item was for the second demo mission. Go check it out before you fly a final approach on the third mission, which was to finally get all the way up to the Space Station, berth with the Space Station, and then demonstrate that you can transfer cargo in and out.

We had some problems with the MPLMs on the early flights, where getting racks out of the MPLM was pretty easy, but getting them back in for return and getting them bolted back down turned out to be a little problematic. There’s some things that “the best laid plans” and thoughts about the design—when you finally get into space, you realize that things expand a little bit because you’re in an airless environment and you’re pressurized. Well, take the racks out, they don’t go back in the same as when they came out because things move. You’re optimizing design for weight, but then the structure isn’t as rigid and changes a little.

We wanted to prove all that out in the final mission, that it could do all the basic functions of delivering cargo to and returning cargo from the Space Station without issues. Partway through the program, SpaceX approached us and asked if it would be possible to combine those two missions, the second and the third demo mission. At first we didn’t receive it very well. It seemed like they were looking for ways to just save money by eliminating a flight, and adding a lot more risk.

In the end we did a lot of work with them to do some integrated system-level tests. A lot of things like full-blown thermal vacuum test of the capsule, a full-blown EMI [Electromagnetic Interference] test of the vehicle. A lot of deployment tests of a lot of their mechanisms, the solar arrays, their door and grapple fixture. A bunch of testing of the LIDAR [Light Detection and Ranging] and proximity ops sensors that were going to be used on that final approach to Space Station.

We added a bunch of new milestones and tests into the overall program that they didn’t have in their original plan. As we added all those more traditional tests that NASA would have done on a system, we got more comfortable with then being able to combine the two missions and not being so risky that it was just a throwaway mission. We came to another fairly reasonable compromise in my opinion, doing some things to reduce the risk, but still meeting some of their objectives.

It also saved schedule and manpower for SpaceX, since they had been taking quite a bit longer. They were running into some financial issues. As I said, they were still having to pay for this standing army, more or less, that was working on the program, even though they weren’t meeting new milestones and getting new payments. So it helped them in that they only had to design one more new vehicle.

There would have been slight differences between the design of the C2 vehicle and the C3 vehicle, so it saved them a little bit of nonrecurring engineering time, as well as just time to complete. Although that’s a double-edged sword, because the flyby mission wouldn’t have had as much insight and verification that the Space Station folks required of it as the mission that came all the way up and approached Space Station. They had to do a lot more work to get Station comfortable that their vehicle was safe and ready to fully approach Space Station. So, that made the next launch seem like a long time from the first.

Wright: Were you surprised, or did you start to see some hints that they were going to want to combine those two missions?

Horkachuck: They’d been hinting at it for a while I think. I wasn’t sure it was a great idea at first, seemed like the benefits to NASA weren’t all that good when they first proposed it. They did caveat it, always saying that if for some reason they were unsuccessful they would go fly another mission and try again. We were still getting what the original value was out of the original agreement. We’d still, if need be, get three flights. Turned out this got us there a little bit faster than if we had had to really need three flights.

Like I said, we did a lot of work with them, both the Station folks and some of our CATS support folks, to help them see where some big integrated system tests might be useful. We found some problems that delayed the flight by a little bit. Some of those big system tests found real problems that would have been a mission failure if we didn’t catch them on the ground.

Wright: I guess that’s the success of melding the new with the old.

Horkachuck: Yes, sometimes the old guys know what they’re talking about.

Wright: Combining all these efforts, there was a lot on the line when they launched that flight.

Horkachuck: Yes. I think there were a lot of hidden agendas as well. There was a lot of questioning, “Does this whole COTS thing work?” There was a lot of politics in the political world thinking is this the smart thing to do, should we really be messing with COTS and this whole commercialization, or should we be doing it the more traditional way.

Wright: You’re no longer on the back burner now. You were in the forefront, people were watching.

Horkachuck: Yes, that whole commercial crew thing was starting to take off. Quite frankly, if that flight was not successful I’m not sure commercial crew would have taken off. There was a lot of scrutiny as to, “Is this new approach going to work?” and “Could they build something that was reasonable and successful?” I think a lot of people breathed a sigh of relief after it achieved its objectives, because I think there would have been a lot more scrutiny and rethought about “Are we doing the right thing with this approach?” if it wasn’t successful.

We had enough history, based on rocket developments over the years, to know that in the first three flights of a new vehicle, there’s normally a failure. A lot of times, it’s a catastrophic failure of a new rocket, whether it was built as a commercial rocket or the government had sunk its full resources into that development. The curves typically show in the first three flights there’s usually a problem, then it levels off for a little while.

Then, at about six or seven flights, there’s another spike up in failures as they go from the development organization to the production phase, and lose track and focus as they transition from the guys that designed the vehicle to the folks that are more the operations and builders. Seems like there seems to be that spike later on as they go into production.

We knew there was a lot of risk. The FAA and range safety, as part of their normal process for licensing and certifying that the flight is okay to launch, do calculations based on history and what the understanding of the vehicles are, what is the likelihood and probability of the vehicle being successful. So yes, there was a lot of risk on a lot of those flights.

Wright: Did you find yourself in a position of explaining, and possibly justifying, more of the processes of reaching that flight as you approached? Were you in meetings with the higher ups?

Horkachuck: Yes, we ended up briefing both here at the Center and at Headquarters. We managed to keep it fairly streamlined and not have to brief a lot of different organizations and panels. We managed to get them all pulled together. Mark helped quite a bit influencing, based on some of the reviews that had typically gone on for Shuttles, and combining them together with briefings to the Associate Administrator. Making sure that all the parties were in the same room at the same time, we didn’t have to do it multiple times. That was really helpful.

Wright: That’s helpful, yes. Did you ever feel like you didn’t have the support of upper management to continue on the journey that had begun?

Horkachuck: No, I didn’t necessarily have that feeling during the majority of the development phase. I had one manager at one point that went on vacation and gave me some of the wisest words I have gotten in my career. I asked him, “So what’s my authority while you’re gone?” His response was, “Proceed until apprehended.” And I’ve had people tell me over the years, “Ask for forgiveness and not permission.” That is the way you really get things done.

Wright: At least you had a little more flexibility in this environment than you would have before.

Horkachuck: I was happy not to get too much help. They gave us the authority to try and make the right decisions. Some of the upper management are really really smart and have a lot of experience. They were able to help us, point out things that we should look for based on their experience. I never found our direct line management to be a burden, they were really helpful. It was just really good, it’s been a pleasure working with them.

Wright: It’s always good to know that you’ve got someone taking care of that side. The days leading up to that last launch, were you constantly with the SpaceX folks as part of their whole process?

Horkachuck: I was there with their team at the launch site during the final processing. They do a lot of things that are a little bit unique. They test fire the integrated first stage out in [McGregor] Texas and make sure that the whole integrated stage works. A lot of companies don’t test fire their rocket before they go fly it. Then they do a wet dress rehearsal on the pad, which a lot of the companies do. Fully fuel the rocket, go through simulated countdown, make sure all the systems are working.

But then they go through one that’s a static fire on the pad where they actually light the engines for a couple seconds, still hold it down and make sure things are working. Part of that is because they designed their engines to be a little more robust than typical. It paid off in some cases. We went through a countdown and they actually lit the engines, and then had a check valve fail and aborted.

If we had done that with the Shuttle, it would have been months and months before we flew again. They turned it around in a matter of days. They found where the check valve was that had failed, had pictures of the part that had come loose in it, knew exactly why it came loose. We jointly discussed it and reported to our upper management, changed it out and within a couple days they launched. They turned around and had the vehicle ready to go again. If you’d lit the engines on a Shuttle, you’re not going again in a few days.

Wright: Talk to us about the launch, and the pros and the cons that came from the results of all that work. What was the best part for you, the relief of it working out as well as it did? If you had to find something that didn’t work out well from that launch, or maybe the best lesson learned that came out from the mission itself.

Horkachuck: I was actually in shock for days, if not weeks after, that everything worked as well as it did. We had done a lot of work, and a lot of due diligence and had a lot of review and tried to find where the problems would be, but as complicated and as complex as the whole thing was—and then to combine the two missions together, add the complication of trying to do both missions all in one—it really surprised me that it all worked as well as it did.

The only thing that got a little dicey is on final approach when some of the rendezvous and prox sensors weren’t reading the same values. There was some built-in aborts that we were worried were going to trigger and have the vehicle back away. So it took longer on that first approach to come into Station, but the SpaceX ops team and the [NASA] MOD [Mission Operations Directorate] ops team worked really well together, and had done a lot of training and simulations together, and were pretty comfortable with the data they were seeing that they weren’t too far off from each other.

Quite frankly, the crew could see the vehicle and knew where it was. It may have been more of a risk to really abort and try again than to just proceed. Then Don [Donald R.] Pettit did a great job capturing the vehicle, it’s amazing how fast he grabbed it.

Wright: It’s nice to have competent help on the other end.

Horkachuck: Don was amazing, yes. He had done some stuff for us when I was in the Payloads Office. We had some freezers up there that had, after a year, died on orbit. Thought, “All right, why don’t we give this guy a shot at trying to fix them?” They were never designed to be repaired in space, so I had our guys build basically a Chilton’s [auto repair] manual. I said, “Go take another unit that we have on the ground, take pictures of all the screws you have to take out, step by step, and lay it out in pictures. Little arrows pointing to what to do next and everything. Send that up.” It violates all the typical MOD procedure rules, but quite frankly I don’t care. Either he’s going to fix this thing and it’s going to work, or it’s trash.

He was hilarious. It turned out it was his birthday, and he was thanking us for giving him something fun to do on his birthday. Completely disassembled this thing and resoldered the leads to the thermoelectric chips, and actually got it up and running for a while. That proves that you can do some things on orbit that we didn’t think you could do. Didn’t have to be all these ORUs [Orbital Replacement Units] that we typically thought for spaceflight. Very important if we do a Mars mission, where you can’t call home for spare parts.

He’s an incredible guy. Some of the Saturday morning science that he did with the students on the ground—it was nice to have him on the other end, when our vehicle was coming up.

Wright: I bet it was. Then of course it returned home safely, and hopefully as you wanted as well.

Horkachuck: Right, reentry. It didn’t get very warm. The heat shield worked like a champ, even on the first flight. I remember looking at some of the thermocouples they had underneath the heat shield, and I was thinking, “Are we getting still data? The temperature hasn’t changed.” It just didn’t get through the heat shield.

Wright: How do you feel like your relationship with SpaceX changed from the day you walked in and met them until the day you sat down at a debriefing after this last mission? How would you explain that relationship?

Horkachuck: It was an evolving, trust-building partnership I think. As new people came in—I’ve been around more than 90 percent of their company. I had a lot more history than most of the employees at SpaceX on what was going on and how things worked. When they first came in, they had the “I know it all” kind of attitude. It took a while for them to learn that maybe they could use some help in some areas.

I think what they were constantly fearful of was getting slowed down and bogged down by the NASA process. It was a bureaucracy on making decisions, so I tried not to have anything that would really slow them down too much. More of that “proceed until apprehended” philosophy.

Wright: What do you think you learned from them in this whole process?

Horkachuck: It’s unique to me—when we talked to them about a given part or component in the design, they’d say, “Well, we could go buy this from this vendor, but it’s like $50,000. It’s way too expensive, it’s ridiculous. We could build this for $2,000 in our shop.” Almost every decision that they made had cost built into it.

It was unique because I almost never heard NASA engineers talking about cost of a part when they were making design trades and decisions. They were worried about is it going to work, is it going to work reliably and safely and meet all the requirements. Cost generally, although a factor, was never really in the forefront as much as mission success. Here they were making more trades on, “Well, you could do it that way, but this way is a whole lot cheaper and probably just as good.” That was a different mindset, and I think something that maybe the rest of the Agency needs to look at a little bit more.

I worked with some really great people over in the Space Station Program on being able to use commercial electronics on their vehicle. NASA is used to using these S-level rated parts, components that go into all the avionics boxes. They’re either RAD [Radiation Absorbed Dose] hard and/or screened through this process to make sure that they’re highly reliable. SpaceX wanted to use more of the commercial-type parts.

Turns out that both the aircraft industry and the automotive industry have done a lot of work with a lot of these commercial component part vendors to make sure that the parts coming off the line are really reliable. Because you can’t have cars breaking down all the time, or they’d lose their shirt. Or airplanes. So, a lot of the reliability things that we’ve been concerned about through some burn-in process and testing, they catch. A lot of the vendors seem to be building much more reliable electronic components that they could use in their systems.

Then it was a question of how do we deal with the radiation effects. We got SpaceX plugged in with [NASA] JPL [Jet Propulsion Laboratory, Pasadena, California] and some of our JSC ISS radiation experts that have been doing testing. I tried to get them to do testing early on, based on some of the work that we had done with some of our payloads where we’d sent them to one of the universities, I think Indiana [University, Bloomington], and done some level of radiation testing.

It may not have been exactly the high rel [reliability] radiation testing that we’d done in the past, but it gave you a good screening, gave the payload guys a good feeling as to whether or not their parts were going to work in space. I tried to get them to think about doing some of that testing early on, and they didn’t want anything to do with it. Station finally put their foot down and said, “Look, you’re not going to make it to Station unless you prove that you’re not going to have all these radiation effects.” Then they eventually did a lot of the radiation testing that we had talked about.

It basically showed that through smart part selection and some limited amount of this type of radiation testing at the more integrated component level, you could get pretty good reliability in a radiation environment. The combination of that and the redundancy that they had built into their system let you tolerate some radiation upsets. And reset, and be ready to work again.

What was really important for me is back before I even came to Space Station I was working out at Ames for the centrifuge project. We’d done a lot of cost modeling trying to figure out how much the project was going to cost, so I’d done a lot of work with some price modeling people. We did a lot of iterations on the design and concept and folding into the price models.

What I found was no matter how I sliced it and diced it into the price models, the avionics subsystem almost always came out as the longest lead time and highest cost. That, and the software which wrapped in with it, drove the project. You could have thousands and thousands of pounds of structure and just a small amount of avionics, and it still was the longest lead time and biggest cost subsystem.

Finding ways for the Agency to be able to build avionics faster and cheaper, I think, is going to be a huge savings for projects in the future. Just my knowledge of overall project development says that’s always going to bite you. If you can cut down the cost in that phase of the project and the scheduling in that part of the project, you’re going to have some big paybacks.

Wright: I know we have more we want to ask, but we’re about through for today on our time, so you can get back to your life as you knew it before we entered it. So we thank you, and we’ll see if we can catch up with you again.

Horkachuck: All right.

Wright: Thanks.

[End of interview]

Return to JSC Oral History Website

Go to NASA home Go to JSC home

Curator: JSC Web Team | Responsible NASA Official: Lynnette Madison | Updated 7/16/2010
Privacy Policy and Important Notices

Information JSC History Portal