NASA Johnson Space Center Oral History Project
Commercial Crew & Cargo Program Office
Edited Oral History Transcript

Michael J. Horkachuck
Interviewed by Rebecca Hackler
Hawthorne, California – 16 January 2013

Hackler: Today is January 16, 2013. This oral history interview is being conducted with Mike Horkachuck at the Headquarters of the Space Exploration Technologies Corp., or SpaceX, in Hawthorne, California, for the Commercial Crew & Cargo Program Office History Project. This interview is a continuation of the oral history conducted on November 6, 2012. The interviewer is Rebecca Hackler, assisted by Rebecca Wright.
We talked about the beginning of your working relationship with SpaceX in the first interview, and we wanted to ask you a few follow-up questions. The COTS [Commercial Orbital Transportation Services] Space Act Agreement with RpK [Rocketplane Kistler] was terminated in October 2007. From your office, was there ever any consideration to giving the remaining funds to one of the runners up from the first competition, or even giving those to SpaceX to help them further their vehicles faster?

Horkachuck: I think we always wanted to maintain competition, so that was kind of a driving principle in the whole project. There wasn’t that much entertaining of giving it all to one participant. I’m trying to remember what the rational was for opening the competition up again. Probably because the scope had changed, and the dollars and the schedule were tighter. It seemed like they were going to need to get some updated proposals anyway, so in general it was just easier to open it up to a full and open [competition].

Hackler: From what I understand, SpaceX did compete in that second round, but they weren’t awarded any additional funds. Were you aware of that at all, or have any involvement?

Horkachuck: I was busy managing the current contract and all the other unfunded partners that we had. I was busy doing the work while they were off in the bunker on that second round, holding down the fort.

Hackler: Did you not have any involvement in the CCDev [Commercial Crew Development] competition, either?

Horkachuck: No.

Hackler: We heard from legal that as both the COTS office and the commercial partners learned more about the Space Act Agreements and how to use them, the proposal process evolved a little bit. I didn’t know if you had heard any of that from your perspective?

Horkachuck: No, not really.

Hackler: Then later on, the budget for COTS was augmented with an additional [Congressional] appropriation. Did you have any role in allocating those funds, or helping define the milestones that were updated? I think it was $128 million extra for each partner.

Horkachuck: Yes, Congress authorized I think originally close to $300 million. Then there were some taxes taken off the top of it, and they pulled back some money. It eventually split out to $128 per partner. Then I negotiated the milestones that we were going to add to the Space Act Agreement with SpaceX. We did a lot of back and forth, and tried to identify the critical areas in the program that it would make sense to add additional work to. I think, quite frankly, that was a really good idea by Congress. It was probably one of the oversight committees that had been looking at NASA and the overall program that recommended it to Congress. I don’t know if it was the NAC [NASA Advisory Council] or one of the other ones.
For a development project like this, adding a large budget influx is a great idea because you’ve started the project, you’re starting to see how the contractor and the project is forming up, where the strengths and weaknesses are in their development programs, and you can start plugging holes in where there are big weaknesses. In the case of SpaceX, we saw that they hadn’t been doing some integrated testing. They hadn’t planned to do some things that we typically would do, like integrated module EMI [Electromagnetic Interference] testing and thermal vac [vacuum] testing.
We added those as new milestones under the augmentation money. Though the thermal vac test came through pretty clean, we gained a lot of data on how to operate the vehicle. They correlated their models, which helped a lot with the thermal analysis. We didn’t find any big surprises there. In the EMI testing, it turned out that there were a couple of sensors that were very susceptible to EMI, and it would’ve been a mission failure if we had flown in that condition and not known about it. The EMI test caught that, and we ended up standing down a couple of months to get those repaired. Delayed the initial flight, but was well worth finding on the ground before we got into space.

Hackler: From your perspective, how did communicating with two NASA points of contact work? SpaceX had to work with the ISS [International Space Station] Program Office, and then through you in the COTS office. Were there ever any issues in communication, or different styles of working between those two areas?

Horkachuck: Yes, I guess there probably were. There were different roles and responsibilities. The Station was primarily making sure that when they got close to the Space Station they were safe, and they met the Station interface requirements. We were looking at it from the bigger picture of overall mission success, getting it off the ground. Was the [launch] pad working, was the rocket going to work, was the vehicle going to be able to fly up close to Station?
Then the whole reentry and return was all part of our requirements that was outside of what the Station requirements were. We had a much different and bigger picture of things. There were certainly times when I had to get in and influence how hard Station was pulling on a particular requirement, and try to add a little bit of sanity to meeting the letter of the law versus the overall intent. That was partially because I had a lot of history on where some of the requirements had come from in the past.

Hackler: In the first interview, you talked a lot about SpaceX’s focus on cost, which is not traditionally a NASA focus. And instead of designing to the optimum first, you have something a little more robust, and then you can optimize later as you understand how the vehicle works. There are obviously advantages to that approach, but did it ever result in any drawbacks that may have impacted the cost or risk later on?

Horkachuck: I think the biggest drawback was late in the development we found out that they had a weight problem, and it resulted in lack of total performance of cargo to Space Station. They’re mitigating that by upgrading the rocket. They’ve had plans to do an upgrade to the rocket anyway, and once they’ve come up with the Falcon 9 1.1, supposedly all our major mass constraints are going to go away.
There’s some disadvantages. If you’re continually evolving the design, it’s a real good way to go. If you’ve got one shot at it, then you tend more toward the traditional NASA method of doing a lot more analysis up front, and making sure you’ve optimized early.

Hackler: Are you also involved with the development of the 1.1?

Horkachuck: That’s what I’m out here watching now, the development of the 1.1. And there’s some big upgrades to the capsule itself that are being done to accommodate more payloads. I’m primarily overseeing and providing insight back to the CRS [Commercial Resupply Services] office for those big changes.

Hackler: SpaceX is a very well-publicized company. They get a lot of media attention, and you also did some interviews with the media. Can you talk about that experience, because I don’t imagine you were trained for that as a NASA engineer. Especially when the second demonstration flight was flying [C2+], there was a lot more focus in the public eye.

Horkachuck: The first one or two on camera got me a little bit nervous, but our PAO [Public Affairs Office] folks coached me through what to typically do.

Hackler: Was that the NASA PAO or from SpaceX?

Horkachuck: NASA PAO. All the interviews were set up through NASA PAO. I think they did a really good job giving me an idea of what I should talk about, what I shouldn’t talk about, plus I had a lot of coaching from Alan [J. Lindenmoyer, Commercial Crew and Cargo Program Manager] on where political hot points might be. NASA PAO also did a good job of not putting me in front of an antagonistic press audience that was going to ask a bunch of not-so-fun questions. They did a good job, I think. Overall I thought it was great experience.

Hackler: On the first CRS flight, after they had demonstrated their capabilities under COTS, there was the engine failure. Was that a concern for you, or do you see that more as just a part of the development process for rockets?

Horkachuck: Anytime you lose an engine, it’s a concern. It was sort of two-fold. During the actual launch it was hard to really tell that it happened, because it happened fairly quickly and the rocket still ended up taking the capsule to orbit. It proved Elon’s [Musk, SpaceX founder and CEO (Chief Executive Officer)] initial claims that he had designed this rocket with an engine-out capability. They literally proved that in flight.
That was kind of a pro to it, but anytime you have an engine failure it means you’re going to have to go back and find out what went wrong, because you don’t want multiple engine failures in the future. There’s been an ongoing investigation that I’ve been privy to looking into what the cause of the failure was, and what we can do on the next mission to make sure that we don’t have the same situation. In the future, the new rocket has new engines that don’t seem to have the same failure mode.

Hackler: You described SpaceX as a Skunk Works-type organization when you first started working with them, but they have had to take on some more bureaucratic processes as they’ve worked with NASA. How do you see their trajectory, as far as the balance between maintaining innovation and avoiding excessive overhead, versus having to have proper documentation?

Horkachuck: I think it’s not just because of NASA. I think they’ve been doing it as part of their evolving to a bigger production company. They’ve been putting some of the processes in place that they need to. Their new production manager put into place a process for limiting changes, because he has to cut them into the production runs. If you’re constantly changing the design, he can’t build the vehicles. He’s forcing some of that just as a natural evolution, which has been one of the big heartaches that CRS has had.
We were constantly changing the design to some degree. They needed to be changed, but it meant that they had to do re-work on their analysis, and re-work on some of the verification activities that they had done to make sure that the vehicle was good to come close to Station. There was a lot more work on the NASA side because of all the changes that were going on. I think they’re being a little more conscious about how many changes to push through, and recognizing which ones are critical and which ones aren’t so critical to follow up on.
They’re putting into place a lot of general processes that you would expect a bigger company to have in place. I think they’re becoming more like a traditional aerospace company, because as you get bigger you can’t just pop your head over the partition to talk to your neighbor who’s doing the entire work on structures. You have to have some controls and ways of doing business that communicate information between a bigger organization.
I think they’re starting to become more of a big organization, and I think some of the change in innovation will slow down a little bit as a result of that, but they still have a lot of capabilities by having engineering and manufacturing co-located. They can do a lot of prototyping, and testing here without a lot of paperwork to get approvals. They can still do some innovations, but it’s definitely changing from where it was a few years back.

Hackler: We heard from Pete [Peter] Capozzoli this morning that NASA is trusting SpaceX on their delivery missions with more important cargo as it goes on. Do you help with that at all, as far as determining the manifest for what goes in the capsule, or making sure they have the capabilities for live specimens?

Horkachuck: Determining the manifest I don’t do. That’s all done back in the Space Station Program Office. It’s driven by science priorities, and other Station resupply priorities. I am helping with some of the additional accommodations to be able to accommodate live animals, because it’s a change to the module design, new development work. They’re using some of my experience being able to develop projects to help with that activity, and adding it into the vehicle design.

Wright: That’s got to be fun.

Horkachuck: Yes. It’s nice working with the [NASA] Ames [Research Center, Moffett Field, California] folks again, too. I originally started with NASA up at Ames Research Center, working with some of their live animal programs, and I’m getting connected back with a bunch of folks that I used to work with 15, 20 years ago.

Wright: They have lots of expertise in that field

Horkachuck: Yes.

Hackler: That was something that you mentioned at the beginning of your first interview, looking at ways to get science samples back [from ISS]. Is that becoming more of a reality, that you can actually conduct those experiments, thanks to the COTS program?

Horkachuck: I think so. There was a gigantic program originally out of Ames to build a two-and-a-half meter centrifuge that I’d worked on, and a life sciences glove box, and animal and plant habitats. A lot of that got shut down because the [Space] Shuttle was being retired, and there was no way to be able to bring up all the supplies and the live specimens and bring them back down again. The Russian vehicles wouldn’t accommodate it, and there was no other return vehicles. Logistically, you just couldn’t support that big program.
Now that we have SpaceX, there’s a capability to do at least a limited amount of that work again, be able to bring them up, and sometime in the future maybe to bring them back down again. It opens up that whole field of science again that had been kind of in hibernation for quite a while.

Hackler: What other types of roles do you see opening up for commercial flight as a result of this effort, since that was one of the goals of COTS to open more commercial markets?

Horkachuck: I don’t know. Personally, I don’t see a lot of commercial science going on. I know SpaceX is talking about DragonLab, but my experience has been that the science community doesn’t have the budget in a lot of cases to be able to buy a launch. They’re used to the free rides on the Space Shuttle. They had enough budget to develop their science equipment, not to buy a ride.
Whether it was $20,000 a kilogram, or $10,000 a kilogram, or even something less than that—you’re talking about a 50-, 70-, 100-pound locker, $10,000 a pound, that’s a big chunk of change. They just never had in their budget, and I don’t see them having budgets to buy launch capability in a lot of cases. They’re used to the free rides. I don’t really see a market for DragonLab, unless there’s some big tenant customer that buys the whole flight, and then has extra capability that he gives away to some of these smaller payloads.
I think in the long run, as they start developing a crewed vehicle, I could see that there’ll be spin offs of the crewed vehicle where they’re taking tourists up to space because there’s some deeper pockets there to buy a ride. If some U.S. millionaires or billionaires can spend $20 to $30 million with the Russians to buy a ride, then there’s probably the potential market out there for taking other people to space in U.S.-built vehicles. I’d like to, in the long run, see that evolving to more of an airline kind of transportation system, and maybe creating a whole market for a whole industry in the United States.

Hackler: We’ve talked about and heard from many people that the COTS program was very successful in achieving its goal of helping commercial companies develop these capabilities. Do you see any role for a COTS-type program in the future? What other areas do you feel like it can be applied?

Horkachuck: That’s a great question. We’ve been trying to figure out what to do in our office, and apply the model to other capabilities. We talked a little bit about lunar comm, having a communications system to provide the service of transferring comm. That hasn’t really sunk in much. The other one that we’ve been talking with some other parts of the Agency about is maybe a Mars transportation system, to take science samples and do landers, pre-position some equipment on Mars. I think that would be a great project, but I’m not sure there’s budget in the Agency right now to support something like that.

Hackler: What kind of lessons learned would you share with anyone trying to apply the COTS model in the future?

Horkachuck: I think you have to do a lot of homework with the company, and make sure that you can have a good working relationship, because there are certainly some different controls that you have in the COTS model than you had in the traditional contract. Having a good working relationship, where it’s really a partnership, makes a big difference. Certainly lots of advantages for the government in that you’re basically in a fixed-price contract. You’re paying them for work that gets done, not just continuing to spend on the cost-plus contract. There’s built-in controls that limit that there won’t be a cost overrun, basically.
You have to be flexible on schedule. A lot of times these guys are going to be very optimistic on proposals of schedules, and as long as they’re making good technical progress, I think you need to be flexible when they start, later in the project, not meeting milestones on the schedule that they were talking about. Stay focused on the technical requirements, and that they’re meeting the technical intent.
Be flexible on ways to do business. Sometimes ways that certain companies do business can be a lot more cost effective, and be flexible to understand their rationale, and how it may still meet the intent of what you need makes a big difference.

Hackler: From what we understand, you did and still do have an excellent working relationship with the people here at SpaceX, and they’ve certainly expressed how invaluable NASA’s help was as they were building up their company and vehicles. I’m curious about what was unique about working with SpaceX that might have been different from working with one of the other commercial partners? Was there something about SpaceX that attracted you to choose to work with them in the beginning, or was that even your choice?

Horkachuck: Yes, I guess it was a good part my choice. I like their innovative style, and it just seemed like they were a more natural fit with my management style. It was more of a working partnership than a more traditional organization and management style than NASA was used to. I was a better fit to be able to work cooperatively with them, than what the other partner was at the time.

Hackler: What was unique about working with them that you may not have had the same experience working with another partner?

Horkachuck: They were learning a lot, and receptive to learning from some of the NASA technical experts. It gave me the ability to pull in folks that I had known in the Agency, and let them pass on some of their knowledge to SpaceX of things that work and things that didn’t work in the past, and even go as far as to suggest ways of doing things, which typically we wouldn’t do under a contract. I think it was good for me, and it was good for a lot of the NASA experts to be able to pass on some of their knowledge and wisdom over the years, and feel like it was getting received and used to good purpose.

Hackler: You did also mention in your first interview that some of the members of the CATs [COTS] Advisory Teams didn’t fit as well with SpaceX’s way of business. Did you ever observe any former NASA employees getting hired on here that maybe didn’t work out because they had a similar conflict in their organizational culture?

Horkachuck: It seems like there may have been one or two, but I don’t remember specifically. We didn’t get into a lot of the details of their hiring and firing practices. What went on internal to SpaceX was kind of their business, as long as it wasn’t affecting some of the real key personnel. I think in some cases they probably hired people because they had some of those processes and knowledge of ways of doing business. Certainly some of their newer hires have brought some of the history of here, and given them that experience base to learn from and use the best parts of.

Hackler: Thank you. I’d like to ask if Rebecca Wright has any questions.

Wright: I’ve got a couple. Going back to the experts—did you ever ask anybody to weigh in and they just said, “No, I don’t really want to have anything to do with the COTS model,” or, “I don’t agree with this, so I don’t want to participate?”

Horkachuck: Not that actually ended up helping the program. There were certainly people throughout the Agency that didn’t necessarily like the COTS model. When I got down to the technical experts, for the most part all of them were happy to help, especially after I explain to them how we were going to interact with SpaceX. A lot of them actually would have preferred more interaction. We ended up limiting how much activity there really was. They may have been a little frustrated from that perspective, but when we got down to real technical issues and problems, they were very helpful and very receptive.
I think in general, virtually everybody I know in NASA wants to do a good job and make sure that NASA projects are successful. There were some comments that had gotten to the press over the years, maybe taken out of context. Some of the statements that Elon made that rubbed some people the wrong way. I tried to explain what I thought the real intent of some of those comments were, and they tend to get over it and realize what was going on. They’ve generally been really helpful, and enjoyed working with this company and passing on their knowledge.

Wright: You definitely were able to build up a premiere working relationship, building a true partnership. Is there a time when you were first starting out in your role that there might have been a tipping point, when you knew that they were listening to you as a part of the project team? Tim Buzza mentioned that it wasn’t “us and them” anymore, it was “we.”

Horkachuck: It varied, I think, by relationship to some degree. Some of the engineers that I worked with early on—like the previous lead for the Dragon, I had a great working relationship right from the start because we interacted a lot. I confided in him my concerns, and he explained where he was coming from. It was an open relationship right from the start, where we were trying to make sure we were doing the right thing. Other organizations, it took time to get to know each other and work together.
Some of the earlier launches, Hans [Koenigsmann] and Tim tried to keep a lot of things to their side, they didn’t want NASA’s help. As they started getting more into it, and realizing they needed to include us, it became easier. Especially when they had a main point of contact for them to interact and bounce things off, and let me go work it with the rest of the NASA organization. They saw me as an advantage to them, as opposed to some roadblock in the way of them getting to a decision that they needed to make. I think it just varied with each organization, when it made more sense to open up and feel like we were a part of the team.

Wright: You always felt like you could ask questions, that was kind of your role. They may not answer, but you got to ask.

Horkachuck: Right, right. I wasn’t too shy about asking questions. Sometimes I’d just ask questions in a framework to make them think. Not necessarily looking for an answer immediately, but let them go off for a week or two and think about what I just talked about and maybe they needed to change something. I didn’t necessarily force it down on them, but it’s a seed.

Wright: They put themselves in a position that gave them an opportunity to move forward in what they wanted to accomplish as a company, but there was also some amount of pressure to succeed because this was a chance and an opportunity that could lead them to more things. Did you feel, in your position representing the COTS office in NASA’s new way of doing business, some type of pressure to do everything you could to see if this new way of formulating commercial work with a federal agency would work?

Horkachuck: I don’t know that I did that because it was a COTS model. I did that just because it was my project, and I was going to make sure my project worked. I think I brought that to every project I’ve worked on. I’m going to do everything I can to make sure that I try to see where the potential problems are well in advance if possible, and bring resources to bear there. Some other problem will probably crop up that you didn’t foresee, but at least you were able to take out a lot of the big ones early on. I was going to do that regardless, whether it was some new model for the Agency or not. It was a project to go do and make successful.

Wright: I just have one more. You’ve been with this project from the beginning, and of course you’ve been with SpaceX when they had a lot fewer employees than they have now. In your expert opinion, explain the impact that this opportunity with NASA has afforded SpaceX.

Horkachuck: I think we brought them up from being a little 100-man company, if that, to what they are today. The vast majority of everything that they’re doing today has been funded primarily by NASA and the CRS contract, and then some of the follow-on NASA contracts. They’re getting some amount of money from some of these commercial contracts, but not big amounts of money that have changed the way they’re doing business.
I think the CRS contract has been a big anchor tenant for them, and it’s kept them literally afloat for a couple years, between the COTS program and the CRS Program. Early on, COTS was what was keeping the lights on in the company. They’ve just evolved into less of a hobby shop and more of a real aerospace company that’s building production rockets.
I’ve been pretty impressed with some of the upgrades that they’ve done to the facilities. Some of that was funded with some of that augmentation money. We sprinkled around some of that money to upgrade the production capabilities here, test facilities and launch site facilities, because we knew we wanted to build a basic capability from the COTS program to be able to support the CRS ongoing contract.

Wright: It will be interesting to see the next few years, how it goes.

Horkachuck: It will.

Wright: Well thanks.

Hackler: You talked about the upgrade to the facilities. Did you exert a lot of influence over that decision making process when they decided to move and have everything under one big roof?

Horkachuck: Oh, that was a long time ago. I think they did that on their own, the move from El Segundo [California] to Hawthorne. Once they got here though, I noticed problems that they had in the production of the first few rockets and capsules, and kind of forced some of that money to be spent on fixing problems in some of their production areas that we had noticed on the first couple builds. We helped with an additional friction stir welder, and some other production capabilities here that are making a big difference to them being able to build more rockets.

Hackler: Before we close out today’s session, do you have any final thoughts or reflections on your experience of working with SpaceX the last six years that you’d like to share?

Horkachuck: It’s just nice working with an energetic group of people that want to do the right thing. I’m noticing a slight trend towards wanting to have more of the requirements defined upfront, and if you make a change, then we’re going to have to write a contract change. They’re getting into the old, typical contractor mode, and I’m a little worried that that’s going to sour some of the relationships in the future.
It’s fair and it’s reasonable if they’re losing money, but it also slows things down and makes it less efficient in some ways. I hope they don’t go too far over into the typical government-contractor relationship mode where it stifles a lot of the innovation, and ability to go forward quickly and do the right thing for the development program. In the big picture, they obviously want to make things work. Their livelihood as a company depends on them building rockets that work and are successful, and I wish them all the best.

Hackler: Thank you very much for taking the time to sit down with us a second time today. It’s very much appreciated.

[End of interview]

Return to JSC Oral History Website