NASA Johnson Space Center
Oral History Project
Commercial Crew & Cargo Program Office
Edited Oral History Transcript
Michael
J. Horkachuck
Interviewed by Rebecca Wright
Houston, TX – 6 November 2012
Wright: Today is November 6, 2012. This oral history interview is
being conducted with Mike Horkachuck at the NASA Johnson Space Center
[JSC] in Houston, Texas for the Commercial Crew & Cargo Program
Office History Project. Interviewer is Rebecca Wright, assisted by
Rebecca Hackler. Thank you again for taking time out of your busy
schedule, we appreciate it.
Horkachuck:
You’re welcome.
Wright:
You’ve been involved with the COTS [Commercial Orbital Transportation
Services] program since September of 2006. Share with us how you first
got involved with NASA’s commercial efforts.
Horkachuck:
I guess going all the way back, I was working in the [International
Space] Station [ISS] Payloads Office. At one point we were looking
for ways to bring samples back from the Space Station more frequently
than was planned with the Space Shuttle. I had looked at a bunch of
small reentry vehicles over the years, and I think I even wrote a
short white paper for [William H.] Gerstenmaier [ISS Program Manager]
looking at some options that the ESA [European Space Agency] folks
had been working on in combination with the Russians. At the time
we thought we just needed a small reentry capsule to bring some critical
science samples down on a more frequent basis than we were getting.
That went into the ether for a little while, and then I got called
to help these guys that were working on this COTS proposal to help
write the requirements for the initial RFP [Request For Proposal].
I rolled a lot of the science support requirements into the very top
level requirements that they were developing for COTS. At that time
there was a requirements document, top-level goals and objectives
of the COTS vehicles that were going to be built. It didn’t
get down to a lot of real specifics. It was how much total mass do
you need, what was some of the top-level functionality that we wanted
out of the vehicle, without dictating what the design was. I worked
with them over in the procurement bunker for a little while developing
the requirements.
At about that same time the Constellation Program was ramping up,
so I went off to a job in test and verification over in the Constellation
Program [Ares rocket and Orion Crew Exploration Vehicle]. I was one
of the managers for test and verification for all the Level II requirements.
Then after being there about a year I got a phone call saying, “Well,
we down selected to some COTS providers. Would you like to come over
and manage one of the COTS companies’ development programs?”
That seemed to be a pretty good fit for me because I always liked
hardware development, I tended to do that and enjoy that more than
the ongoing operations and sustain functions. And I’d had a
lot of projects in the past that were more partnerships than strictly
NASA contractor relationships. I worked a lot with the European Space
Agency developing a bunch of the freezers that are on Space Station
right now, and we worked as a partnership.
Technically they were building them for me, but since it was another
space agency it wasn’t like NASA was contracting with [The]
Boeing [Company] or some other company to build it strictly for us
and we could really dictate exactly how we wanted it built. We had
to work in cooperation with their constraints and budgets, and some
of their other political constraints, and jointly come up with the
best solution.
It seemed like the COTS program was going to be very similar in that
respect, where it was a partnership instead of a formal contract.
I had a lot of that experience, and I had also had a lot of experience
when I was in the Payloads Office. Not only did I manage the whole
fleet of the freezers and some of that laboratory support equipment
that was used by all the science community, but I was the manager
of all the payload engineering and integration. By that I mean I owned
all the requirements for how the payloads would interface with the
Space Station—both the basic requirements and the verification
of those requirements—as well as the integrated stage analysis
that we’d do to make sure that when we plugged in that complement
of payloads to the Space Station, the integrated spacecraft was within
its limits. We’d do a whole module level power balance, data,
thermal, acoustics. A bunch of different analyses, so I had a whole
team of different discipline engineers working for me.
I did sort of the same thing in Constellation. Although we were trying
to use other NASA centers a lot more in the Constellation Program,
so I’d have people reporting to me from all different NASA centers
with different technical expertise. That fit really well into the
concept they were trying to roll out for COTS and having these COTS
Advisory Team members, CATs. Effectively, it was a matrixed organization
of people that we could go reach into, pulling their technical expertise
when we needed help on a particular area or problem. Then they’d
go back to their normal day job when we didn’t need them anymore.
They weren’t around full time, and that had some pros and cons
to it, but in a lot of respects it was similar to a lot of the things
I’d done in the past.
Wright:
Since you’re talking about the similarities, what were the non-similarities?
What was so different now about this new way of doing business with
industry partners?
Horkachuck:
Well, it wasn’t a contract. It was a Space Act [Agreement],
which is NASA’s Other Transaction Authority. We’ve got
some unique authority under the original authorization to create NASA
[National Aeronautics and Space Act of 1958] that lets us do these
Space Acts. That has a big benefit in that you can get things done
a lot faster than you can in a normal contract.
Quite frankly, I think our normal FAR [Federal Acquisition Regulation]-based
contracting system is broken. It’s so out of date and so slow
and antiquated that it’s, I think, a big part of the reason
that not just NASA, but all federal agencies have such massive cost
overruns and complete screwups on contracts. It’s just so antiquated,
it doesn’t work anymore. There have been cases where we’ve
tried to make a change in requirements on a contract, and it’s
taken the pricing and proposal and contracts folks a year to get the
proposal back on making a change. By then it’s too late. You’ve
wasted a year of your development program dicking around with pricing
and negotiations of the cost.
It’s just so slow that it becomes a major burden. Your cost
overruns go through the roof because you’re still designing
it to your old baseline, which you know is wrong, but you’re
still wasting money doing that because you haven’t been able
to authorize what you know is right, because they haven’t gotten
around to doing the pricing for you.
Wright:
With today’s technology you have new materials as well.
Horkachuck:
Yes. If anything comes up new, you can’t implement it because
you don’t have it on contract.
Wright:
So you found not using the FAR system to be a more liberating way
of doing business?
Horkachuck:
Oh yes. Since it was a partnership, they went in with their eyes open
knowing that we’re going to do our best to determine what the
basic top level requirements were. We weren’t going to go down
really deep into the requirements, but they knew what the end goal
was, so if there were changes in things that had to be done, if we
could convince them that technically it was the right thing to do,
then they just went and did it. There was some technical convincing
that would have to happen sometimes. You’d have to have a discussion
on the merits and why a certain change needed to be implemented, but
once you got through the technical convincing, they just went and
did it. It wasn’t a one-year monotonous, gory fight over “It’s
going to cost this much,” “No, it’s not going to
cost that much,” “It’s going to cost this much.”
That was part of what their risk was going into the partnership. We
came in with a fixed amount of money, and they had to come up with
the solution that met the top level requirements, and how they got
there was their business. We’d help them along the way and give
them as much technical expertise as we could, but in the end it was
their responsibility. They knew that either they were going to come
up with a solution that was going to work, or they weren’t going
to get the follow-on contract.
Wright:
As we walk through your first year, talk about when you came on as
part of this team. You became the [NASA] Project Executive for SpaceX
[Space Exploration Technologies Corp.]—some of those first encounters
with them, and how you started laying out that foundation to move
forward with this new way of doing business.
Horkachuck:
They were really small back then. In fact they were in small buildings
in El Segundo [California]. They were in multiple different buildings.
They had some of their manufacturing and avionics in a different building,
and some of their engineering in a different building. It was spread
out. They’d still only been developing a Falcon 1 rocket, which
was a single-engine small rocket that was supposed to take maybe 1,000
pounds to orbit.
We wanted a much larger class rocket in order to take the capsule
to orbit, so they had to develop a whole new rocket, and the whole
capsule. They’d been launching out of Kwajalein [Marshall Islands],
which is way out in the middle of the Pacific, Reagan Test Range [Ronald
Reagan Ballistic Missile Defense Test Site], a series of small islands
and an atoll. The Air Force wouldn’t let them launch out of
Vandenberg [Air Force Base, California] for risk that their rocket
wouldn’t work and it would destroy some other national asset,
some other [launch] pad for an Atlas or Delta [rocket].
It was a big leap for them in those early days. They had a lot of
really good smart people that were very enthusiastic, but they had
a lot of work to go. They had some very early concepts, and they seemed
to do a really good job of coming up with concepts and developing
the concepts a little beyond what we would normally expect at a PDR
[Preliminary Design Review] level. They actually would start building
the hardware and testing some of the components even at the PDR level,
then they’d roll into a CDR [Critical Design Review] without
the level of maturity that we’d sometimes normally see.
They wouldn’t have all the drawings released, although they
were trying to move into more of the automation and electronic data
and not do as much paper drawings and design. That worked to some
degree. I think they are learning that when you’re still manufacturing
parts there still needs to be some amount of paper trail there. They’re
finding where that balance is today—how much can they do purely
electronically, and how much do they still need to go from the 3D
CAD [Computer-Aided Design] models to actually making drawings for
the individual parts that they’re building.
The advantages—they were a lot like the Lockheed [Martin] Skunk
Works [Advanced Development Programs]. It was a separate brand-new
company, and it didn’t have a lot of the baggage that a lot
of the bigger existing companies have on processes and how to do business.
So they could innovate in what processes and systems would work for
them—which were lean and efficient and would help them get things
done quickly, and which ones were burdensome.
They’re still, I think, learning some of that. They’ve
made a lot of progress today, they’ve come a lot closer to what
we’re used to than where they were when they first started.
Having that clean Skunk Works organization gives you a lot of ability
to innovate and do things more quickly and more efficiently.
Wright:
How were you involved in those first months with them? Were you in
residence with them helping them along the way?
Horkachuck:
Yes. We’d have milestone meetings on a fairly regular basis
in the early couple of years. Then, in between those, I eventually
set up weekly meetings with their lead engineers for the rocket and
for the capsule. We’d have regular tag ups to see what progress
they were making, what areas they were doing well in, how they were
progressing to schedule. In fact, I spent a long time trying to get
them to actually build a schedule, because they had really no idea
how to lay out a big project schedule when we started. They’ve
learned a lot over the years.
Wright:
You mentioned being able to share NASA’s expertise with them
as they were formulating those plans in those first months. Can you
give us some examples about whether you and/or your advisory groups—some
examples about how NASA was involved in those days. Did you help with
actual development of the spacecraft, or things other than helping
them develop schedules? What are other examples of what you did?
Horkachuck:
One thing I did was I tried to coach all the technical experts. Early
on I tried to hand select a lot of the folks that we were using as
advisory team members because I’d worked with a lot of them
in the past, whether it was on the Space Station Program or on Constellation.
I tried to pick folks that I knew really knew their subsystem and
technical area, and were reasonable to work with.
What I didn’t want to do was get a bunch of people that were
strictly by the book. “The requirement says this, and you have
to do it this way because that’s what the requirement says”
types. I’m one of those people that asks, “Why is the
requirement like that? What is the end goal that we’re trying
to get to with this requirement? Isn’t there another way to
get there if we need to?” The point of most requirements is
to make sure that the system works the way that you want it to. The
way we sometimes word requirements is a combination of a lot of history
and “That’s the way we’ve always done.” That
doesn’t go very well with innovation, so I tried to get people
that really knew their stuff and their system, and then I’d
coach them and let them know that this is different.
It’s not the typical NASA-contractor relationship where you
can’t tell the contractor how to do business for fear that you’re
violating some contractual requirement. “Giving a constructive
change” is the parlance I think in contracts, where if you tell
them, “You have to go do this,” they may go do it, and
then the government is liable for the cost and takes on not just the
cost liability, but also the technical liability if it doesn’t
work.
I would tell the NASA people, “This was a partnership. You can
tell them what you see the issues are, and explain to them some of
the things that you learned in the past, what’s worked in space
and what hasn’t worked in space and why. You can even suggest
how they may want to do it, and they may take that and they may not.”
I’d say, “They’re going to make some internal corporate
cost trades and tradeoffs on what’s the best thing to do is
for them. That may not necessarily fit with the way that you’d
recommend, but they’re trying to do things a lot more cost effectively
than we have in the past. Quite frankly, cheaper. They may come up
with a different way of doing it than you suggested, but they’re
going to listen and they’re very receptive to understanding
from our history and what’s worked and what hasn’t worked
in the past.”
I tried to coach them into a more open and friendly dialogue from
the start, when I got them on board the teams. Sometimes that worked,
sometimes it didn’t. There were some CAT members that I had
in the early design reviews that didn’t work out. I still had
to use some methods that we’ve done in the past. We call them
RIDs, Review Item Discrepancy. You’d write up a little discussion,
on a form, of what you see that may be a problem with the design and
why you think it’s a problem. And then we’d have to go
through that issue, understand it and how to mitigate it as we did
the rest of the review. The big ones went up through the board and
were reconciled.
When I started getting a bunch of RIDs in from somebody that were
just picking at the paperwork and the wording of something, I’d
try to coach them that we weren’t trying to get to that. We
were trying to get down to the meat of the technical areas and we
weren’t so much worried about “Is the paperwork perfect?”
If they continued down that route, I didn’t invite them to the
next review. So there was some turnover in the CAT teams over the
years. Some good ones moved on to different projects, but some of
them just weren’t a good match.
Wright:
Was their participation voluntary in the sense that they did it as
part of their normal job? Or did they become a COTS team member that
charged to a code to come work as part of COTS?
Horkachuck:
We had a charge code.
Wright:
Was it that if you asked them they could, but if they didn’t
want to that was okay too? There was no mandate that they needed to
have to do that?
Horkachuck:
Yes, we didn’t have that big a stick. At the time that we started,
there were a lot of major projects still going on at the Center. The
Shuttle was still flying, so the Shuttle employed a lot of people.
The Space Station was still going full-bore in assembly and had a
lot of people dedicated to it. Constellation was ramping up and still
had a lot of people. So it was really hard to get people. It was hard
to get people at JSC to give us the time of day.
I think the only reason early on that we did get some of the experts
that we did was because I knew them from the past and they had worked
with me on previous projects, and they found this interesting and
challenging. But otherwise they were dedicated full-time to other
programs. The JSC Center didn’t give us any allocation of people,
so we were more or less forced to go to other NASA Centers for a lot
of the help. That’s good and bad.
There are a lot of smart people throughout the Agency, but there’s
a unique skill set in manned spaceflight that you don’t have
in say some other research or unmanned centers. It just that that’s
not where their primary expertise lies. If I got a recommendation
about an acceptable material from somebody at an unmanned center,
it could be wrong because they don’t know the toxicology requirements
or the flammability requirements of manned spacecraft. So a lot of
times, I’d have to temper the responses I got from some of the
other organizations. That’s why it was really critical to be
able to pull out some key folks at JSC and at [NASA] Marshall [Space
Flight Center, Huntsville, Alabama].
Wright:
I’m curious, was there a type of broadcast system—you
knew what experts to tap into that you sent a request for help, and
then got responses? How did you tap the people at the other centers?
Horkachuck:
We compiled a list of names on a spreadsheet. First we looked at each
of the reviews. Myself and my deputy Warren [P. Ruemmele] and one
of our contractor support people would sit down and look at what review
was coming up, what the major topics of the review might be, and what
kind of technical discipline support we had. We’d get a feel
for what documents we were going to be getting in from SpaceX for
that review, then we’d map out each one of the documents to
a set of skills that we wanted it to be reviewed by: structures; thermal;
power; avionics folks; guidance, navigation and control; some operations
folks; maybe some ground ops [operations] folks. We’d map the
review to the types of technical disciplines that we wanted to have
for the review. We assigned each one of the documents to at least
one, if not more than one, of those technical disciplines, so we knew
we were getting coverage on each one of the documents submitted.
We had acquired over time this list of people that we could call on
and I’d send out an email to them en masse asking if they’d
be available for a review during this period of time that was coming
up and get responses back. If we got enough people in each one of
the technical areas then we were okay. If not then we’d go out
to some of the other centers and ask for additional help in specific
areas.
It took a while, but we eventually got a main point of contact at
each one of the different NASA Centers. They’d be able to go
out to their engineering directorates and pull some of the technical
support that we needed. But that took a little while to get going.
It’s a well-greased wheel now, but it wasn’t in the beginning.
Wright:
Just identifying people had to be a task in itself, trying to find
the right people at the right place, then having them carve out time
to help you.
Horkachuck:
A lot of times they were in the middle of another review that their
management said was more critical. There was a major design review
for Orion going on, and they didn’t have time to support us
because that was where the Agency’s focus was. Flying out the
Shuttle safely, continuing assembly of the Space Station, and then
building up the Constellation Program, Ares and Orion, were the Agency’s
focus. This was the back burner project. It was the side bet, literally,
is what it was. I think at one point I heard that the [NASA] Administrator
[Michael D.] Griffin was quoted as saying, “We’ll put
in $500 million—no more, no less—to this COTS commercialization
idea and see how it goes.”
Wright:
Look what happened. When do you think it didn’t fit into that
term anymore, being a back burner project or being a side bet? What
was the tipping point that moved you out of that spot?
Horkachuck:
Continual schedule slips of Constellation, Ares and Orion delays.
Kept on slipping further and further out to the point where it wasn’t
going to make it to Space Station. Shuttle’s retirement was
becoming more and more imminent and was more of a firm date. The things
on the outside moved around us and we were still tracking.
Granted, we originally were going to be a three-year development program,
and it almost took twice as long to get the flight off to Space Station
as what SpaceX originally proposed. I told you they had some difficulties
with scheduling early on. They were very optimistic on their schedules,
very aggressive in what they thought they could get done in a certain
amount of time, and had no schedule margin built into any of their
schedules when they built them.
I think, when we even signed the agreements, we knew that their schedules
were ridiculously optimistic, and assumed it was going to take quite
a bit longer than what they had proposed. But that was okay. As long
as they were making good technical progress I think we were happy,
because it wasn’t going to cost NASA any more.
On a typical government contract, if the contractor is late and the
schedule for a development program drags on by a year, you’re
paying a lot more money. It’s not just that the rocket takes
longer to build; you’re paying for a standing army of people
that are on that project for another year. So you’re paying
for maybe 10,000 people at $200,000 per year. That’s some cost
overrun compared to what you originally thought.
In this case, NASA wasn’t liable for any of those cost overruns.
We had a fixed milestone-based agreement. When they met a technical
milestone, we reviewed it. If they met those technical criteria, then
we paid them the amount that we agreed on for that milestone. If it
took them longer to get to that milestone, we didn’t pay them
any more. It was just the amount that we had agreed on. Any delays
were costs that they had to incur on their side, it was their contribution
to the project.
Wright:
Things were running pretty much on schedule with your reviews when
you first started, is that correct?
Horkachuck:
Early on. The first couple years I think were pretty much dead on.
Wright:
When it got to the point where [milestone deadlines] started to slip,
can you share with us—not specifics of the conversation, but
the conversation as a whole—how to get them on a track to move
forward? Were there concerns on the NASA side that since they had
begun to slip that it’d continue slipping, maybe not reach fruition?
Horkachuck:
Well, SpaceX was really scared that we would terminate them if they
missed a date in the early years, because the agreement said, “You’re
going to have this milestone in this month in this year.” It
didn’t give you a lot of flexibility; it said you had to do
it on that schedule. We laid out a schedule for the milestones partially
so that we knew what the funding profile would be for our internal
budgeting, just so that you had some kind of general guideline for
what we were doing.
There wasn’t a lot of language in the Space Act that said if
you miss a milestone what happens. I think what it said was if you
missed a milestone, NASA had the rights to terminate the agreement.
So they were really scared that if they didn’t meet a milestone
on a given date we were going to just end the agreement with them.
I don’t think even on the NASA side we knew exactly how to deal
with some of that early on, the first schedule delay.
We terminated Rocketplane Kistler mostly because they were making
no progress. It wasn’t that they had missed some milestones.
I think they even rewrote the Space Act at one point and let them
combine some milestones to try and help them, but they just weren’t
making progress. In the case of SpaceX, they were making progress.
They had made a tremendous amount of progress through the early design
reviews.
But when you get to the hardware build and assembly and test phase,
things happen. It’s just the nature of actually designing new
hardware and building new hardware and software. When you start integrating
all those things together, some things work that you thought were
going to work, and some things don’t work that you thought were
going to work. Then you have to go redesign it and try again. That
leads to schedule delays.
We were getting to that phase of the program, but they were still
making a tremendous amount of progress. I had lots of insight into
how they were doing technically on all the different subsystems, but
they weren’t exactly on the schedule that we had laid out in
the beginning. So we had some long conversations here with our folks
and legal to make sure that if they were showing good technical progress,
was it okay to let them miss a milestone date? Then if they met the
technical content of that milestone at a later date, was it okay to
still pay them for the milestone? The first couple times there was
some wringing of hands. We wrestled with that quite a bit.
Wright:
Were the eyes of the Agency beginning to look at what you were doing
at that time period? Or were you still on the back burner when you
were starting to make those decisions of letting them stretch that
date out?
Horkachuck:
At that point we still weren’t too far in the limelight, so
I think they weren’t all that worried about us. They had much
bigger problems to worry about.
Wright:
Were you given the authority, based on your position, where you could
make some of those decisions to allow them to have leeway one way
or the other? Or was part of your responsibility to come back and
get that authority from here within your program?
Horkachuck:
Alan [J. Lindenmoyer, Commercial Crew and Cargo Program Manager] had
the real authority, although I think he needed to still check with
his boss and with the rest of the NASA community, legal and other
folks, to make sure it was okay. I’d make recommendations based
on whether they had met the technical content of what the milestone
was, did I think they were making technical progress. I was basically
making recommendations on where we should go with things. I didn’t
have the ultimate decision making authority, Alan had the final sign
off on those things.
Wright:
We’re learning more about how the partnerships work and how
traditional protocol was set aside on some things, but it sounds like
on this it still went up the ladder.
Horkachuck:
In a lot of cases we still followed a lot of the protocols that were
done in the past. Even the proposal and procurement process, although
it was streamlined. It was done a lot more efficiently than a normal
procurement. They followed a lot of the same procedures and protocols
and did a lot of the things that you’d do in a normal FAR-based
procurement, even though it was a Space Act Agreement and they didn’t
necessarily need to follow all those rules. Just to make it clean
and not show even an inkling of favoritism or being unfair in the
selection process. They tried to follow the typical process in a lot
of respects so that it, by its nature, made sure we were fair and
equitable.
Wright:
As the process began for the development phases, how was the safety
community within NASA involved? Or even the safety folks from the
FAA [Federal Aviation Administration]. Can you share with us on how
they were brought in, or not brought in, and how they impacted the
vehicles’ development?
Horkachuck:
Right at the beginning of the project, we had a Chief Safety [and
Mission Assurance] Officer that was resident with us. In fact, Mark
[D. Erminger] lives right across the hall from me, so I’d have
him coming to all our reviews, giving me his observations as we were
going through different design reviews and understanding where they
were and what the risks were. He’d sometimes call directly to
the contractor and talk to their safety folks, get their Failure Mode
and Effects Analysis. At some point I think he did a Probabilistic
Risk Assessment, or got the contractor’s data on that.
He was really helpful, and he was reasonable. He understood that there
were trades that had to be made in order to actually get anything
done, so he was aware of what those trades were. I tried to explain
behind the scenes what was going on technically, and why there may
be certain issues where it wasn’t strictly the way we would
have done things in the past. He’d weigh that and balance it
with what his understanding of the overall agency’s safety philosophy,
and come back with recommendations in some cases, and just agreement
in other cases.
Wright:
Did the influence or impact stop here on his level? Was the NASA Headquarters
[Washington, DC] safety involved, or the FAA?
Horkachuck:
As time went on we started getting help from Headquarters. They requested
to participate in a lot of our reviews. It was okay, they were mostly
in listen mode. I didn’t see a lot of influence that put big
roadblocks in the development program. They were interested to see
how we were doing things, interested to know that Mark was an active
participant. He was reporting directly up to the Headquarters [Chief]
of Safety [And Mission Assurance] at the time, Bryan [D.] O’Connor,
so he was passing on a lot of the information.
They were more worried if we were going to go implement a crew program,
because early in the project we were both cargo and crew. The idea
was we were going to start out, see how they do with cargo, and if
they did a pretty good job with cargo then we’d turn them on
to do commercial crew as well. In fact the SpaceX Space Act Agreement
has an Option D in it that would have let us turn on the crew program,
the crew Dragon [capsule]. We just hadn’t exercised it at the
time. The outside organizations were much more interested, if we were
going to try to fly crew on board the vehicles.
Then, as part of normal safety, as we started to have missions that
were going to go to Space Station, the Space Station safety process
started to take over the normal phased safety review process. Phase
0, 1, 2, 3 reviews, hazard reports and going to the Safety [Review]
Panel. A lot of those safety requirements were Space Station requirements
for any vehicle that was going to rendezvous and attach to the Space
Station. There’s a certain number of requirements that you just
have to have in order to make sure the vehicle and the subsystems
on that vehicle are safe, and not going to damage the Space Station
or any crew member on board the Space Station.
Wright:
Were you the main liaison between SpaceX and ISS to make sure that
that was covered, or were there other people involved with that?
Horkachuck:
I did some of it. Originally our office had a position for that, but
Station really wanted to be the primary point of contact for most
of that, so I just worked with their folks. A lot of those processes
were standing processes in place, so once I got the SpaceX folks to
learn how to play into them I didn’t need to be directly involved.
There were too many different things going on all at the same time
for one person to be a bottleneck of information. There’s no
way I could have been the liaison in the middle of all that, as well
as making sure the development was going right, and on schedule, and
we were getting payments to them.
Wright:
It’s another nontraditional way of doing business to allow the
flow and not have to go through one central point
Horkachuck:
To actually trust people to know what they’re doing. They always
knew they could call me if something didn’t seem right and they
were getting into a problem that they couldn’t get out of. Then
I could talk to both the NASA side, whoever was raising the concern,
and understand, mediate our way through it. It wasn’t all that
frequent, quite frankly. Once in a while we’d have to do that.
Wright:
If I understand correctly, going to the Station was not mandated.
That was something that SpaceX could choose to do, is that correct?
Horkachuck:
I think that was the way it was originally written, although the whole
point was to be able to take cargo to the Space Station, so not going
to the Space Station—I guess you could not do that, but I don’t
know what the point would be.
Wright:
So far as you understood, that was always their intent, to bring cargo
to the Station, from the beginning of their development phase.
Horkachuck:
We’d given them information on what size packages they were
going to take and be able to accommodate, and how much power you’d
need to give to powered payloads, and the kind of cooling you’d
need to give to powered payloads. They were all directly from our
requirements that we had from EXPRESS [Expedite the Processing of
Experiments to the Space Station] racks that were on the Space Station,
and cargo bags that we’d normally fly in MPLMs [Multi-Purpose
Logistics Modules]. It was all directly logistics for Space Station.
They could not go into Space Station I guess, but they would have
never gotten the CRS [Commercial Resupply Services] contract.
SpaceX had three flights in the basic agreement. The first flight
was just an orbital flight. Prove that the rocket works, prove that
the capsule can get to orbit, can maneuver in orbit, communicate in
space. Prove that their thrusters worked to make it point at a target,
and that it could reenter, that the heat shield worked and the parachute
system worked.
Quite frankly that was the one that worried me the most, the first
flight. They had one flight of the [Falcon 9] rocket before that COTS
1 [demonstration] mission, then the flight with the Dragon on board
that was actually going to maneuver in orbit. Did their thrusters
that they built themselves work, did they communicate with the TDRSS
[Tracking and Data Relay Satellite System] satellites correctly? Then
did the heat shield work through reentry, and did the parachutes work?
I influenced them to do some more parachute testing than they had
planned in the past. They had tried to leverage on the Orion parachutes,
buying them from the same company that was building the parachutes
for Orion. Then the Orion schedule dragged out so long that they never
finished certifying and qualifying parachutes. So now SpaceX was left
out in front all of a sudden, without the budget to do the full qualification
program.
We suggested they may want to do some things like some more drop tests,
some more parachute tests. They eventually did a helicopter drop test.
We learned from some of the things that the Orion folks did, dropping
out of the back of airplanes, and how some of the systems to stabilize
after you come out of the airplane cause more problems than the actual
test itself.
We tried to keep it a little simpler. Some folks would say that didn’t
prove as much as you necessarily could from some other tests, but
it showed the basic systems worked. They tried to keep it fairly simple.
Simple release of drogue parachutes, and then have the drogues pull
out the mains and make sure they were in a good packaging, would inflate
well. The capsule comes down really slow if all three mains are inflated;
it’s a very soft ride. Even if one or two of the mains is out,
it’s still coming down slower than Apollo or Orion.
Wright:
That’s interesting. Share with us what it was like on that test,
the whole launch aspect. You said you were concerned for that first
one because there was so much that had to go right. You really had
put in a lot of investment, your technical expertise and your time
with them. If you can just share with us what it was like to be there,
and know that things were launching and working the way you wanted
them to.
Horkachuck:
Well, obviously it was nerve racking. The uphill ride is always really
worrisome. Getting a rocket into space—those first nine minutes
or ten minutes really worries you, because lots of things are going
very fast. There’s not time for something to go wrong and still
have a very good outcome. So that was nerve racking. We did a lot
of things to “buy insurance” is probably the way to put
it.
Warren and I scoured the Agency—and other agencies—to
find assets to help us have some additional insight into the demo
[demonstration] missions, like bringing up some of the Shuttle radar
systems. Some of the debris radar that they had implemented after
Columbia [STS-107 accident] to look for foam coming off the Shuttle,
we train those same radar on Falcon 9 for the ascent phase. If something
goes wrong, we would have very high quality images of what might have
been coming off the vehicle, so we’d be able to do a reasonable
investigation if we had to. Luckily we didn’t have to.
We sent the SRB [Solid Rocket Booster] ships up the coast, and out
farther than they’d gone before on a Shuttle mission, so that
we could optimally deploy them for staging events and other events.
We looked at the history or rocket launches with The Aerospace Corporation.
Where do you typically have failures of rockets? A lot of the events,
when things are changing, is when things don’t seem to work
right. So we tried to deploy assets, both optics and radar, to be
able to see those specific events.
Then we briefed all the way up to the [NASA] Associate Administrator
[for the Exploration Systems Mission Directorate] what our plan was.
We’d even worked with a group out of [NASA] Langley [Research
Center, Hampton, Virginia] called HYTHIRM [Hypersonic Thermodynamic
Infrared Measurements] that has used some Navy P-3 Orions [aircraft]
with some optics on them to look for Shuttle reentries. They started
looking at what happens if some of the tiles aren’t in place
right, or some of the gap fillers are sticking out.
We used a lot of the same assets that they had to watch the reentry
of the Dragon capsule, so that if something went wrong during reentry
we at least had a chance of figuring out what the heck happened. You’re
in a data blackout during reentry, so you’re not going to be
getting telemetry back. You might get lucky and be able to recover
some data recorder out in the middle of the Pacific, but in order
to have good insight to what was going on we deployed a plane to go
look and watch the reentry, especially during the max [maximum] heating
phase. Luckily things went really well.
Wright:
Doesn’t sound like too much luck. Sounds like a whole lot of
preparation, a lot of analysis.
Horkachuck:
We did a lot of analysis, and we did a lot of work with NASA experts.
But we tended on this project not to go as deep into any given technical
problem as maybe we would on a manned mission. We didn’t look
at all the corner cases to the absolute extreme; we didn’t do
a bunch of ground-based testing of every single thing. They did a
bunch of arc jet testing of the basic material. [NASA] Ames [Research
Center, Moffett Field, California] had actually developed the main
heat shield material, something called PICA [Phenolic Impregnated
Carbon Ablator], for some unmanned missions. They worked with SpaceX
to design the heat shield, and used some of the Ames code to figure
out what the heating rates would be and how thick the material would
need to be.
SpaceX built a lot of margin into their design early on, so that if
they didn’t do some calculation just right they had some robustness
in the design. Granted, it added weight to the vehicle, but it gave
you some flexibility for changes that inevitably happened. Or finding
out that maybe they hadn’t designed to the right requirement
for some structural margin for Space Station, or some load case that
they hadn’t thought of before.
They had enough robustness in their design that they didn’t
have to completely scrap the design and start all over again. They
could redo the analysis and show that they still had margin. That
saves a lot of money in the project development cycle because you’re
not scrapping out the hardware design and starting all over again.
You don’t completely optimize performance, but I like their
general philosophy where you retain some margin early in the first
couple flights, and then you maybe redesign later to gain more performance.
As opposed to trying to optimize everything right from the start,
and you spend a lot of time and a lot of analysis trying to optimize,
and then something from outside your influence throws you a curveball
and you have to go redesign the whole thing anyway because you have
no margin, you designed right to the limits.
NASA tends to do that a lot. Not sure it’s necessarily all that
efficient. Certainly builds you a spacecraft that is a Ferrari as
opposed to a Volkswagen, but if all you need to do is have the function
of a Volkswagen—
Wright:
You’re giving credit for that decision to SpaceX, so that was
their decision to not fully optimize and retain that margin with their
design?
Horkachuck:
It’s their general design philosophy. They tended a little bit
more towards the way the Russians do business. They design something,
and test it, and then see how it worked, and then make tweaks, and
build it again and test it again. They used a lot of state-of-the-art
computer code to design the hardware, don’t get me wrong, but
they left some margin in there and didn’t try to overoptimize
right off the bat, which I think is good.
You have to have a booster that’s going to be able to take that
extra mass to orbit, and I think we’re paying for a little of
that right now. They’re working on the next generation booster
that’s going to give them a lot more performance to orbit. Depends
on where you want to get to and if you’ve got the flexibility
to be able to continue development and make changes.
There used to be somebody at Headquarters that was trying to do what
they called spiral development. [Rear] Admiral [Craig E.] Steidle
had proposed that in the Agency. I don’t think most of the Agency
knew what he was talking about, where you design a little, test a
little, fly a little, redesign, test, fly. You keep on iterating the
design and evolving it. Not trying to come up with the optimal design
right off the bat, which is virtually impossible to do. But we try
to do it on almost every damn project, and it shows in how some of
the projects go.
Wright:
After that first test flight, were you involved and invited to sit
down with them to hear their debriefing and their evaluation of what
they felt went right and went wrong?
Horkachuck:
Oh yes. The flight itself was a milestone—successful demo flight.
We had determined a set of mission objectives for that demo flight.
We jointly agreed on them and had a document that defined what those
top level objectives were. It was short, maybe a couple pages. The
major elements of the goals for that flight I think were on one page.
In order to get payment they had to show that they had met those major
objectives. So we had a debriefing two days after the flight, where
they showed proof that they had met each one of those objectives.
Then a couple months later, when they had gotten all the data back
and processed all the data, we had a much more detailed debrief for
both the rocket and the capsule. We went through what systems worked
and what systems didn’t work, and how they were going to fix
the ones that didn’t work for the next flight. Or if they were
making such significant changes in the design from the first demo
flight to the second demo flight vehicle that it didn’t matter
anymore. The vehicle changed significantly from the first flight vehicle,
primarily because it had different mission requirements.
Wright:
Which leads to a question—what measures did you have to take
to maintain the level of confidentiality that NASA and SpaceX have
agreed to on the level of this partnership? We understand there’s
so much of what they do that you’re privy to, but you don’t
discuss outside of your NASA counterparts. I’d like for you
to give us your definition of how the proprietary information, or
confidentiality of their technical expertise, stays within their own
environment. It’s not a traditional role of NASA when they develop
spacecraft.
Horkachuck:
Yes and no. In a typical contract, NASA is paying for the design and
basically buying the design and the vehicle. So in that case we more
or less own the design. We’re paying a contractor to do it,
but we own the majority of the rights to it. There are some areas
that are still sometimes corporate proprietary. If they have come
up with a certain process or material through their corporate IR&D
[Internal Research and Development] funding, or some other method
that is unique to them, that you can’t share with anybody else.
Typically the government would own the design because we paid for
it. In this agreement, SpaceX retained a lot of the rights because
they were putting in a lot of their own money into the development.
As part of the agreement, it lets them keep a lot of the rights to
it unless they for some reason default.
We tried to keep it fairly tight, and remind anybody that was working
on the project that they can’t share it with other contractors.
NASA has so much contractor support, especially on big programs, Space
Station and Shuttle Program. We flew a couple [SpaceX] DTOs [Detailed
Test Objectives] on the Shuttle and some other hardware on the Shuttle.
You have to work with some of those contractors during the integration
process, so we had to get Non-Disclosure Agreements signed with those
individuals or their corporate to make sure that they’re not
using any knowledge they have on the SpaceX design for some other
vehicle that maybe they were designing.
In some cases they were really worried about Boeing, because Boeing
was a potential competitor on the crew vehicle. Boeing is the prime
integrator for the Space Station, so we had to firewall off some of
those people from working on any proposals for a crew vehicle for
Boeing. That was all done with the contracts folks.
In general we tried to keep the data in a secure repository that had
very limited access to just people that we had cleared for the project,
had a right and a need to know and had signed agreements either themselves
or with their corporation on that particular contract that had the
nondisclosures in there.
The other aspect that we had to deal with is ITAR [International Traffic
in Arms Regulation]. It’s a launch vehicle, so got to make sure
that we’re only dealing with U.S. persons, because you can’t
share ITAR information with foreigners in any sense. So we had both
of them coming at us like a double whammy to make sure that we kept
things close to the vest.
It made it difficult because it was hard to get data transferred between
people when you needed it to be transferred to certain people, trying
to follow the limitations of how they’re supposed to handle
that type of data.
Wright:
I can’t even imagine. In December 2008, SpaceX was awarded one
of the two Commercial Resupply Services contracts. What role was yours
in that process? Did you have one?
Horkachuck:
I didn’t have a role in the process. I was just trying to make
sure that we built and demonstrated the basic capability. They had
a whole separate group doing procurement for the services phase of
the contract. I knew that ours was the demonstration phase, and if
they did the demonstration phase successfully there was a follow-on
services contract. I was just making sure that it worked, then I figured
the rest of it was going to take care of itself.
Wright:
From what we understand it did work. There were other pieces of the
demonstration phases. Would you like to talk about the other flights
and the roles that you played with moving those forward?
Horkachuck:
There were supposed to be two other [demonstration] flights, a total
of three flights. The second flight was supposed to take the capsule
to orbit, be a more capable capsule. It would be in space for nine
to ten days, and really let a lot of the systems start to see how
well they worked in space over a long period of time, through lots
of thermal cycles and orbits around the Earth and day-night cycles.
Just see how well they tolerated the radiation environment.
The surface objective was to do a flyby of Station and close the comm
[communications] link with Space Station. There was no comm system
on board Space Station that would talk to a visiting vehicle other
than the Japanese [Japan Aerospace Exploration Agency] system that
they had built for the HTV [H-II Transfer Vehicle].
The ESA ATV system came up through the Russian side of the ISS vehicle.
We had thought about that briefly, but then you were berthing to the
Russian side of the vehicle. We didn’t know if the Russians
were going to charge us a docking fee, and you had to go through the
much smaller hatch that was available on the Russian side. Ruled that
out early in the program.
I guess I influenced SpaceX quite a bit. I had worked with the Japanese
quite a bit on Space Station, and although they’re very good
technically, they are very slow and methodical in their decision making
process. I knew that working with the Japanese team probably would
clash with the SpaceX culture and way of doing business. SpaceX wouldn’t
understand why it would take three or four weeks to make a decision
on some simple question. Whereas the Japanese culture would normally
go back and have some team huddle offline before they came back and
gave you a real yes or no.
Quite frankly, I thought that the basic premise of COTS was to develop
a U.S. capability to take cargo to orbit. SpaceX was building a vehicle
completely in the United States. They were building their own engines,
their own tanks. The rocket was a U.S.-built vehicle, their capsule
was all built in the United States. I found it disconcerting to think
that we were going to build all that capability, and then have the
last mile getting to Space Station dependent on a foreign government.
The idea behind COTS was to make sure that we weren’t dependent
on foreign governments anymore. We had the Shuttle, which gave us
this huge capability and was a tremendous asset. Then the only other
way to take cargo to Station was going to be HTV and [ESA] ATV [Automated
Transfer Vehicle], foreign governments or the Russians. I think the
idea behind COTS was to build a capability in the U.S. I didn’t
want us to be held hostage for the last part of the mission because
we didn’t have a communication system.
I started working with the Space Station Program and the comm folks.
It seemed to be too much to go put a new antenna outside Space Station
at the time. Space Station wasn’t paying that much attention
to the COTS program yet, and having us do an EVA [Extravehicular Activity]
to go install an external antenna, and either finding penetration
through the pressure shell or making a new penetration just didn’t
seem like it was in the cards.
So we worked with the comm folks and decided that we could use the
UHF [Ultra-High Frequency] system and antennas that were already there
to talk to the spacesuits, and build a comm system that would be able
to talk to the visiting vehicle. It had a lot of drawbacks because
it was UHF, and it was a shared frequency band so we had to deal with
the FCC [Federal Communications Commission] and the NTIA [National
Telecommunications and Information Administration] for regulations
on how much power you could communicate out of the Space Station.
It was originally installed as just being for communications to spacesuits
and very close to the Space Station, very limited power. Whereas we
had a vehicle coming up from a far distance that we wanted to be communicating
with at least 20 miles away, so you had to turn the power up. There
were lots of complications and negotiations with the powers that be—both
the frequency managers here at JSC, and at Headquarters, and then
the whole rest of the government regulatory process—to be able
to use the comm frequencies.
But in the end it gave us a capability to have a comm system that
wasn’t a foreign government’s comm system. We ended up
launching two boxes that SpaceX developed in middeck lockers. I built
a bunch of middeck lockers for all the payloads back in my days with
Space Station. So I loaned them a couple middeck lockers to install
their comm system into, because it got us away from a bunch of structural
analysis and safety hassles.
It integrated easily into some EXPRESS racks that were already on
orbit. Then we hooked up the wiring. You need cable harnesses, so
the crew had to go lay in some new cabling to tie into the existing
UHF system. We ended up with two boxes on board Space Station that
talked between the Space Station and the upcoming SpaceX vehicle.
The objective of the second mission was to close the communication
link with that new system on board Space Station, make sure it really
worked, before we came in a lot closer to Space Station. Part of that
system had a bunch of switches on a switch panel for the crew that
if it looked like the vehicle wasn’t following the right approach
corridor to the Space Station, they could hit an abort and it would
send a command to the upcoming SpaceX vehicle to abort, fire its thrusters
to back away from the Space Station.
So it was very important that that comm link worked. It gave us data
on range and range rate, information about the vehicle approaching
to make sure it was still okay to continue the approach. That was
what the big ticket item was for the second demo mission. Go check
it out before you fly a final approach on the third mission, which
was to finally get all the way up to the Space Station, berth with
the Space Station, and then demonstrate that you can transfer cargo
in and out.
We had some problems with the MPLMs on the early flights, where getting
racks out of the MPLM was pretty easy, but getting them back in for
return and getting them bolted back down turned out to be a little
problematic. There’s some things that “the best laid plans”
and thoughts about the design—when you finally get into space,
you realize that things expand a little bit because you’re in
an airless environment and you’re pressurized. Well, take the
racks out, they don’t go back in the same as when they came
out because things move. You’re optimizing design for weight,
but then the structure isn’t as rigid and changes a little.
We wanted to prove all that out in the final mission, that it could
do all the basic functions of delivering cargo to and returning cargo
from the Space Station without issues. Partway through the program,
SpaceX approached us and asked if it would be possible to combine
those two missions, the second and the third demo mission. At first
we didn’t receive it very well. It seemed like they were looking
for ways to just save money by eliminating a flight, and adding a
lot more risk.
In the end we did a lot of work with them to do some integrated system-level
tests. A lot of things like full-blown thermal vacuum test of the
capsule, a full-blown EMI [Electromagnetic Interference] test of the
vehicle. A lot of deployment tests of a lot of their mechanisms, the
solar arrays, their door and grapple fixture. A bunch of testing of
the LIDAR [Light Detection and Ranging] and proximity ops sensors
that were going to be used on that final approach to Space Station.
We added a bunch of new milestones and tests into the overall program
that they didn’t have in their original plan. As we added all
those more traditional tests that NASA would have done on a system,
we got more comfortable with then being able to combine the two missions
and not being so risky that it was just a throwaway mission. We came
to another fairly reasonable compromise in my opinion, doing some
things to reduce the risk, but still meeting some of their objectives.
It also saved schedule and manpower for SpaceX, since they had been
taking quite a bit longer. They were running into some financial issues.
As I said, they were still having to pay for this standing army, more
or less, that was working on the program, even though they weren’t
meeting new milestones and getting new payments. So it helped them
in that they only had to design one more new vehicle.
There would have been slight differences between the design of the
C2 vehicle and the C3 vehicle, so it saved them a little bit of nonrecurring
engineering time, as well as just time to complete. Although that’s
a double-edged sword, because the flyby mission wouldn’t have
had as much insight and verification that the Space Station folks
required of it as the mission that came all the way up and approached
Space Station. They had to do a lot more work to get Station comfortable
that their vehicle was safe and ready to fully approach Space Station.
So, that made the next launch seem like a long time from the first.
Wright:
Were you surprised, or did you start to see some hints that they were
going to want to combine those two missions?
Horkachuck:
They’d been hinting at it for a while I think. I wasn’t
sure it was a great idea at first, seemed like the benefits to NASA
weren’t all that good when they first proposed it. They did
caveat it, always saying that if for some reason they were unsuccessful
they would go fly another mission and try again. We were still getting
what the original value was out of the original agreement. We’d
still, if need be, get three flights. Turned out this got us there
a little bit faster than if we had had to really need three flights.
Like I said, we did a lot of work with them, both the Station folks
and some of our CATS support folks, to help them see where some big
integrated system tests might be useful. We found some problems that
delayed the flight by a little bit. Some of those big system tests
found real problems that would have been a mission failure if we didn’t
catch them on the ground.
Wright:
I guess that’s the success of melding the new with the old.
Horkachuck:
Yes, sometimes the old guys know what they’re talking about.
Wright:
Combining all these efforts, there was a lot on the line when they
launched that flight.
Horkachuck:
Yes. I think there were a lot of hidden agendas as well. There was
a lot of questioning, “Does this whole COTS thing work?”
There was a lot of politics in the political world thinking is this
the smart thing to do, should we really be messing with COTS and this
whole commercialization, or should we be doing it the more traditional
way.
Wright:
You’re no longer on the back burner now. You were in the forefront,
people were watching.
Horkachuck:
Yes, that whole commercial crew thing was starting to take off. Quite
frankly, if that flight was not successful I’m not sure commercial
crew would have taken off. There was a lot of scrutiny as to, “Is
this new approach going to work?” and “Could they build
something that was reasonable and successful?” I think a lot
of people breathed a sigh of relief after it achieved its objectives,
because I think there would have been a lot more scrutiny and rethought
about “Are we doing the right thing with this approach?”
if it wasn’t successful.
We had enough history, based on rocket developments over the years,
to know that in the first three flights of a new vehicle, there’s
normally a failure. A lot of times, it’s a catastrophic failure
of a new rocket, whether it was built as a commercial rocket or the
government had sunk its full resources into that development. The
curves typically show in the first three flights there’s usually
a problem, then it levels off for a little while.
Then, at about six or seven flights, there’s another spike up
in failures as they go from the development organization to the production
phase, and lose track and focus as they transition from the guys that
designed the vehicle to the folks that are more the operations and
builders. Seems like there seems to be that spike later on as they
go into production.
We knew there was a lot of risk. The FAA and range safety, as part
of their normal process for licensing and certifying that the flight
is okay to launch, do calculations based on history and what the understanding
of the vehicles are, what is the likelihood and probability of the
vehicle being successful. So yes, there was a lot of risk on a lot
of those flights.
Wright:
Did you find yourself in a position of explaining, and possibly justifying,
more of the processes of reaching that flight as you approached? Were
you in meetings with the higher ups?
Horkachuck:
Yes, we ended up briefing both here at the Center and at Headquarters.
We managed to keep it fairly streamlined and not have to brief a lot
of different organizations and panels. We managed to get them all
pulled together. Mark helped quite a bit influencing, based on some
of the reviews that had typically gone on for Shuttles, and combining
them together with briefings to the Associate Administrator. Making
sure that all the parties were in the same room at the same time,
we didn’t have to do it multiple times. That was really helpful.
Wright:
That’s helpful, yes. Did you ever feel like you didn’t
have the support of upper management to continue on the journey that
had begun?
Horkachuck:
No, I didn’t necessarily have that feeling during the majority
of the development phase. I had one manager at one point that went
on vacation and gave me some of the wisest words I have gotten in
my career. I asked him, “So what’s my authority while
you’re gone?” His response was, “Proceed until apprehended.”
And I’ve had people tell me over the years, “Ask for forgiveness
and not permission.” That is the way you really get things done.
Wright:
At least you had a little more flexibility in this environment than
you would have before.
Horkachuck:
I was happy not to get too much help. They gave us the authority to
try and make the right decisions. Some of the upper management are
really really smart and have a lot of experience. They were able to
help us, point out things that we should look for based on their experience.
I never found our direct line management to be a burden, they were
really helpful. It was just really good, it’s been a pleasure
working with them.
Wright:
It’s always good to know that you’ve got someone taking
care of that side. The days leading up to that last launch, were you
constantly with the SpaceX folks as part of their whole process?
Horkachuck:
I was there with their team at the launch site during the final processing.
They do a lot of things that are a little bit unique. They test fire
the integrated first stage out in [McGregor] Texas and make sure that
the whole integrated stage works. A lot of companies don’t test
fire their rocket before they go fly it. Then they do a wet dress
rehearsal on the pad, which a lot of the companies do. Fully fuel
the rocket, go through simulated countdown, make sure all the systems
are working.
But then they go through one that’s a static fire on the pad
where they actually light the engines for a couple seconds, still
hold it down and make sure things are working. Part of that is because
they designed their engines to be a little more robust than typical.
It paid off in some cases. We went through a countdown and they actually
lit the engines, and then had a check valve fail and aborted.
If we had done that with the Shuttle, it would have been months and
months before we flew again. They turned it around in a matter of
days. They found where the check valve was that had failed, had pictures
of the part that had come loose in it, knew exactly why it came loose.
We jointly discussed it and reported to our upper management, changed
it out and within a couple days they launched. They turned around
and had the vehicle ready to go again. If you’d lit the engines
on a Shuttle, you’re not going again in a few days.
Wright:
Talk to us about the launch, and the pros and the cons that came from
the results of all that work. What was the best part for you, the
relief of it working out as well as it did? If you had to find something
that didn’t work out well from that launch, or maybe the best
lesson learned that came out from the mission itself.
Horkachuck:
I was actually in shock for days, if not weeks after, that everything
worked as well as it did. We had done a lot of work, and a lot of
due diligence and had a lot of review and tried to find where the
problems would be, but as complicated and as complex as the whole
thing was—and then to combine the two missions together, add
the complication of trying to do both missions all in one—it
really surprised me that it all worked as well as it did.
The only thing that got a little dicey is on final approach when some
of the rendezvous and prox sensors weren’t reading the same
values. There was some built-in aborts that we were worried were going
to trigger and have the vehicle back away. So it took longer on that
first approach to come into Station, but the SpaceX ops team and the
[NASA] MOD [Mission Operations Directorate] ops team worked really
well together, and had done a lot of training and simulations together,
and were pretty comfortable with the data they were seeing that they
weren’t too far off from each other.
Quite frankly, the crew could see the vehicle and knew where it was.
It may have been more of a risk to really abort and try again than
to just proceed. Then Don [Donald R.] Pettit did a great job capturing
the vehicle, it’s amazing how fast he grabbed it.
Wright:
It’s nice to have competent help on the other end.
Horkachuck:
Don was amazing, yes. He had done some stuff for us when I was in
the Payloads Office. We had some freezers up there that had, after
a year, died on orbit. Thought, “All right, why don’t
we give this guy a shot at trying to fix them?” They were never
designed to be repaired in space, so I had our guys build basically
a Chilton’s [auto repair] manual. I said, “Go take another
unit that we have on the ground, take pictures of all the screws you
have to take out, step by step, and lay it out in pictures. Little
arrows pointing to what to do next and everything. Send that up.”
It violates all the typical MOD procedure rules, but quite frankly
I don’t care. Either he’s going to fix this thing and
it’s going to work, or it’s trash.
He was hilarious. It turned out it was his birthday, and he was thanking
us for giving him something fun to do on his birthday. Completely
disassembled this thing and resoldered the leads to the thermoelectric
chips, and actually got it up and running for a while. That proves
that you can do some things on orbit that we didn’t think you
could do. Didn’t have to be all these ORUs [Orbital Replacement
Units] that we typically thought for spaceflight. Very important if
we do a Mars mission, where you can’t call home for spare parts.
He’s an incredible guy. Some of the Saturday morning science
that he did with the students on the ground—it was nice to have
him on the other end, when our vehicle was coming up.
Wright:
I bet it was. Then of course it returned home safely, and hopefully
as you wanted as well.
Horkachuck:
Right, reentry. It didn’t get very warm. The heat shield worked
like a champ, even on the first flight. I remember looking at some
of the thermocouples they had underneath the heat shield, and I was
thinking, “Are we getting still data? The temperature hasn’t
changed.” It just didn’t get through the heat shield.
Wright:
How do you feel like your relationship with SpaceX changed from the
day you walked in and met them until the day you sat down at a debriefing
after this last mission? How would you explain that relationship?
Horkachuck:
It was an evolving, trust-building partnership I think. As new people
came in—I’ve been around more than 90 percent of their
company. I had a lot more history than most of the employees at SpaceX
on what was going on and how things worked. When they first came in,
they had the “I know it all” kind of attitude. It took
a while for them to learn that maybe they could use some help in some
areas.
I think what they were constantly fearful of was getting slowed down
and bogged down by the NASA process. It was a bureaucracy on making
decisions, so I tried not to have anything that would really slow
them down too much. More of that “proceed until apprehended”
philosophy.
Wright:
What do you think you learned from them in this whole process?
Horkachuck:
It’s unique to me—when we talked to them about a given
part or component in the design, they’d say, “Well, we
could go buy this from this vendor, but it’s like $50,000. It’s
way too expensive, it’s ridiculous. We could build this for
$2,000 in our shop.” Almost every decision that they made had
cost built into it.
It was unique because I almost never heard NASA engineers talking
about cost of a part when they were making design trades and decisions.
They were worried about is it going to work, is it going to work reliably
and safely and meet all the requirements. Cost generally, although
a factor, was never really in the forefront as much as mission success.
Here they were making more trades on, “Well, you could do it
that way, but this way is a whole lot cheaper and probably just as
good.” That was a different mindset, and I think something that
maybe the rest of the Agency needs to look at a little bit more.
I worked with some really great people over in the Space Station Program
on being able to use commercial electronics on their vehicle. NASA
is used to using these S-level rated parts, components that go into
all the avionics boxes. They’re either RAD [Radiation Absorbed
Dose] hard and/or screened through this process to make sure that
they’re highly reliable. SpaceX wanted to use more of the commercial-type
parts.
Turns out that both the aircraft industry and the automotive industry
have done a lot of work with a lot of these commercial component part
vendors to make sure that the parts coming off the line are really
reliable. Because you can’t have cars breaking down all the
time, or they’d lose their shirt. Or airplanes. So, a lot of
the reliability things that we’ve been concerned about through
some burn-in process and testing, they catch. A lot of the vendors
seem to be building much more reliable electronic components that
they could use in their systems.
Then it was a question of how do we deal with the radiation effects.
We got SpaceX plugged in with [NASA] JPL [Jet Propulsion Laboratory,
Pasadena, California] and some of our JSC ISS radiation experts that
have been doing testing. I tried to get them to do testing early on,
based on some of the work that we had done with some of our payloads
where we’d sent them to one of the universities, I think Indiana
[University, Bloomington], and done some level of radiation testing.
It may not have been exactly the high rel [reliability] radiation
testing that we’d done in the past, but it gave you a good screening,
gave the payload guys a good feeling as to whether or not their parts
were going to work in space. I tried to get them to think about doing
some of that testing early on, and they didn’t want anything
to do with it. Station finally put their foot down and said, “Look,
you’re not going to make it to Station unless you prove that
you’re not going to have all these radiation effects.”
Then they eventually did a lot of the radiation testing that we had
talked about.
It basically showed that through smart part selection and some limited
amount of this type of radiation testing at the more integrated component
level, you could get pretty good reliability in a radiation environment.
The combination of that and the redundancy that they had built into
their system let you tolerate some radiation upsets. And reset, and
be ready to work again.
What was really important for me is back before I even came to Space
Station I was working out at Ames for the centrifuge project. We’d
done a lot of cost modeling trying to figure out how much the project
was going to cost, so I’d done a lot of work with some price
modeling people. We did a lot of iterations on the design and concept
and folding into the price models.
What I found was no matter how I sliced it and diced it into the price
models, the avionics subsystem almost always came out as the longest
lead time and highest cost. That, and the software which wrapped in
with it, drove the project. You could have thousands and thousands
of pounds of structure and just a small amount of avionics, and it
still was the longest lead time and biggest cost subsystem.
Finding ways for the Agency to be able to build avionics faster and
cheaper, I think, is going to be a huge savings for projects in the
future. Just my knowledge of overall project development says that’s
always going to bite you. If you can cut down the cost in that phase
of the project and the scheduling in that part of the project, you’re
going to have some big paybacks.
Wright:
I know we have more we want to ask, but we’re about through
for today on our time, so you can get back to your life as you knew
it before we entered it. So we thank you, and we’ll see if we
can catch up with you again.
Horkachuck:
All right.
Wright:
Thanks.
[End
of interview]
Return
to JSC Oral History Website