NASA Johnson Space Center
Facilities Oral History Project
Edited Oral History Transcript
Ned J.
Robinson
Interviewed by Jennifer Ross-Nazzal
Houston, Texas – 7 October 2009
Ross-Nazzal: Today is October 7th, 2009.
This interview with Ned Robinson is being conducted at JSC for the
JSC Facilities Oral History Project. The interviewer is Jennifer Ross-Nazzal,
assisted by Sandra Johnson. Thanks again for taking time to meet with
us this morning. I thought we would start off by just talking about
your career with JSC, if you could give us some of the highlights.
Robinson: I started working here in
August 1974, the 19th, matter of fact, and straight out of college.
I wasn’t really associated with the laboratory at that particular
time. I was an intern. So I was working projects. They want you to
build up stuff. I got into the laboratory around about 1975. The project
I was working on was part of the initial look at how they were going
to do communications for the Shuttle Ku-band system, where they were
going to use a different technique than what they had been using.
I went down to the laboratory. Then once I finished that, I was more
or less working in the laboratory. So I’ve been here since ’74
working.
I started out as a junior test director. What that means is that you
help the test director conduct the test. You’re working with
the contractors in trying to decide which direction you’re going
to go, does the data look correct. From there I became a senior test
director over a number of years, I don’t remember exactly when
that crossover occurred. Was that around ’90 something, ’91,
’92? Basically then I was responsible for all the testing, being
the test director. I had a junior too to take care of that, but basically
you work with the contractor in getting the test set up, what do you
need, what does the customer expect, what are the objectives of the
test, what equipment are you going to need, what people are you going
to need, all that. You do it beforehand. Then you run the test, make
sure the test results look correct, and direct the team to where we
need to go next, and then write a report on the test results.
I did that for quite a while. Then somewhere in ’95, I think,
is when the person who was over the laboratory moved up. They decided
that I was a good candidate to become the laboratory manager. As a
laboratory manager, it was not so much doing tests, but I was still
overseeing how the whole team was getting ready for a test, scheduling
the test, what the priority was. That’s working all the administrative
stuff: contracts, evaluations of the contractor, and all of that.
I’ve been doing that since that time, as well as the civil servants,
I was the lead for them for the laboratory. Any and everything about
the laboratory I manage. The environmental stuff when there’s
environmental problems. I have to deal with the facility manager,
says, “Hey it’s too hot, too cold, too wet, too whatever.”
I work with them on trying to keep the laboratory up and running;
when we need to shut down the whole laboratory—that’s
when hurricanes are coming—making final decisions on what we
need to do to protect the laboratory.
We have a number of alarms in the laboratory that will actually call
you in the middle of the night if something goes wrong, because we’re
dealing with the old system. They have the old air handlers. The monitoring
for facilities is not as precise as we would like. We have monitoring
deals in the laboratory, so if temperature gets too high, it calls
and goes through a list of people. Water appears in certain critical
areas, we have an alarm system there, and it calls. We come in and
find out what the problem is, work with facilities, and see whether
we need to shut the whole laboratory down. Is it just something they
can come and work on and fix without us shutting down?
Basically I’ve been doing that, making the transitions through
the programs. Just trying to think ahead and play the devil’s
advocate for the team so that I’m a little bit removed from
the testing, but not far enough so I can’t ask the questions
to keep them in a good direction so we accomplish what we want to
accomplish. Kind of what I’ve been doing.
Ross-Nazzal: It’s good information
to have. Would you tell us about some of the tests that you were doing
before the Shuttle flew? What were some of the focuses of those efforts?
Robinson: One of the things that the
laboratory was doing was testing all the different communication systems
that were expected to be used on the Shuttle.
Ross-Nazzal: What were those?
Robinson: Well, you have the PM system,
phase modulation com [communication] system, the frequency modulation
system, which does mostly TV. Then you have the audio system. that’s
what they use to talk, all through the different systems. The recording
system, working with that. We were doing verification testing to verify
with the units that we had, these were what they were called, either
engineering or qual [qualification] units. They weren’t flight
[ready]. This was what the vendors had built and said, “Okay,
this is what we hope the flight unit does.” We would test it
to make sure that it would meet mission requirements, how well did
it work, if there was any problems, trying to get to it before they
finally built the flight unit, so that they could fix stuff, or would
it be a workaround with MCC [Mission Control Center].
Because the testing that we do, we do keep in mind what MCC would
have to do with it. Matter of fact, our test results are given to
people in MOD [Mission Operations Directorate] who do the mission
planning, so they can come up with contingency plans and know how
the system works. We were doing it, like I said, before the Shuttle
flew. We had a large amount of data that we had collected, even before
the Shuttle flew, so that people would have understanding how the
communication system was going to operate and what were some of the
things that we’re looking for [or] might be on the watch out
for.
Ross-Nazzal: How many people usually
worked a test before the Shuttle was flown?
Robinson: Well, back then it was a fairly
large test team. It was on the order of 15 to 20 people. Basically
because all those different systems, we had people who operated them
and were part of the test team. We had a ground station, so we had
a ground station member. We had the spacecraft, whoever was in the
spacecraft. Then we had the team. There were other different parts
where we had to generate our own data. So that guy, he was involved.
CTRA is what it’s called, command, telemetry, and recording
area. He would generate data streams which were similar to what we
expected the Shuttle to have, and that’s what we would test
with, because we were trying to get as close as possible to the real
thing.
That’s why you had a number of different people who operated
systems, but they also brought their expertise on their system to
the test. If there was an anomaly, when we started troubleshooting
they could also help. “In my area, this is going on, and this
is what I’m seeing, and this could possibly be the cause or
not the cause.” So you needed all these people as part of your
team.
Nowadays I think the typical test is probably about I’d say
15 people directly associated with the test. We have a number of people
on call because we have systems that are working, like the intercom
system; it usually works, but if you have a problem you got to call
that guy in to work it. We have little pieces that normally work;
they don’t need a lot of maintenance, but if something goes
wrong, we have to call people in.
The test team is made up of a number of different positions. I don’t
know if you want me to talk about that or not.
Ross-Nazzal: Sure, that’d be great.
Robinson: All right. I mentioned the
test director that usually was a civil servant. They’re responsible
for directing the test. This is working with what we call a test conductor,
who was a contractor, who actually did the conduction of the test;
he was the overall coordinator, because we have contractor who’s
doing the test.
One of the other important cogs was the test project engineer. This
was the person who from the get-go was working on preparing to do
the test. He would get the information from the customer: what are
the objectives, and what is it you want to get out of it. Then he’d
meet with us. We’d have a technical meeting with the internal
people, “How can we accomplish this, what else do we need to
look at for NASA’s sake?” You have a customer come, he’d
just want to make sure his stuff is working, but we had to look at
it, say, “From the NASA point of view what would we need to
test to make sure we got the whole story?” He’s just going
to be the point of contact. He follows that project, that test, from
beginning all the way to end. During the test he’s helping get
the data documented. He’s working with the test conductor and
the test director, making inputs on what we think the direction of
the test needs to be based on the test results.
Then we collect all the data, and when the test is complete, the test
project engineer is responsible for organizing it into a document,
making a summary, and helping get the document which we call a data
package, getting that published. So he stays with it from beginning
to end.
Then the test director will take the data that we took. In the early
days, the civil servant would write a report. You would get the test
result data package, and then you’d get the test result summary
from the civil servant side about the same time. That process has
been changed over a period of time due to cutbacks. We’ve had
to consolidate stuff, where usually now the test project engineer
is also responsible for writing a summary to go in front of his data
package, with inputs from the NASA people, but things have changed
on how we’re able to do things.
When we were doing the stuff for Shuttle, people rotated. People would
have a test that they would follow. Then when the test was over, they
would come upstairs and write the report. Meanwhile another test was
going on with another test director and another whole test project
engineer and maybe another test conductor. All that was staggered,
so by the time you finished writing a test report, it’s time
for you to get back in.
Since then with cutbacks we don’t have that luxury. We’re
juggling a number of things, which is why we had to put some of the
stuff off on the test project engineer. Also as part of the test team
we have people who are responsible for helping take the data, and
they’re under the test project engineer, where they help formulate
the data sheets. At that time we were manually writing in the data,
because that’s all the technology we had at the time.
Depending on how much data we’re taking, we may have one or
two data takers, as they were called, that would record the data.
Then we had what we called area engineers who were supporting; their
systems were part of the test. We needed the ground station, so we
had the ground station area engineer [who] was part of the test. The
spacecraft operator was part of the test, we called them area engineers.
They’re responsible for maintaining their area, knowing about
their systems. They were like pseudo subsystem managers, that’s
what they brought to the test. They would have that input on what
they were seeing with their equipment. If we had anomalies, say, “Hey
I noticed this was happening, you might want to investigate this.”
They were basically part of the team.
Ross-Nazzal: How long did a test normally
take? Can you give an example? Or did they always vary?
Robinson: What we were doing for Shuttle
at that time (back in ’78, ’79, ’80, ’81)
was we were doing what we called verification. We were trying to verify
all the com systems. We had to break it up into pieces. Just to verify
the PM, phase modulation com system, and that’s the one that
they would normally use for telemetry coming down and sending commands
up. That was like about a six-week test.
Ross-Nazzal: Wow! Really?
Robinson: Yes, because you’re
testing the whole system.
Ross-Nazzal: Was it constant like 24
hours?
Robinson: No, we were working one shift,
and in a couple instances we got in a bind for time. We had two shifts
going; we had enough people for two shifts. Each com system by itself
was about a six-week test. To go through the whole Shuttle com system
and verify it took about 24 months of constant testing. We were making
sure the ground station worked with the Shuttle, and this worked with
the Shuttle. When you’re doing a verification test of the whole
spacecraft system, that was about how long it took.
Now there were some other tests that came later once you knew how
that worked, like a payload test. That would take about ten days,
two weeks. Like Hubble would come in, and they wanted to verify that
their payload com gear was compatible with the Shuttle payload gear.
So we’d do a bunch of tests with that and verify that that worked,
that was a typical test. After you verified that the Shuttle systems
were indeed ready to support a mission, then all these payloads started
coming in.
In the early days, the payloads, like Hubble is one of the later ones,
but you had some—Solar Max [Maximum] was a satellite that somebody
had come up with. Numerous ones [flew]; I could give you a list of
them if I go back and pull them out. It’s been a little bit
long. Those are the type that were coming in. These people who were
putting stuff on board the Shuttle as a payload, they were going to
come back. They wanted to make sure that when they were out doing
whatever they were doing they could get the information back to the
Shuttle, through the Shuttle telemetry, back down to the ground, over
the MCC, and out of the MCC into the Payload Operations Control Center,
where they were sitting on their ground unit so they knew what was
happening with the payload. We tested all that in our laboratory first
before they even went over there.
(Break in audio)
Ross-Nazzal: We were talking about the
payloads.
Robinson: When Hubble came, they wanted
to make sure they could get the data off the telescope payload. They
wanted to make sure they could get data through the Shuttle, so everybody
knows it’s okay to let it go. Everything’s working right;
okay, we can let it go. That type of deal. We did quite a bit of that.
Some of that even included, in the later years, our international
partners like the Italians and the French and the Germans. They had
payloads that we were doing and that was during the early part.
Later on, some more stuff came as the program matured, but the payload
tests were typically two weeks. There were a couple of them that took
a little bit longer than that. When the TDRSS [Tracking and Data Relay]
Satellite System was coming online, we tested that during ‘82.
We had the ground station, and we had built up a simulator that simulated
the satellite. Then we were playing our Shuttle data rates through
that, trying to make sure that it would work and get down to the ground,
and could it get to MCC in the correct format. We did that. When they
completed our tests, then we did actual tests talking to the real
satellite and that was a little bit more difficult. Those tests maybe
were about five- or six-week tests that we did.
Also about that time was when they brought in the Ku-band, which was
a higher frequency com system. That was a system that was going to
bring down video, was going to bring down payload data, and telemetry
data, all at the same time, so that had to be tested. They brought
that in. Once we verified that you could send three data rates down
at the same time and get them out to the right places, verified that,
then of course you got these payloads coming up, and one of the first
payloads to use that was the Spacelab, which was a German payload.
Basically what it was was a container that was like a cylinder with
openings. It was in the cargo bay. They would go through the hatch
and then go into this cylinder which had all these experiments that
they wanted to do. They were sending the data rate down at a very
high data rate at that time, that’s what they were using the
Ku-band system for. We wanted to verify that we could get the data
not only down to the ground but get it split out, so that it could
get back to Germany, where the payload people were. That was about
a six-week test.
That was interrupted by [Hurricane] Alicia, I think. Yes, that was
in ’83. Right smack in the middle of that, Alicia hit. We had
to regroup because everybody left, and then we had some damage. The
roof leaked, and we had some water that got into some of the electronic
equipment. We had to dry all that out, even got in some of the German’s
payload. Things happen. Then we had to wait and dry things out. Things
slowly came back online, and we resumed testing once we verified that
we were back to where we were. It’s those kind of things that
pop up.
Ross-Nazzal: Were there other challenges
that you encountered besides Mother Nature when you were running some
of these tests?
Robinson: Didn’t happen often,
but you could get power glitch here on site that would shut everything
down. You’d get the air handler shut down, just one of those
things that happen. Hopefully you got to it soon enough so you didn’t
damage any equipment. Once everything came back, verify that it was
back, verify everything was working the way it was working before
it went off, before you continue testing.
So it’s those challenges that we had. Every now and then they
happen while we’re in the midst of testing. Sometimes we were
fortunate enough that it didn’t. You’re susceptible to
some of the stuff out here. If the power goes off at JSC, well, you
go off too.
Ross-Nazzal: Any equipment that you
were testing that you found, “Whoa, this is not going to work
for Space Shuttle,” when you were running tests? Or did it usually
work pretty well?
Robinson: Basically people had done
their homework and had worked in their own laboratory to make sure
they understood what was working. It wasn’t a big thing. It
was the little bitty gotchas, and that’s what we were trying
to drive out. Because yes it worked, but was it compatible with the
equipment that it was supposed to talk to? Sometimes you found out
that there was a timing difference on how they were getting the data
in or something like that. So you had to resolve that, the interfacing.
By itself it’s not that it didn’t work, it was just that
they had to figure out a way to make it work with this, because you
can give at an interface two different people the same specification
and they come about it a different way. Of course because of the nature
of science you have tolerance where you can be plus or minus off.
One of them could be plus off and one minus off, and it don’t
work because they don’t see each other. That’s one of
the things that we were driving out. “Hey, you need to make
this a little bit closer, or the way you’re handling your data,
this guy can’t handle.” So I have to figure out how you
all are going to make it work.
In the laboratory, you’re just trying to drive out not the major
things but the little bitty gotchas that could get you, so that you
either come up with a way technically to get around it or a procedure.
That’s why we were working closely with MCC, and this is how
you do things, this is how you’re going to do this, say, “Hey,
maybe you might need to do it. Would this be viable?” Then we’d
check it out here in the laboratory beforehand, so they would know
how to approach it.
Ross-Nazzal: Did you ever run sims [simulations]
with the Mission Control Center?
Robinson: Yes, there were sims. Basically
we would act like the Shuttle, and once TDRSS came on, that was one
of the bigger things. We would transmit to TDRSS, and we still do
that today. It’s called verification and validation test. Before
each mission we work with the TDRSS network, which includes White
Sands [Ground Terminal, Las Cruces, New Mexico], Goddard [Space Flight
Center, Greenbelt, Maryland], and MCC. We would flow all the data
rates they expect to see during the mission. If there was a payload,
we would try to get the payload people back, even though we may have
tested them two years before, so they could flow their data, and it
would be just like it would be during a mission. This usually occurred
ten days before the mission. We still do that today, because it’s
hard to get time with the vehicle when it’s on the pad.
It also gives a chance to troubleshoot, make sure that everybody has
their equipment ready to support. The TDRSS network (White Sands)
they support multiple users, and they have Shuttle-unique equipment.
They only use it when the Shuttle gets ready to fly, that’s
when they turn it on. They don’t know whether it’s working
correctly or not until you actually flow the data through it. That’s
what we would do. We would actually transmit with our antennas to
the real TDRSS, just like the Shuttle would. It would go through the
whole network back to MCC. They would look at the data, make sure
they could get it to the payload people, everything looked like it
was supposed to. Sometimes the ground units had a bad unit that they
would have to switch out, a bad piece of equipment, there might be
a bad cable; those were the things we were driving out. It also provided
training, because some of the people were new, and they didn’t
quite understand how the whole system worked, so it gave a little
training on that.
Ross-Nazzal: What was the involvement
of this facility for STS-1?
Robinson: Well, for STS-1 we had done
all the preliminary stuff. Then we also had done like two or three
verification and validation tests because somebody had a problem,
and we couldn’t check off that they were ready. Fortunately,
we were doing it far enough in advance that we had time to do it.
Then for the first one when we were in configuration to support, we
stayed in that configuration for the whole mission. We didn’t
do any other testing. Our configuration was locked down, as well as
MCC and everybody else, for the whole [flight], because if there was
an anomaly during the mission, then we could get on it if we were
called to right away. We were in that mode, posture, for the first
four flights.
I think we were like that for two or three afterwards. Then they decided
that we didn’t have to lock down our configuration. We had an
agreement with the program that if they had a problem during regular
working hours that we could begin troubleshooting within four hours,
and if it was at night, within eight hours. We would be out of configuration.
We’d try to stay as close as possible, but we could go off and
do other testing.
Ross-Nazzal: Currently do you do any
postflight anomalies?
Robinson: Yes. They’d have us
look at, “Well, we saw this happen, we didn’t have time
to investigate, could you see what possibly could have been the problem?”
We’ve done that. Once we found out what the problem is, we’ve
also helped troubleshoot units, I think. In a couple instances, they
thought that a piece of gear which controlled the antenna switching
had gone bad, because they were having problems with the switching
on board. So after they got it down, they sent us the actual flight
unit, and we tested the actual flight unit. In both cases when that
happened I think we found out that the unit was fine. We saved them
$1 million from sending it to the vendor for the vendor to tell them
it was fine. They went back and looked, and they found they had a
bad cable on board.
We do do post anomaly testing, when requested. During the early days
we did quite a number. We also did anomaly testing during the mission.
One of the first missions with Ku-band, the actual Ku-band equipment
would not stow. The significance of that is that if the Ku-band system
cannot be stowed, you have one of two things to do. You would have
to jettison it and let it go floating off in space, because the payload
bay doors need to be closed for you to come home. We helped, along
with some other organizations, come up with a maintenance procedure
on how they could actuate the stowing operation manually. This involved
two astronauts having to do things, basically putting voltage on a
signal inside the Shuttle and having an EVA [Extravehicular Activity]
astronaut outside too.
That maintenance procedure was developed here in the laboratory with
people from other branches and divisions, and it tested what it would
take for them to do that. Came up with a procedure. They worked the
procedure here and verified that we could send the data, that it worked
with our Ku-band equipment, and that everything would work, before
they passed the information on to the astronauts. They verified the
in-flight maintenance procedure here before they sent it up. Because
that has happened, I think each flight sends astronauts over here—mission
specialists—for a class, so they can see in three dimensions;
it’s in their books, but they can see in three dimensions what
they can touch and what they can’t touch, because it still is
a $50 million piece of gear.
You don’t want to grab the wrong thing and something breaks
off. They come over here, so they see it in three dimensions, along
with their trainers. So that if it were to occur, then at least they’ve
seen what they would have to do and how the stow pins would engage
to make it work. Before each mission they might come two or three
months before the mission or six months before. At least they come
over to see as part of their training, because we have the actual
antenna and electronics here in the laboratory for them to come and
see. Otherwise, the first time they’d see it would be when they’re
out there looking at it.
Ross-Nazzal: Probably not the best time.
Robinson: Those units are, like I said,
$50 million. Basically they’re irreplaceable because they don’t
make them anymore. To jettison them would be a big loss.
Ross-Nazzal: So do you come up with
the classes? Or is that something the trainers do?
Robinson: That’s the Training
Office. They have their trainers come over to show the astronaut and
also the subsystem manager. We do not do the training. We supply the
support. They said, “Okay, we’d like to put the voltage
on the line now,” and we put the voltage on the line. We have
a test system for that so they can see how things work, but it’s
arranged through the Astronaut Office. We’re just supporting
them and the subsystem manager.
Ross-Nazzal: As the program has gotten
much more complicated over the years—we started out with those
first test flights, and then we had more and more payloads and EVAs
and things like that—how have things changed here in the facility
in terms of testing or anything else?
Robinson: Well, let’s see. For
changing, we were always trying to be two or three years ahead. We
did some modifications to the lab beforehand. One of the first modifications,
we did this I think before Shuttle flew, maybe after. I can’t
remember. We took over half of the high bay area, so we expanded the
laboratory out so that it could include shielded enclosures, because
we knew that as opposed to Apollo that with all these different things
coming in, we would need to separate some of the payloads or transmitters
from the rest of the part of the laboratory. That was one thing that
we did.
Also when Ku-band was getting ready to come online, in order for us
to test it and actually transmit to the satellite, we had to build
a temporary building called the satellite interface test area. It
was a temporary building, which is still out there now. It had a radome.
It’s in the back of the building. It was a radome. What we had
was a platform and scissor jacks to bring the antenna up into the
dome so it could actually transmit. Then in ’90-something we
added what we called the satellite interface test area addition, which
was not a temporary building, but a permanent addition to the building
to extend out, so that we had the capability to transmit to TDRSS.
We had a room for Shuttle, and because Station was getting ready to
come online, we had a radome for Station, even though we didn’t
have nothing to put in it.
The difference was that we had hydraulic jacks to lift up the platform
into the dome area, as opposed to scissor jacks getting it up there.
That was the other addition that we made to accommodate what we would
have to do for testing.
Internal to the laboratory, right before Station was to come online,
they were talking about Station. We had to move our computers out
of the laboratory into another room, because we needed that space
for a test control center. The thought process was that Shuttle was
going to be flying, Station was going to be coming online and needing
a lot of this predevelopment testing like we did for Shuttle. In order
for us to do that, we needed to have another test control center and
more people. First of all, we put in the COF [Construction of Facility]
for another test control center, got that built up, and moved the
computers into another room which was compatible for them. Matter
of fact I think Room 117, 125 was where the computers were moved,
and that was chosen because since at that time we really had a high
heat load, they wanted it to be as close as possible to where the
cooled air was coming in to keep them cool. That was some of the major
stuff that we did to keep up with the Shuttle Program, to make sure
that we were still viable.
Ross-Nazzal: So the facility is constantly
changing.
Robinson: Yes, it’s constantly
changing. We try to be flexible because we don’t know who’s
coming to the door next for testing. We’re sitting over here,
and there could be somebody here on site working on tests. Like the
first electronic digital camera was being developed by manned systems.
Took a Nikon camera and married it to a hard drive so they could take
pictures and store the image on a hard drive. This was before we had
digital cameras that you could buy in the store. They had developed
it using computer type stuff. They had it all done and said, “Okay,
we think we’re ready.” Somebody says, “Well, have
you tested it over in ESTL [Electronics Systems Test Laboratory]?”
They said, “What’s ESTL?” They didn’t know
anything about us. We didn’t know anything about them when they
came through the door.
So we wound up doing a test, which showed that what they had come
up with, they could get the images onto the hard drive, but in order
to transmit it through the TDRSS and the network to get it back, you
wouldn’t get the pictures because they were sending data in
a burst mode, where they’d send data for a while and then not
send anything. The way that the communication system is set up is
that all the pieces, again, like to see constant data. We worked with
them on figuring out a way of how to come up with—I think they
wound up putting in fill data so they’d actually make it work.
That’s how it became successful.
They did that in a short period of time, because I think they finished
that project in something like four or five months before flight,
and then they came in to do some testing, and that’s when they
found out that it was a big broke. Well, it could work with another
computer, but it wouldn’t work through the communication system,
because the computers and the radio frequency components of this weren’t
compatible. We call it bit synchronizers. They want to see data all
the time. They want to know that you care; they want the data to come
all the time, not like computer when you send data for a piece of
a second and you stay quiet for five seconds. By the time that the
equipment locked up, the still camera was finished transmitting that
particular piece. So it was constantly coming up late.
That’s when people bring stuff in. You have people who are doing
payloads and experiments, but they may not be experts at communication.
So when they come into the laboratory, that’s where they find
out that the way they envisioned things to happen was not reality,
because we had a test bed with real equipment and real stuff that
they could actually find out, “Oh, this is not going to work.”
They would have to go in and do stuff.
We had one group that flew on the [Space Shuttle] Columbia, and they
were called SPACEHAB. They could get their data through unless the
data became inverted. With RF [Radio Frequency] systems, when you
transmit data and you lock up, you can lock up to the plus side or
you can lock up to the minus side. If you lock up to the minus side,
your data winds up inverted. Most people who will realize that, they
will put in what they call an automatic polarity corrector, where
the data will see okay, the data is upside down, I’ll turn it
over and everything’s okay. They didn’t have that in there.
Whenever the data was inverted, they wouldn’t get anything at
their ground unit. When we discovered that, we told them about it.
They were getting close to the mission so they didn’t have time
to go in to redesign to put in an automatic polarity corrector.
What they did was they built up a little circuit with an LED [Light-emitting
Diode] on their ground unit so if the data was inverted, the LED would
light up, and they would flip a switch over the POCC [Payload Operations
Control Center]. They were so close to the mission, they wouldn’t
have time [to fix it]. That’s why we kept for payloads and stuff
an initial part—people coming in like 18 months to two months
ahead of time—so if we found something, they could go back and
fix it internally so there would be less changes to the way that MOD
operated in handling the mission. In the later part, they were coming
in closer and closer to mission that sometimes you had workarounds.
Ross-Nazzal: So that’s a requirement
for all the payload customers?
Robinson: No. Matter of fact, the Shuttle
Program would recommend to the payload customers that they go to ESTL
to have tests. It was not a requirement.
Ross-Nazzal: So they could be up in
space and not get any of their communications.
Robinson: They could, right. There was
one. I can’t remember which payload, but they didn’t come
through here. They violated, it wasn’t a flight rule, but the
payload system on the Orbiter couldn’t handle that data for
some reason because they didn’t have enough what you call information
in what they were sending. One of the pieces of gear, the payload
interrogator, needed to see that information. Because they didn’t
come in to test it, well, that’s one of the problems they had
on board. So they were having problems getting the data in.
There was another one. I think this was a payload that Shuttle launched
that was supposed to go off. I forget what it was, but I remember
because one of the things that happened is they were sitting on the
pad, and what happens on the pad is that before you get ready to launch,
about five or six hours, MILA [Merritt Island Launch Annex], which
is a direct link to Shuttle at the launch site, goes to high power.
When they went to high power, it turned on the payload inside the
payload bay. Nobody knew why. So they shut things down, and then they
came to us. Come to find out that the frequency that they were using
was close to the frequency that MILA was using. Shuttle has two frequencies:
a high frequency and a low. The solution was for MILA to go to the
low frequency. That’s one of the things they would have found
out if they would have come through the laboratory, but they didn’t.
So they wasted three or four days with the Shuttle on the pad that
didn’t launch because they were trying to figure out what that
was until we told them, “Oh, just go low frequency.”
It was a recommendation [to use the facility], and a lot of people
took that up, because you want to make sure it works. There were some
who because of schedule or whatever did not. Sometimes they did okay,
and sometimes they did not.
Ross-Nazzal: Did you do a lot of work
with the Russians, especially when you were working on Shuttle-Mir?
Robinson: We worked not directly with
the Russians for the Shuttle stuff. When we went to the Mir, we didn’t
work with the Russians, but we had in our division people [who] wanted
to be able to use the Shuttle audio system with the Mir. They wanted
everybody to hear. The radio the Russians used, they had a radio system,
VHF [Very High Frequency], which is like a CB [Citizens’ Band
Radio]. When you click the mike to speak, you have control of the
channel. When you talked, nobody else could talk to you, and that’s
the system that they used. Well, the Shuttle is more what they call
duplex. The astronaut could be talking, and even though protocol says
you shouldn’t, sometimes they over talk with passing back, because
you can see it, you can always see it. So they designed a box and
tested it in our laboratory which allowed the Shuttle audio system
to talk to the Mir and its VHF system. They came and tested. We made
sure you could get the audio through.
Then we did a demonstration where we had a setup for the Mir using
a VHF transceiver, and we had one in our Orbiter system. Then we had
somebody sitting in the lab acting like they were CapCom [Capsule
Communicator], and we had somebody in our EVA room acting as if they
were EVA, just having a conversation. The astronauts came over for
the demonstration, put them in there, and they started talking. They
made suggestions to the designers. “Hey, it would be much better
for us if it was like this.” The designers were able to take
that, go back, implement the requests from the astronauts, and then
put it in so it made it easier for them to talk in that situation.
That was my only interface with the Russians, I think. When they did
Apollo-Soyuz [mission], that was before Shuttle. I think I was here
for that mission, but I hadn’t been working on it. Like I said,
we had the Germans. The Italians had a satellite; they had a number
of them. We’ve dealt with them. The Japanese had a payload.
So it’s an international flavor. You can get to talk to people
from around the world without having to leave anywhere and find out
how they did things. It was good because some of the international
partners did things a little bit different than the way NASA did.
We could iron those out so everybody was aware of them beforehand.
Ross-Nazzal: You had mentioned when
working on the Shuttle-Mir for that system [having] somebody in EVA,
someone acting as CapCom. Do you have the same kind of setup as Mission
Control?
Robinson: Yes. We have the same voice
system and what have you. Basically our whole deal is fidelity so
we have MCC type of equipment in the laboratory. We have an EVA box,
and we have a Shuttle room. All we had to do was just add a room for
using the same equipment for the Mir stuff, but we have all that in
our laboratory. We have actual units that MCC uses. We may just have
one of them, whereas MCC may have ten of them, but it’s exact
flavor, exact firmware or whatever. The test results that we get,
we can say that there’s 99% [probability] that this is going
to happen. The same thing with ground station—we have a ground
station for TDRSS. We have a ground station for MILA. We worked out
with them that whenever they made a change to their hardware or firmware
that we would get that change and we would implement it in our unit.
So that even though we may operate it with different software, but
the basic premise was that it was the same ground station that you
were dealing with.
Ross-Nazzal: How else do you work with
the people at White Sands and MILA and then also at Goddard? What’s
your relationship like?
Robinson: We have some synergy with
them, in that since White Sands TDRSS Ground Terminal knows that we
have an exact copy, if they run into a problem or see something, they
can ask if we’ve seen it. Matter of fact, for the ground station,
usually we get the new copy first and run it to make sure it’s
compatible with Shuttle before they bring it out there to White Sands.
Matter of fact, on this last deal they brought in, they had an old
system. They brought in a new system set for the TDRSS Ground Terminal,
which was a new ground unit to go out at White Sands, and they had
the vendor in the building.
The vendor, when they finished that first, second copy or whatever,
I think they sent us a copy, and we started testing to see if it was
compatible. Well, one of the things that happened was that the request
for proposal, the specifications for the second unit was almost exactly
the same as for the first ground station and had not incorporated
all the changes that people had learned and put in along the way.
That’s one thing. They didn’t understand how the Shuttle
worked. There was an instance where if they were locked up to the
Shuttle, and the Shuttle for whatever reason lost the forward link,
that the ground unit station would not go and search for the frequency
because it’d expect it to show up in the same place. Whereas
with Shuttle, when they lost the forward link, they would go to what
they called auxiliary oscillator, which means that they weren’t
quite as fine-tuned as it was when they were locked up. The signal
would wind up over here, and they would looking over here and never
expand to that. When we found that, they came in, they looked at it,
and they made a change right here in the laboratory to see if it worked
before they implemented it.
Those are the type of things we would drive out with these new ground
stations to make sure they were compatible before they went and put
them in. We’d drive out that stuff. So that’s what our
whole deal is, is if you’re getting ready to do something to
your ground station, let us get a copy of it. Same thing with the
government-furnished equipment, people who are building stuff to go
into the Shuttle, like the OCAs (Orbiter communications adapter),
which is one of the things they put on to help integrate computer
type data so they can come down the RF. Come, and not only come and
test it, but leave us a copy of it so that we’ll have it as
part of our test bed. So that if later on there ever was a problem,
we’d have it there. Or for doing our verification and validation
test, we’d have the equipment so that we could actually send
MCC what they’d expect to see, without everybody having to go
try to grab equipment to bring in for that one particular time.
Ross-Nazzal: How do you work with Goddard?
Robinson: Goddard is kind of we help
you, you help us. They are responsible for the TDRSS network. Sometimes
they would make software changes. They would say, “Well, the
only way to prove that,” they need to run the Shuttle data through
it. Getting hooked to a Shuttle to do that is not feasible, so they
call us. It’s called an operational readiness test, SNORT (Space
Network Operational Readiness Test). That’s where they’re
trying to verify either a procedure or some firmware or equipment.
We act as the Shuttle.
We help them out, then when we have testing to do, like a payload,
or somebody comes in to do testing—and as a graduation exercise
they like to go through the real thing. We call Goddard up and say,
“Hey!” They’re interested, because they want to
see the data beforehand anyway. We’ll schedule a time when we
could possibly do this. They’ll come up and support us for our
test as we come up to support them to do their test. So it works like
that. There might be a letter of agreement that was done years ago,
but it’s not a big official deal that I’ve seen lately.
But it’s like that. We help you out, you help us out.
Same thing with the MCC. They’d have to be scheduled. Everybody
has interest, because if we’re testing something new, they would
like to see it as soon as possible. They know what they’ll have
to do, what addresses they would have to change, what does it look
like, how it’s going to work with their own equipment. Everybody
has a little something in the show that they would like. Usually it’s
not hard to convince them to participate. “Hey, wouldn’t
you like to see this data from this such and such and such?”
“Sure. When are you going to do that?”
Everybody wants to make sure that when it flies, they’re prepared
for it. One of the last things you want to do during a verification
and validation test is that you can’t check off and say you’re
ready to support the flight, because that causes a lot of activity
between then and the next time. Usually we might do another ver/val
like two or three days later. As you remember, the first time we usually
do one is ten days before the mission. That was a flight rule, I think.
If it’s not working then you have a few more days to work to
fix it, so you can sign off to say that you’re ready.
MCC has a signoff—it’s a document that says they’re
ready to support for each mission. I think it’s called a COFR
[Certification of Flight Readiness], that’s what they have to
sign off. So for verification and validation, they want to make sure
that their ground unit is ready to support. I don’t know if
White Sands has to sign the same thing or Goddard has to sign something.
I know MCC, before they support, have to sign a piece of paper saying
they’re ready. Once we do that verification and validation test
and everything works, everybody locks that configuration down. No
changes. You can’t put equipment. You can’t move cabling,
anything like that. You’re locked down. That shows you’re
ready to support the mission. So if anything happens, there has to
be something, like equipment failure that happens you can’t
account for. At least you can say you were ready. You have backup
systems, so they can slide a backup system in.
Ross-Nazzal: You do this for every mission?
Robinson: For every mission.
Ross-Nazzal: So you’re part of
that flight readiness review. If you’re not ready, the Shuttle
can’t launch.
Robinson: Right. We’re supplying
the Shuttle stuff. If they can’t say that they’ve [got]
all their little checkmarks, then they can’t say they’re
ready to fly, unless they get a waiver. Like I said we’re just
trying to make sure that all the equipment is ready, and that people
understand the procedures, and they said, “Okay, we think we’re
ready, everything passed.” This has been developed since oh,
1983.
When we started testing the TDRSS satellites and they took that test
procedure—testing the TDRSS satellite is a little off the subject
here—but testing the TDRSS satellite, that was a cooperation
between MCC, Goddard, ESTL, White Sands, and I forget, some others.
To come up with a test procedure that all these elements were going
to be involved in. So the test, which was a step-by-step procedure,
was developed, and it was probably about two inches thick. It took
them six months to iron it out, that was to test the TDRSS satellite.
Then what they did was use a part of that test procedure to say, “This
is what we need to do to show that we’re ready to support the
test.” That was an agreement between Goddard, MCC, White Sands,
all that. That was a cooperative effort that we were involved in.
This is what we need to do, this is how we can, can we do this.
Ross-Nazzal: You also mentioned MILA.
Is that part of Goddard?
Robinson: MILA is under Goddard. It’s
a ground unit, but it talks directly to the Shuttle. So when the Shuttle
is launching, they’re talking directly to MILA. Then when they
get up a certain way that’s high enough and without a range,
then I think MILA comes down, and they can talk to TDRSS. We do have
a MILA ground station laboratory, because before TDRSS came online,
we were using the GSTDN. That’s what they referred to it: the
Ground [Spacecraft] Tracking and Data Network, and these were ground
sites all over the world. MILA was one of them. They had something
in Australia. They had them spread out over the world.
We had to test that to make sure you could talk directly and launch
configurations, to make sure that it could talk without having false
locks. So all these units—we had one. When they would travel
around the world, on that 90-minute pass they might get 20, 25 minutes
out of each 90-minute orbit, because they’d fly over within
range for like five minutes or something like that. Once TDRSS came
online, then they were able to get more data, like the coverage is
something like 85 minutes out of 90 minutes. With our MILA ground
station, we’ve been requested oh, for the last 14, 15 missions.
When they launch from Florida, when they come around the first time,
they’re within sight of Houston. They have us act like a MILA
station.
This is before the payload bay doors open, so you cannot use Ku-band
to send video down, but they can use the S-band FM system (frequency
modulation system) to send video down. What they will do is they try
to get the astronauts and everybody to coordinate so when they come
over us that they turn on the video, send the frequency down to us.
We lock up to it and send the video and telemetry to MCC, which can
see the payload bay doors open. I think one of the things they want
to make sure is nothing goes floating out. Up until that time, they
cannot get any video through the Ku-band system. So for each first
orbit, when they come over, that’s one of the other things that
we do for real-time mission support.
That came about—I don’t remember when the question came
up, maybe it was before TDRSS came online. When they would come in
for a landing, they’ll come in for a landing at Florida. They
go through blackout, and one of the questions was how soon could you
get data once they came out of blackout, because for three or four
minutes, nobody knows anything. So there was a question, “Would
it be feasible to put a ground station here in Houston?” Since
we had the ground equipment, like MILA, we have an antenna on top
of our roof, it’s a 16-foot, as opposed to MILA having a 30-foot.
One of the things they asked us to do is to see if you couldn’t
track the Shuttle when it comes out of blackout so we’d know
how much more data we could get before we get within range of MILA.
That started our ground station activity. I think we found out that
you could get maybe about a minute, a minute and a half data before
they got within range of MILA. They were contemplating putting a ground
station here, but then TDRSS came online. Because they weren’t
transmitting through the blackout, they were transmitting backwards
up to TDRSS, they could follow it all the way to the ground. So they
decided not to put a ground station here, but they use us as a ground
station, and they also use us for troubleshooting purposes when they
have to do stuff like that.
Ross-Nazzal: I wonder if you could describe
for us the ESTL and the different rooms that you have in this facility
that support Shuttle obviously.
Robinson: Yes. For Shuttle. Well, most
of our rooms are multipurpose. So it’s not Shuttle-specific.
I think we got only two rooms that are Shuttle-specific. We have the
Orbiter communications room that holds the LRUs that the Shuttle uses
for communication.
Ross-Nazzal: What are LRUs?
Robinson: Line replaceable units, that’s
what they’re called. They’re the boxes. We have a room
that houses that so that, for all intents and purposes, that is the
Shuttle—on the communication side. That’s what we use.
That’s one of the rooms. It’s in a shielded enclosure.
We have a number of the different systems. We have the PM, FM, audio
system, the power system, all that is in that room. It’s not
an exact duplicate, but the functionality is there. We do have the
boxes, and in each box (LRU) that has a place, we can put a flight
unit in. We’ve done that a number of times, used that for testing.
It’s in a big shielded enclosure. This way we can isolate the
Orbiter.
We have another room. It’s called the test control center, and
that is where the test is usually coordinated from. That is also where
we gather the data from. The measurements of the data performance
or the RF performance is done there. So that’s where the data
is gathered and documented. They look at the data performance. They
can do video subjective quality, audio [subjective] quality. Usually
the test conductor, the test director in that room [is] coordinating
the tests from all of the different pieces. That’s the main
cog where all the data stuff takes place.
We have a MILA GSTDN room which houses the ground unit, which talks
directly to MILA as I mentioned that we use as a ground station. Basically
it’s just exact copy. During the early days it was used quite
extensively because that was the only system they had. It was used
not only to make sure that you could talk to the Shuttle, but there
were a number of things that happened when you launched that caused
problems. So they had to come up with a way to keep from what they
call locking up to a false signal, so that they weren’t processing
data. That’s because everything was fluctuating. You could lose
lock. One of the things you don’t want to do is lose lock when
you’re going up. Not that you want to lose it anytime, but definitely
not when you’re launching. We worked with the subsystem manager
to help come up with a mode that cut down on the amount of dropouts
that you would have so you could prevent some of these false locking
problems. If you drop lock, you usually lock back up, so you could
process data, and you could send commands if you had to.
We don’t use that room quite as much anymore. Actually, they
use it for the first orbit video. We use it to track satellites. Every
now and then, they might want us for troubleshooting purposes to track
the Shuttle as it comes over. Like if the TDRSS network goes down,
they’ll probably call us up, because we’re a ground station
almost in the middle of the country. So whenever they’ll be
passing over the United States, they can process data, get payload
data down, anything like that they would need to do. We’re the
emergency ground station, so that GSTDN room along with the 16-foot
S-band dish we have on top of the building. We can act as a ground
station.
Another area we have is shielded enclosure number two, which houses
our TDRSS satellite simulator. It is a unit that we built here in
house to emulate the TDRSS satellite. It’s the electrical equivalent
of a TDRSS satellite, so it can handle all that. We just don’t
have the antennas and all the other stuff that it uses, but it can
handle two S-band, two Ku-band streams, or whatever. So we can set
up a TDRSS network, because we also have a TDRSS ground station. We
have the satellite tied into that. Just like at White Sands where
the ground station talks to the satellite, and then from the satellite
it talks to Shuttle and vice versa, we have a room. Again like I mentioned
before, it’s an exact duplicate of what they have out at White
Sands. They may have six, seven, eight copies of it. We just have
one, but an exact copy of what they have.
Another room that we have is the shield enclosure number three, which
houses our EVA communication system. They could do tests for EVA.
We’d get audio from the astronaut outside and how would it get
inside. Later on they added a GFE [Government Furnished Equipment]
project called the video system that the astronauts have on their
helmet so they can show you what they’re looking at. We use
that room to help with that, because that’s part of the easy
stuff.
Then the satellite interface test area. We have a Shuttle Ku-band
deployed assembly out there, along with the Ku-band. It’s separate
from the Orbiter room, but they work in conjunction with each other.
The reason why we have the Ku-band system and the Ku-band units out
there, they have to be close to the antenna. In order to transmit,
you have to be away from the laboratory.
We have a Shuttle-unique dome. The small dome houses our Shuttle equipment,
and it’s just a Ku-band. That’s where they can send out
the video or whatever. All these deals are interconnected either by
cabling or by RF.
Then we also have the command, telemetry, and recording area, and
that is where we generate data that looks like the Shuttle data. We
can generate Shuttle commands. If we’re doing a commanding test,
we can generate it and send it to our Orbiter com gear just like they
would receive it from MCC, and they’d get a command.
We can also generate voice so that we can send voice through, and
we also do the return side. So the telemetry that you receive from
a Shuttle, it can receive that. It can separate out payload data,
so if we had a payload customer come in, we can just give them the
data they would see in the POCC.
One of the things we haven’t talked about—when we’re
testing, our testing goes not just strong signal, everything’s
supposed to work. We’ll take it and bring it and see how low
can it work, what is the threshold. It’s like how far away can
you get from your cell tower before your cell phone quits working?
We find out where that threshold is. It helps the people in the POCC,
so they know when the signal level gets low and they start getting
a lot of errors, to make sure that the ground unit doesn’t just
lock up and give up the ghost and don’t ever come back. They
have to be ready to expect. So they can verify that. We can take pieces
of data out and look at just a small piece to see what happens with
it as we’re reducing the signal level.
Some of the minor areas we have is the communication distribution
center, that’s where we have audio lines that goes to MCC. We
also have fiber lines that go to MCC. We have cabling for data that
goes to MCC; those are interfaces. The audio line goes to MCC, Houston
voice, and we can get tied into any of the channels that the Shuttle
uses, like if we’re doing an on-orbit deal, we can get tied
into the GC [Ground Controller], if necessary. So that audio system
will do that.
The fiber-optic system is what we use to send data. We can send high
rates of data over to them. So it can come out of our ground station.
If we’re doing a test in house, we can take the data out of
our ground station and send it over to MCC just like they would receive
it from White Sands. Then they can be doing a test to show okay, this
is what it’s going to look like.
We don’t use too much of the cabling now because we have the
fiber-optic. The fiber-optic system was built in ESTL by designers.
We have a number of shielded enclosures for payloads. I think we have
two other shielded enclosures that we can use for payloads. The EVA
system is usually housed in the shielded enclosure. It can also house
the payload. I think those are pertinent to Shuttle. I’m trying
to think if there’s any other areas.
We have a vault for encryption/decryption, since the Shuttle encrypts
data that they use on the forward link to send commands, that way
nobody else can send commands to the Shuttle but us.
Ross-Nazzal: Kind of important.
Robinson: Because we have the key, right.
We’ve had that vault with encryptor devices. That has been used
since Shuttle started flying. At first I think when they were doing
Air Force missions they would encrypt both the forward link and the
return link. On the NASA missions, they would just encrypt the forward
link. The return link is unencrypted. So we have that. That supports
Shuttle only. That’s a vault that we have to upkeep. We work
with the security folks so that we have the right keys to do testing.
If there’s an on-orbit anomaly or something like that that involves
forward link or whatever, that we get in contact with them, and they
bring us the proper keys to use so we can be compatible with everybody
else.
Ross-Nazzal: You mentioned the Air Force,
and I was curious. When we were flying DoD [Department of Defense]
missions, did that have any impact on your facility?
Robinson: Not so much when they were
flying. We did do preliminary testing, just like we did with Shuttle,
with the Air Force ground station. They actually brought in a ground
station to our laboratory for us to use with our Shuttle equipment
to make sure that the ground station and the Shuttle were compatible.
This was about a two- or three-month test if I remember correctly,
but that’s one of the things that we did.
One of the [other] things, part of the deal with Shuttle, they wanted
to verify—if you used encryption how did it affect your data,
how much more power would you need, what does it do to you. So those
types of testing was done with it. For the actual DoD missions, we
never did have any DoD payloads that I can remember being in here.
When the mission was up and running, we were in the dark just like
everybody else, unless there was a problem. I don’t recall any
problems on the communication side when they had DoD. There may have
been some, I just don’t remember. We were involved with it when
they were bringing the ground station in to make sure that it worked
before they put it out I think in one of their—they have a number
of sites around the world that they would use. After Challenger [STS-51L],
I don’t think we’ve had any DoD missions.
Ross-Nazzal: Can you tell us who some
of the main contractors are who work in your facility or who have
worked here over the years?
Robinson: Since ’74 up until about
five years ago one of the contractors was Lockheed Martin. Now it’s
the Jacobs Group. We’ve also had a contingent of—see,
in ’74, ’75 they were called Bendix. They went from Bendix
to AlliedSignal to Allied to Honeywell. The Lockheed Martin contractors
are responsible for conducting the tests and maintaining the spacecraft
hardware.
In the early days at that time they were Bendix. They were more of
our logistics people who kept up with the equipment that we had, inventory,
shipping stuff in and out, keeping up with configuration management,
that was their job. Somewhere along in the ’90s, that job expanded
to them taking care of our ground systems. The MILA ground system,
the 2nd TDRSS Ground Terminal ground system, and the command, telemetry,
and recording area (CTRA). So along with the other functions they
took on that, which was the ground units, because basically in the
contract world of NASA, they were under contract to maintain those
real things.
So by us having the same contractor with our ground station as they
have out at White Sands and MILA, whenever there’s a problem
we don’t have to cross contractual lines. We’ve helped
them out by shipping. If they have a piece of gear that’s broken
that they need to help support the mission, we can interchange and
send it to them for a short period of time, a long period of time.
If we have a problem with our box, they can send it. It’s fairly
simple, just NASA talking within the same contract. You don’t
have to get more people involved.
One of the most recent deals was this past mission. The White Sands
Ground Terminal had lost a box which is responsible for supplying
timing. When you’re in a laboratory, everybody needs to be on
the same time reference. Otherwise you’re going to get drifts
all over the place. Well, they were down to one standard and no backup
with the Shuttle up there. They called and asked us if we had a spare
one. We did. With a lot of quick work or whatever through upper management
on down, we were able to get out this unit for them to have as a hot
backup.
The Shuttle was coming in for a landing. If they lost the unit that
they had in there, then they wouldn’t be able to track the Shuttle.
They would lose track, and they’d lose communication, because
everything is on timing. It’s just like a GPS [Global Positioning]
System. Based on how fast you’re going, they know that three
seconds from now you’re going to be over here. So you have to
be exact. That’s what they were worried about. They’re
also still using that right now to support Station, which is a different
subject, but we had the exact same one, and we were able to send it
out to them for them to use as a hot backup.
That is not the first time that we’ve sent stuff out to them
where they’ve had failures and they needed [support]. We always
say support the mission is the first thing. We happen to also have
a backup unit that we could use. Matter of fact, we offered them our
backup unit, but it wouldn’t fit, so we had to send them our
main unit.
That’s just some of the stuff, because we got the same stuff.
Like I mentioned, if they have a problem they can ask us to investigate
it over here on ours, see if we see it. If we see something, we say,
“This is what you might want to look out for. This is what we
saw doing some testing.” Then also we’re doing testing.
We say okay, question, “Well, how does White Sands set up when
they’re flowing this stuff?” Well, we don’t know.
We call them. Get over to the guy who sets it up. Says, “Oh,
based on our procedures, this is the way.” So we can set up
exactly the way they set up their parameters. We have any questions
about the units or whatever, we can call them and ask them, because
we have the synergy. It’s the same contractor. So we don’t
have to go across contractual lines. It saves a lot of time.
Ross-Nazzal: Who else do you think we
should talk to about the building, if we were going to interview somebody
else?
Robinson: I mentioned John [E.] Ross.
That’s one guy. Let’s see. You know Harold [J.] Ferrese,
you’ve met him. Now there are some other people who might have
been around that time. They’ve retired. I don’t know how
far you all want to go, but I have some names of some people.
Tom [Thomas E.] Ohnesorge, he was here when I got here. There’s
Robert [Bobby K.] Vermillion. He’s retired. [A.] Don Travis,
I think he’s with Lockheed Martin. I don’t know if he
was here at the beginning but he was here when I got here, Oron [L.]
Schmidt. I think he’s still here. He’s close to retirement,
but I think he’s still out here. Ralph [S.] Sawyer, he’s
retired. He was the division chief when I got here. I’m not
sure whether he was here at the beginning. Mel [Melvin H.] Kapell,
he’s retired, but I think he was here at the beginning. So those
are some names of some people.
Ross-Nazzal: That’s great. It’s
a huge list. Now it also looks like maybe did you bring us a document,
that was one of our questions, if you had any documents or anything
that might be helpful for the history.
Robinson: You can have it.
Ross-Nazzal: Great.
Robinson: I have it electronically.
But this was written by Tom Ohnesorge back in ’72, [“Apollo
Experience Report – Electronic Systems Test Program Accomplishments
and Results”]. That explains some of the evolving from Apollo
to getting ready to test Shuttle, I think. He wrote it basically telling,
“This is why you need to test.” It started out talking
about an organization, [ESTP, Electronics Systems Test Program], and
then eventually I think either when they first started, ESTL was part
of that ESTA and then broke off. That’s a document that he wrote.
I know Bob Vermillion would probably have some documents because he
kept everything. I don’t know if Tom Ohnesorge did, because
he didn’t have that. I sent that to him.
The little bit of stuff I have may have to do with the people who
were here at the time. I’d have to go and dig it out. Somehow
or another when we were cleaning out, somebody ran across something,
and they had a listing of the team members or the layout of the branch
at the time. Because the branch I moved to when I came here, the branch
was dedicated to ESTL, where I think right now they have another,
they have ESTL and they also have CAIL [CEV (Crew Exploration Vehicle)
Avionics Integration Laboratory] and something else in the cubicle.
At that time, you had a group that was doing the preliminary investigation
for testing. They were going to meetings and talking with people and
say, “It might be good if you come and test this in the ESTL.”
You had people who were doing, “Well, we’re going to need
new technology and what would we need to build,” and be thinking
about how you’re going to design new stuff. Well, actually I
think that was the three major ones, but the whole branch was ESTL.
Or some part, either before, during, or after they were responsible
for at that time.
Ross-Nazzal: Well, it looks like you
made a few notes. I wanted to see if there’s anything we might
have overlooked. I think as we talked about things we covered a lot
of the questions, but if there’s anything else.
Robinson: One of the things I hadn’t
mentioned was some of the specific Shuttle missions that we supported.
One of them, after the mission, was Challenger [STS 51-L].
Ross-Nazzal: What did you do after Challenger?
Robinson: They recorded the data over
at MCC. They sent the data over to us, because over at MCC their equipment
is set up to only look at correct data, errorless data. They know
that we’re a test facility, so we can look at data with errors
in it. We were able to take the data stream, even though it had a
bunch of errors, and work with our equipment using the CTRA and audio
equipment to look at data that was bad and try to glean as much information
as possible. Over a period of time, I think we were tasked with trying
to see if we could pull out any more voice, and we did. That voice
was played in a private room. It was recorded and taken to Washington,
DC, as part of the investigation. It was also played for the families
in a private room, I think. The people who found the audio, and astronauts
and managers, who were responsible, are the only ones who have heard
that tape. I have not heard it. That’s part of the deal.
We did something, again, for Columbia [STS-107]. Our task there was
not so much just looking at the audio, but to see if we could reconstruct
a data stream. At the time that Columbia was coming in, there were
about five different entities that were looking at the return link
data. We took the five streams, and with a lot of hand and manually
putting “this is where the data ought to be based on the time,”
dadada, and constructed a data stream from the five different sources
of about 30 seconds. We played that back to MCC to say “This
is what we found, you’ll have to see if the data makes any sense,
because we had to fill in holes and make our best guess.” We
were able to reconstruct the data stream for 30 seconds, so that’s
part.
We supported all the Hubble Space Telescope missions from the very
beginning. They were acting like they were a payload. Hubble is a
different payload. It’s the only one that doesn’t meet
the standards of the payload, because the requirement for most payloads
is the data would be on what they would call a subcarrier. That is
you have your signal, then you have another frequency out here, and
your data is around here.
Well, Hubble got a waiver, so their data is around the main deal.
They were a little bit nonstandard as far as the Shuttle is concerned.
We tested that to make sure that it would work and how you would get
the data through. Afterwards—and I don’t remember how
long, whether it was a year later or six months later—because
it was supposed to go through the Ku-band system, the first initial
data. But somebody got to thinking, said, “What if we lost Ku-band,
how would we communicate?” They came up with a mode that they
had to specially wire into the Shuttle which would bypass a few boxes.
It’s called the Shuttle bypass mode, where they would directly
input the data and get into the 192-kilobit downlink stream. We tested
the proof of concept of that, that it would work, and they implemented
that.
Then for the servicing missions, they would come in, and each mission
they would start adding new things to get data, because the data rates
off of Hubble were nonstandard. They had a lot. They had like 25 or
40 different formats, which some of them were very difficult for the
common RF units on the ground to handle because they weren’t
built like that. In the first servicing mission, they found out there
were, they were called bit synchronizers, and what they’d do
is they’d shape the data. They’d take noisy data, and
they’d clean data out of it. Well, there were some specific
old bit synchronizers that worked well with the Hubble stuff, when
the new ones didn’t.
We ran a test to find out which ones worked and took the two best
ones to send out to White Sands for them to use. Then we actually
did a test with the specific payload interrogator, because it was
specific, depending on what they were using, it was so sensitive,
it depended on which box and what peculiarity. They’re all supposed
to be the same, but it’s just like your car, your Ford Mustang
might get 20 miles per gallon, and the person next door only gets
18. They’re that different. For each servicing mission, they
would come in to make sure that they could work, and whatever they
were adding to their computer would work when they went up there.
Matter of fact, on this last one I think back a couple years ago,
getting ready for STS-125, they also came back in with even more stuff.
We’ve been quite involved in Hubble. I think maybe we’re
done now.
Another one, I mentioned Spacelab. That was on [STS]-61A, the German
payload I mentioned. Another, a tethered satellite, and this was on
STS-75. This is the second time that flew. The first time they flew,
what they were supposed to do was get a satellite, extend it out,
drag it through the atmosphere, generate electricity. The first time
the gear got stuck so they couldn’t get it out far, so they
had to bring it back. The next time they flew, they did it get it
completely extended, and then the tether broke. One of the GCs over
there says, “Well, we know ESTL tested it, let’s see,
so when it comes back over, see if it’s still on.”
When it came up, we used our ground station, said, “Yes, it’s
still on.” Then they started bringing up all the GSTDN sites
around the world and brought over the payload people so they could
send commands to the tethered satellite. So this was real-time stuff.
We had to work with SAIL [Shuttle Avionics and Integration Laboratory]
to get into the proper formats so that MCC could use it. This went
on for like three or four days because that’s how long the satellite
was up and running.
They were actually coming in and manually typing in commands into
our payload command unit so that they could send the commands to whatever
ground station. If they weren’t flying over us, they would send
it out to Australia, if it was flying over them, to send the command
up and to get the data down and send it back to us. They went from
despair that the tether broke to “yes, it’s still on,”
to “look at the data we have on,” to “oh look at
all this data we got.” That was one of the real-time anomaly
stuff that we did.
Also there was one called Wake Shield, and I don’t remember
when that was, but it was a University of Houston payload in which
they were trying to grow microchip wafers in zero G to see if they
could make a better transistor. It consisted of two units. One unit
stayed in the payload bay, one was a free flier, and they communicated
to each other. When they got up there, they were sending commands,
but nothing was happening with the free flier, and they didn’t
know which one was the problem.
They called us up, and what they did was they wound up getting the
Shuttle up above the free flier so the one in the payload bay could
send commands to the free flier, at the same time when they were over
us and we were looking at it. We verified that the one in the payload
bay was active, sending commands. The free flier wasn’t listening,
and so that’s how they isolated. Then they had to bring them
back in, but one of the things that we participated in.
Back when I started there was about 50 or so contractors and about
20 civil servants supporting the laboratory so total of about 70.
In the mid ’80s they dropped down to about 45 contractors and
about seven civil servants. That’s what we said to be at full
capacity where we could test every day. We have enough people to do
the preliminary work so that we could test every day. Currently we
are at 33 contractors and about six civil servants supporting the
laboratory.
We talked about what we had to do as things changed, supporting the
DoD. We don’t reconfigure for each Shuttle mission. We don’t
do that. We do the ver/val, and then we stay as close as possible
in that configuration, but if we have another test, we’re going
on with the test. We’re listening for any problems that may
crop up. Usually we get reports that say, “Hey, this and such
and such happened, you all have any idea about it?” Say, “Well,
no, but if you want us to test it and try to recreate it, that has
to come out of the Mission Evaluation Room or a request from the flight
director.” Once that request is made, then everything is dropped,
and that’s what we’re supporting. We haven’t had
to do that too much.
Ross-Nazzal: Do you anticipate that
you’ll be running this facility through that final mission that
flies in 2010?
Robinson: Well, it depends on when the
final mission is. There’s a question about that. I’ve
been here quite a while, so I’m seeing the light at the end
of the tunnel. I was hoping that at the end of the Shuttle Program
that I would probably retire, but I don’t know. If they extend
it, I don’t know if I’m willing to stay that long. I make
a joke about, you know, everybody says, “Well, you’ve
been here that long,” I say, “Well, they kept paying me.”
Really it’s the fun part.
You felt like you were making a contribution, you were doing something
significant. Then like I said, during the heyday you were meeting
all these people. New people were coming in. Being the laboratory
manager is a little bit different in that I’m not down with
my hands on stuff, but you still feel that this is a major part of
the success of the Shuttle Program because of all the stuff. We have
found so much stuff ahead of time that if they waited, it would have
been disastrous. I think I might have a list of some of that stuff.
I don’t know if you’d be interested in that.
Ross-Nazzal: That would be great for
the nomination.
Robinson: People tend to take communication
for granted, especially now that we have cell phones and pocket TVs.
Everybody figures you turn stuff on, it works. Well, somebody had
to make sure it worked before you got to it. The Verizon commercial,
where they show all the people, that really is true; that takes that
many people. We’ve had a whole bunch of stuff that we’ve
accomplished, and we have managed to maintain a high regard with MOD
and people with the reputation of being able to provide you with actual
data. We tend to be a little bit pessimistic so that we never had
anybody complain saying, “Well, it’s working better than
you said.” Nobody’s complained about that, but it definitely
never works any worse than what we said.
There’s a sense of accomplishment that we know that we’re
providing a service to each program, to JSC, to the agency. For some
of these people, whether they realize it or not, we have a lot of
people who were forced to come and do testing, because they thought
everything was all right with their project. Then when they got in
they come to find out what we were doing and said, “Man, sure
glad [we] came,” because they got to see how the system worked.
It would have been embarrassing, you know, that type of deal. That’s
what has kept me going, and we have a low turnover. We can get into
the, it’s not quite the technical part, but more or less ESTL
is like a family. There’s a lot of deep loyalty. People feel
a sense of accomplishment. They know that we’re doing stuff.
Sometimes we’re doing exciting stuff. Matter of fact, we saw
high definition TV about 15 years ago before anybody thought about
it. The story is that they were just starting with high definition
TV. The NASA PAO [Public Affairs Office] people out of [NASA] Headquarters
[Washington, DC] wanted to help come up with standards for high definition
TV. They figured the place to test it would be here in ESTL. They
brought in a production van out of Canada. They brought in encoder
cards from the Italians, and so we had a smorgasbord of international
people working in our laboratory to try to come up with standards.
That’s where we saw they had a production van. We went out and
saw high definition TV for the very first time on a 19-inch monitor
that cost $20,000. All the engineers looked at it and said, “When
that comes out, I’m getting one of those.” It was a new
deal.
During the first part of Shuttle we were dealing with new techniques.
Eventually we weren’t doing as much research. We have a low
turnover rate both on the contractor side and on the civil servant
side. We might lose one contractor or two a year maybe, and we’ve
had some that leave and come back. Not once, twice, but maybe three
times, because of the atmosphere.
It’s always been a laboratory. It’s based on what you
can do. Nobody has a bad question, because we have a lot of complicated
stuff. Sometimes it takes somebody who’s never seen it before
says, “Why are you all doing it this way?” We stop and
we look and say, “Yes, why are we doing it this way,”
because it had become a habit. We try to [tell] people that you have
an opinion, a question, you can make a contribution.
We try to, as much as possible, have two contractors in the laboratory
along with the civil servants, which is an oddity, because we’re
down in the laboratory. We’re not sitting up in an ivory tower
waiting for people to come and tell us what the data is. We’re
down there participating and an integral part of the test team. Once
everybody comes through the door, it becomes badgeless. That has maintained
the laboratory since I have been here, and that’s something
I’ve tried to keep fostering. New people come in, their eyes
get wide, say they’ve never been to a place like this with that
type of deal.
Now the other thing that works is that our laboratory eats a lot.
People bring in doughnuts and kolatches. It’s hard to be very
disagreeable with somebody that you’re sitting out there eating
a doughnut with. We have our disagreements, but it’s usually
on the technical side, most of the stuff is technical side. Then we
argue away, and usually decisions are, “What’s your opinion,”
everybody can make an input. Then we try to make a consensus. Usually
that’s how you keep heading down the right direction. Our laboratory
is unique in that way, as far as NASA being down in the laboratory
and how we operate. I’ve heard that from a number of people
who come in from outside and know that it’s different.
It’s always been a can-do organization since I got here. It’s
always been a prideful deal that we want to provide the best service
that we can possibly. If there was any doubt about anything, we wouldn’t
publish it until we drove it to the ground to find out exactly what
we were trying to say. All our test results we present to MOD, the
customer, whoever wants to hear it. Usually wind up doing it again
two years later when it’s getting ready to fly. “You send
it too early.” Oh, we’re not worried about that. Then
when you’re getting ready to fly, everybody says, “Could
you all come and tell us again how this payload is supposed to operate,”
even though we sent them all the paperwork and what it is.
That has kept the laboratory going, by being flexible and being able
to handle a whole bunch of different things, because you never know
what’s going to come up. At one time somebody thought they were
going to use a different frequency for the EVAs. They were going to
go to Ku-band for EVA, and we tested that. They decided not to. One
time there was a guy who had a box. He was telling the NASA management
that if you use this box and ran the audio through this box from the
Shuttle he could eliminate all the fan noise that you would hear,
so it’d be a crystal clear voice. They came to our laboratory.
We stuck it in there. It didn’t work. This was the place for
them to test. They had no way of knowing.
We get all these different things. Like I said, we don’t know
who’s coming, so we have to be flexible and have to be ready
to change. That’s what we’ve done for the last 37 years
since I’ve been here. I don’t know what they were doing
before I got here, but since I was here, it’s been flexible,
a very rewarding experience in that you feel that you’re making
a contribution. Say, “Yes, that worked.” The thrill that
our guys get when they see stuff that we’ve tested during the
mission. As all over the place when there’s a launch, we have
a crowd full of people coming in, when there’s a landing, and
we have special stuff. “Oh yes, this is the part that we tested.”
The deal about the first orbit video coming over, I need only about
six people to do that. Even if it happens in the middle of the night,
you’ll wind up with 15 or 20 people in here. Then when they
put our video up on the screen for PAO, it’s like a baseball
game. Everybody screaming, “Hey, they’re using our video!”
That’s the way it’s been. That’s the human factor
with people, the sense of accomplishment. Everybody feels part of
it. Like I said, it just has been a good experience. Which makes it
hard to leave, but eventually [you have to].
Ross-Nazzal: Well, I think that’s
a good note to end on. I think that that’s a really high point.
Robinson: It’s the people. That’s
the other thing I tell people in tours. You see all this fancy equipment
and all this, it means nothing without the people behind it with the
knowledge and the dedication to find the right answer. Without the
people, you’re nowhere. That’s what has made it great.
My God, they’re so motivated, it makes my job easy, because
once they know my philosophy of how we operate, people come to them
about stuff, and they relay it back to me, and they say, “Oh
yes, we can do such and such and such, we’ll check over here
and get it done.” Work through me and stuff like that. It makes
my job easier that I’m not worried about productivity. My issues
are to try to keep upper management from interfering with the job
we’re trying to do with all the paperwork and the regulations,
which you want to start a war down there, start talking about that.
That’s how they operate. The people keep it going, because everybody
has that sense of accomplishment and pride of what they’ve done.
Ross-Nazzal: Well, I think you’re
the unsung heroes. A lot of this stuff people probably are unaware
of.
Robinson: That’s true. Besides
MCC, nobody knows we’re over here. We keep telling people. Say,
“Well, if you ever hear them refer to where we got this video
off the Clear Lake tracking station, well, that’s us.”
Not many people know that we’re here. MOD folks, some of the
older folks still remember that this is where you come and test stuff.
We’ve gotten a little bit more exposure because of money problems,
because we do consume quite a bit of money. Usually with the programs,
they want test results for cheap. You try to keep going, we have the
directorate and the division. I know we do tours all the time.
The Center Directors come through [like] General [Jefferson D.] Howell.
We’ve had Mike [Michael L.] Coats come through. This is where
they take them, to see this. The astronaut candidates, when they bring
them, tour them, they come through ESTL. During the summer, we have
the teachers come through. We have student just to show this is behind
the scenes of communications. Our running joke is there’s criticality
for boxes. Communication, its criticality three, which means it’s
not necessary for the completion of the mission until it goes out,
then it becomes criticality one, and that’s our running joke.
You don’t think about it until you don’t have it, and
when you don’t have it, everybody says, “Well, what’s
happening up there,” and nobody knows. Everybody becomes very
edgy. I’ll try to stop.
Ross-Nazzal:
It’s been very interesting. Thank you so much.
[End of
interview]