NASA Johnson Space Center Oral History Project
Tacit Knowledge Capture Project
Edited Oral History Transcript

Rodney O. Wallace
Interviewed by Rebecca Wright
Houston, Texas – 28 May 2008

Wright: Today is May 28th, 2008. We're in Houston, Texas to speak with Rodney Wallace, who is currently Technical Advisor to the Space Shuttle Program Integration Program at United Space Alliance. This interview is being conducted for the Johnson Space Center Tacit Knowledge Capture Project for the Space Shuttle Program. Interviewer is Rebecca Wright, assisted by Jennifer Ross-Nazzal. Thanks again for finding time in your schedule to visit with us today. Tell us how you first came to work with the Space Shuttle Program.

Wallace: I started my career in 1971 with the Boeing Company in the Phase A/Phase B timeframe of the Shuttle when concepts were being developed. I worked there until Boeing lost their part of the contract; the program took a different direction in the configuration. Then I went to work for Northrop [Services] and supported the aerodynamics group at [NASA] Marshall Spaceflight Center [Huntsville, Alabama], who was working on the ascent aerodynamic part of the Shuttle Program. After a while, there was a need for a civil servant position here at JSC, and that's how I came to JSC in the Engineering Directorate, working in their ascent aerodynamics. I worked basically through STS-1 and up until [Space Shuttle] Challenger [STS 51-L accident]. I'd been out of the Shuttle effort for a little while when Challenger happened.

Then I went to the Shuttle Program Office in the Systems Integration Office to help out after Challenger, since the program was staffing up to have a bigger presence in the Program Office. I was a systems engineer there and eventually became an office manager in one of the divisions of the Systems Integration Office. I worked that until after Columbia [STS-107 accident]. During that time my office had changed titles, but it basically did the same thing. It was doing engineering integration, making sure design environments and that kind of activities were worked properly.

After Columbia, I took a short rotation back to the Engineering Directorate and worked in the Shuttle and [International Space] Station Engineering Office, which looked after the chief engineer functions for engineering. Then in February 2005 I came back to the Shuttle Program and I was the Systems Integration Office Chief Engineer until I retired in 2007. Then I came over to USA [United Space Alliance] and have been in this position ever since. So that's how I got to the program and my life in it so far.

Wright: Your roles with the program have varied some, even though your basis has been a constant. Share with us the details about some of the challenges that you've encountered along the way and some of the lessons that you've learned.

Wallace: The real challenges to me were the accidents that we had. The challenges were getting to the point of trying to understand what happened and getting past the question of, “Could I have done something different that would have prevented the accidents?” The lessons learned out of those, trying to reconstruct what happened, is that you never have enough engineering flight data to really know exactly how things went. Program is a balance of spending money in the right places, and putting flight instrumentation on vehicles is a very expensive part of the program. It has to be managed very carefully. But, in the end, when you need the data you never have enough to really do what you need to do.

Some of the other challenges were—as the Shuttle Program went along and we were going to head into the Space Station era, we partnered with Russia. So they changed the inclination that the Space Station was going to be at. The Shuttle Program had to adapt some increased performance capability, basically change the configuration, to be able to support Space Station. That was a very extensive effort going through lots of different aspects to gain enough performance to support the Station. We had to come up with the ideas and work with the Shuttle Program Manager to manage the selection of the things that would be implemented. The biggest change was changing the external tank—the second time that we changed the external tank. Made it quite a bit lighter. We changed the way we flew the vehicle in several ways.

Taking that decision process through the implementation process and then getting the analysis that needed to be done to certify the new configuration that it was ready to go fly—doing that in, if you will, a stepwise sequence—we didn't implement everything all at one time. That's a lesson learned: when you're making major changes it's better to take stepwise steps in implementation as opposed to just doing it all at once. STS-1 was a new design. Everything happened all in one flight. Best engineering judgment that we had at the time—we were ready to go, but we learned stuff when we flew the first time. When we did the performance enhancements for Space Station, taking the stepwise approach, making sure we understood the piece parts was very key.

Other challenges I've had over the time were management changes. I've worked for every Program Manager that the Shuttle Program has ever had, and when you change the Program Manager you had to adapt to the way he did business. That was usually not too big a change. In the early days the Program Managers came out of the engineering organizations and then eventually the key operations people came to be the Program Managers. So growing up in the engineering world, I had some adaptation to the new Program Managers.

However, the two accidents changed management in lots of different ways. First accident [Challenger], they just changed out the Program Manager and Center Director and we moved on with a person who'd already been working on the program. However, after Columbia they basically eliminated everyone that had had a significant leadership position in the Shuttle Program. Matter of fact, I was one of the few office managers that was still in the position. Then the people that they brought in—I think that was devastating to the Shuttle Program. There were people with very strong personalities that led things in a way that I didn't think was beneficial to the whole program.

Any engineer or any manager in that position been there as long a time, you have to get over the fact that you don't get selected for a key position. You either leave or you decide that your experience is needed to carry on the program and you support whoever is there. In the Systems Engineering and Integration Office there have been multiple office manager changes, and so I had to deal with that quite a bit. But in any case, you contribute your skills where you can and your experience and you support the new leadership. I think that's the key lesson learned, is you’ve got to put yourself out of the way and make sure that the whole effort is moving in a good direction. Leadership can't do it by themselves.

Wright: You were mentioning step-by-step sequences. I'm thinking that it takes a great deal of planning. Could you share with us some of the ways that you were able to instill good planning, and some of the recommendations that you could pass on to someone else? What are important steps in making sure that the planning is well done?

Wallace: In any endeavor of a major configuration change, it's all in the planning. Task planning saves time, saves money. It also reduces technical risk, because if you're not changing everything all at once, then you get to understand the effect of each. All the flight design changes we made in a package, so we could understand how the flight design was going to change, how much we were really going to get the improvement that we needed. The super lightweight tank was put in at one time. So we stepped up to the full configuration of changes that we were going to make.

We increased the performance capability by about 15 to 17 thousand pounds of payload to orbit more than what we were going to be able to carry to the 51.6 inclination. In the manifesting of all the missions you still had missions that you had to carry out, but you wanted to step up to these things as you could. So you had to plan what you were going to be able to fly to meet the performance capability that you had. Leading up to the fact where you've tested the whole thing, before you take on a major Space Station payload increase.

Wright: I imagine along with the planning you have budget concerns. Can you share with us some things about how to make a program efficient when you've got so many elements to consider?

Wallace: Each office in the Shuttle Program has its own budget, and you have to plan for that every year and plan your task accordingly with your contractors to be able to spend the money efficiently. Planning all your tasks and getting your ground rules for the task agreed to beforehand such that you don't have to rework things. Rework costs money. So you set aside reserve budgets to be able to deal with surprises along the way.

It's all in the planning and agreements on ground rules, and making sure that not just your office management has bought into a task, but that it fits in with overall Shuttle Program's approach to budgeting. We had to be careful, because in the systems integration world you could do something that would make the elements of the program have to go do additional work, too. You could affect your own budget but you could affect others' budgets, too. That was something you always had to be aware of.

Wright: You used the word, and it's one that we'd like to talk a little bit more about, and that's risk—how every aspect has some underlying risk. Talk to us about risk mitigation and risk management and how you believe that it needs to be instilled in every faction.

Wallace: Flying spaceships is a risky business, and there's always risk in every part that we do. What we've learned out of the Challenger accident and the Columbia accident is we did not know as much about the vehicle as we thought we did. The hazard reports, the fault tree evaluations that occurred after Columbia was a significant improvement in risk identification, and gave a tool to the Program Managers to be able to make decisions that could reduce risk. You have to make some kind of change to really reduce risk.

Understanding the risk, or quantifying the probability of the risk, allows you to make better decisions, but it doesn't change the risk. With the external tank and the foam that came off that damaged Columbia—since that accident there's been multiple changes in the design of the foam pieces on the external tank. We have learned what the failure mechanisms are of the foam and how to mitigate those.

It all comes down to really understanding the risk and then making wise decisions about design changes that allows you to actually mitigate it. The risk management systems that are in place today—the SIRMA [Shuttle Integrated Risk Management Assessment] and all of the PRA [Probabilistic Risk Assessment] work that's going on—overall system PRA plus the debris PRA work that is going on—keeps the risk in front of the Program Managers constantly, and I think that is the best that has come out of the Columbia accident.

Wright: Can you give us an example of a risk mitigation activity or management activity that definitely impacted the Shuttle Program?

Wallace: The performance enhancement program that supported the Station Program was an excellent management program—being able to take an already operating system that would not support the performance requirement of the Space Station. We needed to figure out how to change it. There was a very systematic approach taken of coming up with ideas: assessing the performance gain, how much it would cost, what is the technical risk involved of being able to really pull it off. All those assessments were done and a series of items were selected to be implemented, which ultimately provided the lift capability that Station needed.

Through that management activity, also, there was a complete recertification of the vehicle. Bringing all the elements of the program, going through their recertification papers and doing the structured reviews, the preliminary design reviews, the critical design reviews—design certification reviews all the way up through the program, even to the Agency level, was a very excellent activity. That would be, I think, one of the finest examples we've had, other than building the Shuttle the first time.

The other thing in a risk standpoint—understanding the failure mechanisms and the vulnerability that caused the Columbia accident has got to be the most risk mitigation that's been done. Although there were signs that we had problems all along and should have been noticed and should have been acted upon differently—even by myself and a lot of others. What we've done since the accident, understanding how—if you had a piece of it [foam], it is very light material, and intuition would tell you that it couldn't possibly cause any problems, but it does. Learning the failure mechanisms that makes it come off, figuring out the design changes that would help it stay on better—because it's always going to come off—we're trying to limit the size it comes off.

That was one thing that surprised me, being on the ascent side most all of my career, never really studied the Orbiter that much in terms of the tile and how they actually certified the tile and the RCC [Reinforced Carbon-Carbon] material for impacts. Basically they didn't. The requirement really wasn't there. It was just: “Don't generate debris.” That's easy said, but as we found out—and as we knew for a long time—that was a very difficult, if not impossible, thing to do.

What we've learned is the vulnerability of the tiles and being able to model the vulnerability of the tiles and RCC. And then we look at the debris sources that can come off of all the elements, and then we've stepped up to the analytical analysis of being able to predict the trajectory if something comes off and where it's going to hit and how much energy it's going to have. By doing a lot of impact testing on the elements side, they've been able to determine models of what the damage would be for a certain kind of impact.

Now, with those models and knowledge in place, then we have a better way of characterizing risk. We know that in certain phases of flight we just cannot lose something from a particular place, and that's the place we have to concentrate on to make a design change to reduce the risk. I would say we've beefed up our risk mitigation approaches and management processes significantly because of that accident.

Wright: Speaking of management processes, as you mentioned, you've worked for every Shuttle Program Manager. You've worked for Boeing, USA, and as a civil servant. So you've seen a number of management styles and methods. Give us your thoughts on those that have worked well and those that maybe have not worked as well as they should have.

Wallace: If a manager comes in with the idea that he knows everything, he's doomed to failure. He has to figure out that the Shuttle Program, or any endeavor as large as the Shuttle Program, is only as good as the weakest person in it. He has to figure out who are the knowledgeable people and listen to them, and not come in to be dictatorial and say, “This is how everything's got to be.” The most successful Program Managers were the ones that listened well, asked hard questions, expected answers that were supported by data. One thing that the Program Manager has to do is provide enough money to allow testing, statistically significant testing, to provide a good answer. One test does not always provide a real answer.

The Program Manager who has grown up either in the engineering side or the ops [operations] side has a better chance of being effective than a person that is just brought in to take the place of somebody who's moved on to another activity. Bob [Robert F.] Thompson, the first Shuttle Program Manager, was very good. He obviously had had lots of experience prior to becoming the Shuttle Program Manager. But since he was the first one he had a lot to contribute from his past.

Now you take [N.] Wayne Hale [Jr.]—Wayne Hale grew up operating the vehicle and learning about it, and I thought Wayne was an excellent Program Manager. He had a more philosophical approach to things. He wanted you to come show him data. Wayne could be hard at times when he didn't like what was being told to him, because ops guys want to make things work. He didn't want to know why you can't do something; he wants to know why you can. During the Columbia recovery and Return to Flight activities, when he was the manager we wanted to do things quickly and effectively. But sometimes when things took longer than he thought they needed to take, he still had to back up. At times he would have to back off. When a manager knows when he's reached his limit and knows that he's got to step back and get back in the rational world of listening to folks—that's a good trait for a Program Manager.

Wright: Listening, asking hard questions, and looking for answers that have data that supports it. I think those are three really good aspects for that. Following that line of thinking, if you were tasked to train the next group of leaders coming in, what are some of the elements or attributes that you would be looking for in those people? What would be some of the lessons that you would like for them to learn from you or from others that you've gathered so that they can be the leaders that the Agency needs next?

Wallace: It's not just classroom training that needs to take place—whether degrees or through the management training that happens, the leadership development. I think it's experience that is the key. So for the next group of the leaders—for, say, Constellation—I would suggest that they get testing experience and hardware experience. If you're going to manage a program that's going to be all made up of hardware and analytical activities and they all need to be validated—and testing is how you validate the data that's going to be brought to you—I think they need to have a broad experience in technical area management. Whether that's a single discipline or whether they move around and do multiple discipline management activities, particularly if they get into the systems engineering integration world, they have the bigger picture. Which a Program Manager has to have; he has to have the complete picture. Any steps that you can get for a large program, if you can work in multiple sections of the program where you can get a better perspective of the overall activity that you will ultimately lead, is I think very key.

A Program Manager is going to have to know how to deal with requirements. Some have done well with requirements. Others didn't quite know why requirements had to be there. But the requirements are how you structure the program. It tells everybody that's building something or analyzing something what they have to meet. I think that is very key. So you have to be able to manage requirements.

Somebody wants to change a requirement, you have to make sure they're changing in an appropriate way. Mr. Hale did not like waiving requirements, which is saying, “Well, for whatever rationale we can come up, we really don't need to meet that requirement for right now.” I agree with him, I don't think waivers are the right thing to do. If we have a problem we ought to make a change either in the requirement or change the hardware in some way. I think that's something the Program Manager has to deal with.

In the Constellation side, the program management are going to go through the design certification process. They're in the throes of that—the requirements reviews, the preliminary design reviews, the critical design reviews and design certification reviews. The program management needs to be trained in that—how to pull that kind of review off and what the purposes are. That's how you ensure that the vehicle is ready to go fly. I think that is a good training activity for a new leader.

The leaders have got to start from the first day they go on the job—on their path to the leadership—they have to figure out who are the smart folks, observe what they do—not just what they do right, observe what they do wrong. The best experience I ever got was watching folks that really knew how to make this business work. I don't necessarily think we've always picked leaders based on their on-the-job training. In on-the-job training, I think you learn so much more than just what you learn out of a book. I think to me that's key. I would try to structure a training activity or an experience activity for whoever the Agency picks to get on the leadership role.

Wright: Being on the contractor side currently, can you share with us some of the differences in decision-making and/or requirements and recommendations, now that you've seen both sides—a couple times you've seen both sides.

Wallace: I've seen both sides; I've seen multiple roles on the contractor side. USA—my role is basically to apply my experience and give recommendations to my management. But on the contractors as a whole, they're tasked to do specific work to get the vehicle ready to go fly and execute the flight. Their decision process is a little different. We get paid based on our performance, so you have to keep in mind what the customer needs and wants, and you try your best to make recommendations to the customer that would be beneficial for the whole program that are sometimes taken or not taken. The leadership is on the NASA side and the execution is on our side. It is different, but you're after the same objective.

There's also the support contractor role. I worked for Northrop in Huntsville supporting the Marshall Spaceflight Center. Support contractor is a different role. You really are following the lead of your NASA counterparts and helping them get the work done that they need to get done. That's the same as at JSC with Jacobs Engineering. They're following the leadership of the NASA folks and helping them do the tasks that need to get done, and then identifying tasks if something is not getting done. That's part of the job. Hopefully you get funded to do those tasks that you feel like need to get done. But the contractor world—it's a lot to do with funding and whether you get to do that.

The thing here at USA—which is complex, makes me think twice—is the way NASA has structured Constellation. You have Shuttle money and you have Constellation money. And if Constellation would like to have some particular Shuttle experience they're having very hard times getting the mechanisms that allow that to happen, and I find that troubling, because the new program needs all the experience that they can get.

Wright: Could you expand on that a little bit? Because we're looking for recommendations and suggestions for new management processes or new methodology.

Wallace: That is a key one. There needs to be a process developed that will allow Shuttle experience on the civil servant and the contractor side to be utilized to assist Constellation during this phase. NASA has taken an approach to contracting and doing a lot of the work that was normally contracted out, doing it in-house at multiple Centers. That's fine, you're covering highly skilled civil servants. But a lot of them don't have the experience in the hardware that's needed. Dealing with the multiple programs—you got three programs—hadn't noticed that that was as big a problem for Station and Shuttle. But for Constellation right now it appears to be a significant problem.

Wright: You were just talking about programs, but you mentioned you worked at Marshall and you've worked here at JSC. Are there quite a number of differences on how management and planning, the basic aspects we've been talking about, between how one Center does compared to the other?

Wallace: Yes. Marshall and JSC operate in, I would say, significantly different approaches. Over the years it's always been a battle between the two Centers to see who was going to be the lead Center. JSC, in one management system, was the lead Center for Shuttle. So Marshall had to answer our Center Director. But in the management Marshall is more process-oriented. They structure things in a particular way, and once they set up their structure that's how they function. JSC is more program-oriented. Programs lead the way and everybody else tries to get in line and get their piece of that pie. Marshall appeared to be more structured about how they included their technical expertise. They would have project offices and then also have a chief engineer's office, which was not part of the project. It was in support of the project from their engineering organization. They just functioned a little differently. But they build a lot of stuff, manage a lot of hardware. That's a capability that the Agency needs.

Wright: What would you consider to be the hardest or maybe the best lesson that you've learned through your years of experience?

Wallace: The best lesson I've learned is that systems engineering and integration, is key to the success of any program. The project elements of a program have their piece of hardware to go build, and they can get that built and delivered, and then it might not necessarily function with the rest of the elements like everybody thought it would. Having the people that are systems guys that can look across elements and make sure that things are going to work together, [that] can bring up problems that can be solved before they're problems, I think is key.

Through the Shuttle Program, Bob Thompson realized that right up front. He had Owen [G.] Morris, who was the systems integration manager, and he had Max [Dr. Maxime A.] Faget, and he had those two guys beat against each other till they came up with good solutions. Other Program Managers didn't feel the need to have that organization to be as strong as it had been over the years, and I think that led to some misjudgments.

Wright: What advice would you share with someone getting ready to enter the space program or even a similar program?

Wallace: Be well educated. Apply your expertise and look for other opportunities. I think to advance you have to have the big picture perspective and you need to have multiple opportunities. You can't just stay in the one spot. I know lots of guys over there that have been in one division, one branch their whole careers. While they're outstanding at what they do, it doesn't always lead to opportunities, and it doesn't always lead to providing the biggest impact for the overall program.

I think there's been lots of smart people that stayed in one place that if they had broadened their horizons would have made better leaders than the leaders that we chose. But they didn't choose to have that experience. So I would suggest that a new person coming into a program get the broadest range of experiences that you can get. I think you'll be happier over your career by not doing the same thing all the time. If you're good and it's recognized and you advance, then it's even more.

Wright: I'm going to let that lead into my last question, which is: what would you recommend to improve management performance?

Wallace: I have, probably, a little different idea about management than some people. A manager is only as good as the people that you have working for you. My first rule of management is to take care of the people that work for me. Know that the person that works for you is not the same every day. It's not just his expertise or his technical know-how; it's everything else that's going on in his life. That can have just as big effect on the overall team as anything. So you have to encourage them to take care of their family and themselves first and then the work will catch up. Because you can't have people working for you that are divided. When they're at work, they need to be at work. If they feel that you're supportive, then they're going to provide their best efforts.

Other part of management is making sure that planning is done first. You can't spend your money wisely on a plan that you're willing to say is not the plan. You save schedule in the long run by doing advance planning and allowing the work to be done one time and not having to do multiple reruns of the same activity because somebody didn't think of the right things. If you have a broad-based experienced team that reviews the plan before it starts to execute, I think you save time and money in the long run.

Wright: Sounds good. I know you have made some notes. I’m going to give you an opportunity to look through those and make sure that we didn't miss anything.

Wallace: As you grow in your experience and knowledge and have data at hand, don't be afraid to tell your leader that he's not correct. If your leader is dissatisfied with that, if you have data to support it, don't be shy about going above your leader. You have to have data; don't just do it because you don't like what the leader said or did. You have to have data and rationale for why it was wrong or why something else needs to be done. It's all based on data and sometimes that's hard to get. A gut feeling, intuition is never as good as data. Fight for data, even though you have a gut feeling there's something wrong. Management can put you down for your gut feeling, but they can't put you down for requesting data. So I think that's key.

I did talk about the fault tree and the hazard work. I thought it was the biggest improvement in the risk management activities. I think we did fairly well with the performance enhancements for Space Station. But developing a process that allows you to assess the—I call it the “bang for the buck.” You're getting the most for the money that you're going to spend. If there was a formal process where all decisions were run through—that required money—I think that would be an improvement.

Wright: Are there some elements to that process that you feel would be important?

Wallace: It's assessing the cost of what it is, it's the improvement that's going to be realized, it's the technical risk of actually realizing the improvement, and then if it's a risk reduction activity—how much risk does it really buy you. If right now my risk is 1 in 100 and I can make a design change that costs little and I can get to 1 in 1 million—that kind of assessment is what you want to be able to do. As far as someone that's coming into the program, you have to get to know your leaders at all levels. They need to get to know you. Once they know you, that's how they get to trust you. I think having your leader trust you to do your jobs correctly and provide good advice is key.

Wright: Is there a way that you have seen that works when people start to instill trust in others and vice versa?

Wallace: The people that I worked with—if I trusted them, I think they trusted me more. If they weren't quite as reliable and I didn't feel like I could trust them, then I felt that they didn't trust what I was telling them quite as much. I think trust is key.

Wright: Is that something that can be taught to others? I know it can be tested.

Wallace: I don't know if that can be taught. I think it's more of a goal. You need to start as a goal to gain trust from either somebody working for you or who you're working for. Getting to know the leaders—shaking hands, whatever it is, whether you see them outside of work or whether you get exposure to them through your technical work—so they'll know who you are.

Wright: Any other thoughts you'd like to add as a closing?

Wallace: You had one question [about] who else you might want to interview. I've jotted down some names. Bert [James B.] Jackson [Jr.] works here. He was the guy that put the configuration management process in place originally for Shuttle. There's some retired people I know that are still around. I think Dick [Richard H.] Kohrs and Owen Morris are still around. I think Dick Kohrs has got an office over in Building 1 somewhere helping Constellation. I thought he was one of the best Program Managers we ever had. Owen Morris was the initial systems integration manager and I think he's still working with Dick. Others come to mind are Lambert [D.] Austin [Jr.], Don [Donald E.] Prevett, Tom [C. Thomas] Modlin [Jr.] and Jene [A.] Richart—one of the guys that used to work for me, he's been there a long time. Probably good folks to interview.

Wright: All right, sir. Well, thanks so much.

Wallace: You're quite welcome.

[End of interview]