Translate this page from English...
*Machine translated pages not guaranteed for accuracy.
Click Here for our professional translations.
This is a work of the US Government and is therefore public domain and not subject to copyright. Citations from Engineering Reasoning are used with the permission of the Foundation for Critical Thinking. Forthcoming in the Engineering Management Journal, and adapted from a paper presented at ASEE, June 2008, where it won ‘Best Conference Papers.’
The Loss of the Space Shuttle Columbia:
Portaging Leadership Lessons with a Critical Thinking Model
Robert J Niewoehner, Captain, U.S. Navy, Ph.D.
Craig E. Steidle,
Abstract
Business schools have long valued case studies as a tool for both broadening a student’s perspective, and provoking them to deeper consideration of complex situations. The challenge with case studies is assuring the portability of the lessons; we don’t expect students to see situations imitating those they’ve studied, hence the goal must instead be habits of mind and principles of action which the student can portage to the circumstances of their professional lives. This paper evaluates the suitability of Richard Paul’s Critical Thinking model as a template for evaluating engineering enterprise thinking habits and organizational behavior, using the
Introduction
In 1990, as a novice test pilot, I was privileged to attend the first flight readiness review for Northrop’s YF-23. First flight is a risky event in an airplane program, and several dozen experts from across industry and government scrutinized the test team’s preparation and plans. I had thousands of hours of flying in 25 different airplanes, but amongst these grey-beards I was clearly a novice to the hazards of experimental flight test. I had nothing to contribute, and so much to learn.
While impressed with the test team’s professionalism, I was profoundly impressed by the scope and intensity of the questions posed by the gathered reviewers. Most were questions I would never have thought to pose. During a break, I occasionally asked a reviewer for the motive behind their question. Invariably I heard a story of an airplane damaged, pilot killed, or tragedy narrowly avoided. My three days in the back row provided an accelerated education in risk management. I walked away with rich lessons in the questions I should be prepared to answer as a project pilot, and questions I would later ask as a program leader.
Research in the traits distinguishing experts and novices has noted that experts ask richer questions, questions that are broader, deeper, and more complex, questions that do not balk at obstacles but ferret their way through difficulty [Bransford, 2000]. Novices do not even know what questions to ask, let alone the answers. Furthermore, novices either content themselves with simplistic answers, or suspend their inquiry in the face of complexity. The challenges for engineering leaders include, “How can we help our young engineers more quickly learn to ask more expert questions of themselves and others?” Teach them a model of Critical Thinking.
The analysis and evaluation of our thinking as engineers requires a vocabulary of thinking and reasoning. The intellect requires a voice. Richard Paul and his colleague, Linda Elder, from the Foundation for Critical Thinking, have proposed a critical thinking model documented in various sources [Paul & Elder], including over a dozen Thinkers' Guides that apply this model to diverse disciplines. Their Thinker’s Guide to Engineering Reasoning specifically adapts Paul’s model to the intellectual work of engineers, exemplifying the questions that experienced engineers ask of themselves and others [Paul, 2006].
Specifically, the authors sought to answer the following questions: “Does the Paul model of Critical Thinking provide a beneficial vocabulary and construct for evaluating complex technological case studies?” and, “Does the structure of Paul’s model enhance the portability of the lessons?”
This paper summarizes the Paul model including includes brief discussions of our approach in introducing the Paul model and vocabulary to students. Next, the findings of the
A Critical Thinking Model For Engineering
Engineers and scientists are quite comfortable working within the context of conceptual models. We employ thermodynamic models, electrical models, mathematical models, computer models or even physical models fashioned from wood or clay. Paul, Niewoehner and Elder apply a model to the way in which engineers think, an architecture whose purpose is aiding the analysis and evaluation of thought, that we might improve our thought.
The analysis and evaluation of our thinking as engineers requires a vocabulary of thinking and reasoning. The model that follows is not unique to engineering; indeed, its real power is its portability, adapting to any domain of life and thought. In so far as an engineer masters the rudimentary skills of critical thinking in the context of engineering, they have really appropriated the skills of life-long learning for whatever domain of learning their professional and personal lives lead them.
We need a definition of Critical Thinking. We are particularly fond of
“Critical Thinking is a deliberate meta-cognitive (thinking about thinking) and cognitive (thinking) act whereby a person reflects on the quality of the reasoning process simultaneously while reasoning to a conclusion. The thinker has two equally important goals: coming to a solution and improving the way she or he reasons.” [
Hence, critical thinking means much more than “Logic.” Metacognition is vital to this definition. “
Consider a modern fighter, a system of systems, each of which is overseen by some microprocessor. Those computers constantly monitor the health of each system. Vital systems, such as flight controls, have up to four duplicate processors working in parallel. The flight control computers do not simply process the next aileron deflection, they also constantly ask one another, “Do you agree? Are we all healthy?” If one disagrees, it’s “voted off.” These health management technologies have provided much of the astounding improvement in the maintainability of today’s airplanes and automobiles.
Likewise, a robust conception of critical thinking includes not only the process leading from information to a valid conclusion, it must also include the process by which we ask, in parallel, “Is my thinking healthy?” Critical Thinking simultaneously assesses its own quality. Critical thinking certainly entails logic, but it must also necessarily entail health management for our thinking.
The CAIB report provides engineering leaders with a masterpiece analysis of high technology organizational behavior. In summary comments, the board described NASA as bereft of deliberate meta-cognition.
“NASA is not functioning as a learning organization.”[Gehman, 2003, pg 127]
“[NASA mission managers] were convinced, without study, that nothing could be done about such an emergency. The intellectual curiosity and skepticism that a solid safety culture requires was almost entirely absent. Shuttle managers did not embrace safety-conscious attitudes. Instead, their attitudes were shaped and reinforced by an organization that, in this instance, was incapable of stepping back and gauging its biases. Bureaucracy and process trumped thoroughness and reason.” [Gehman, 2003, pg 181]
A bright, hard-working, dedicated team proved dysfunctional because their organizational culture did not demand that they consciously monitor the health of their own thinking. We may also be leading a bright, hard-working, dedicated, dysfunctional team if we’ve not purposefully taught them how to monitor the health of their thinking. [If our team is high performing, then it’s likely we’ve inadvertently taught them metacognition.]
Figure 1 depicts Paul’s model. The goal, at the bottom, is the mature thinker, whose thinking skills and ethical dispositions act in concert, as evidenced by intellectual traits/virtues. The Elements of Thought comprise the tools by which we analyze intellectual work, our own and others, taking it apart to understand its constituent parts. Intellectual Standards are the criteria against which we evaluate the quality of intellectual work. Specifically, the model identifies the vital questions we should be asking ourselves and others. It’s all about the questions!
Exhibit 1: Richard Paul’s Critical Thinking Model [adapted from Paul, 2006]
The engineer does not work in isolation, but in the context of enterprises, cultures and communities, each of which represents divergent interests and perspectives. Furthermore, no engineer can claim perfect objectivity; their work is unavoidably influenced by strengths and weaknesses, education, experiences, attitudes, beliefs, and self-interest. They avoid paths they associate with past mistakes and trudge down well worn paths that worked in the past. The profession engineer must cultivate personal and intellectual virtues. The leader must both model and foster these traits with those they lead.
These virtues are not radically distinct from those sought by any maturing thinker, regardless of the discipline. They determine the extent to which we think with insight and integrity, regardless of the subject. The engineering enterprise does however pose distinct questions for the engineer in pursuit of such virtues.
The intellectual traits/virtues were introduced in a Technical Leadership seminar using a workshop format. Individuals within groups of 3-4 were assigned a trait which they then studied briefly from the Engineering Reasoning Guide [Paul, 2006, pgs 6-8] and then explained to their teammates. Successful rounds of this reciprocal teaching were conducted until the list of traits was covered. Students were then asked to write down a vignette illustrating how they had personally witnessed the positive contribution of one of the traits to a team on which they’d served, and likewise one vignette exemplifying how a deficit in one trait had adversely affected a team. The entire class was then polled to nominate particularly noteworthy stories for the entire class. We’ve conducted similar workshops on this topic in several contexts. By the time they’re twenty, students have no shortage of applicable experiences from which to draw, whether athletic, academic, or extra-curricular, exemplifying virtue’s relevance. Older participants easily recall multiple stories.
All Thinking Builds Upon Eight Fundamental Elements
All thinking entails eight fundamental elements, whether it is about engineering, philosophy, cooking, sports, or business. These eight elements express eight questions that we can pose about any intellectual activity or subject. The eight elements, and their use in analyzing a document, were introduced by asking students to write out the purpose, point of view, data, etc. for the CAIB report [Paul, 2006, pgs 12-13]. These were then discussed Socratically as a class. The below summarizes/paraphrases students’ responses. Note that these questions and this activity work with any topic in any field.
Q- What was the purpose of the CAIB?
A- The CAIB sought to identify the causes of the
Q- What questions did the CAIB principally try to answer?
A- What caused the loss of
Q- What point of view did the CAIB represent?
A- The CAIB was composed of senior engineers and leaders representing the military, government, academia, and industry. The report acknowledged other points of view, including the NASA workforce and astronaut office, the U.S. Congress, the aerospace industry.
Q- What did the CAIB assume?
A- All accident investigations take for granted that all accidents have causal factors traceable to both physical and cultural factors, and that understanding those factors can lead to improved safety in future operations. Additionally, the failures of complex systems are commonly traced to the complex interaction of many cultural and technological features surrounding that system. From the outset, the CAIB assumed that the answers wouldn’t be simple. Additionally, they assumed that their recommendations would be taken seriously and would form the basis for both a return to flight and the future vitality of
Q- What information did the CAIB report?
A- The CAIB report is very expansive in the nature of the information reported. It describes the history of the Space Shuttle Program, including the varying political/budgetary climates in which it was conceived and operated over 30 years time. Additionally, it reports specific technical details of the
Q- What are the most significant concepts upon which the report rests?
A- The span of the report is very, very broad, including
Q- What did the CAIB conclude?
A- The CAIB concluded that the shuttle’s loss was directly attributable to a breech in the left wing, caused by foam shed from the external tank during the shuttle’s ascent. That breech allowed a hot jet of air into the left wing’s structure which burned through the structure, causing its failure. Tragically, the loss of foam was acknowledged by NASA as a persistent problem, but not viewed as a threat to an orbiter’s safety. Consequently, the board concluded that the accident was attributable as much to poor organizational and leadership practices, as it was to foam. “It is the view of the Columbia Accident Investigation Board that the
Q- What are the implications of the CAIB?
A- The CAIB provided a foundation for the return to shuttle service two years after the publication of their report, reestablishing
Universal intellectual standards must be applied to thinking whenever one is interested in checking the quality of reasoning about a problem, issue, or situation. To think professionally as an engineer entails having command of these standards. The standards are not unique to engineering, but are universal to all domains of thinking. They may however have particular meaning or significance which is contextual or disciplinary. While there are a number of universal standards, we focus here on some of the most significant to engineering. Unlike the elements above, this list is not necessarily comprehensive and lists found in Paul’s work do not always agree in detail.
Importantly, participants must be explicitly introduced to the notion of intellectual standards. High school and undergraduate students seem to recognize only two standards: “Did I get the right answer?” and “Am I done?” Defining intellectual standards, and helping students see that they are universal, helps them understand that good intellectual work is characterized by more than the right answer.
Clarity
Clarity is the gateway standard. If a statement is unclear, we cannot determine whether it is accurate or relevant. In fact, we cannot tell anything about it because we don’t yet know what it is saying. "Could you elaborate further on that point?" "Could you express that point in another way?" "Could you give me an illustration or example?"
Accuracy
A statement can be clear but not accurate, as in “Most creatures with a spine are over 300 pounds in weight.” "Is that really true?" "How could we check that?" "How could we find out if that is true?" "What is your confidence in that data?"
Precision
A statement can be both clear and accurate, but not precise, as in “The solution in the beaker is hot.” (We don’t know how hot it is. "Could you give me more details?" "Could you be more specific?") Engineers commonly express precision in quantitative terms associated with the calibration of our instrumentation. We can’t lose sight however that precision is also qualitative, bearing on the precision of our prose.
A statement can be clear, accurate, and precise, but not relevant to the question at issue. A technical report might mention the time of day and phase of the moon at which the test was conducted. This would be relevant if the system under test was a night vision device. It would be irrelevant if it had been a microwave oven. "How is that connected to the question?" "How does that bear on the issue?"
Depth
A statement can be clear, accurate, precise, and relevant, but superficial. For example, the statement “Radioactive waste from nuclear reactors threatens the environment,” is clear, accurate, and relevant. Nevertheless, it lacks depth because it treats an extremely complex issue superficially. (It also lacks precision.) "How does your analysis address the complexities in the question?"
Breadth
A line of reasoning may be clear, accurate, precise, relevant, and deep, but lack breadth (as in an argument from either of two conflicting theories, both consistent with available evidence). Broad thinking suggests questions such as: "Do we need to consider another point of view?" "Is there another way to look at this question?" "What would this look like from the point of view of a conflicting theory, hypothesis or conceptual scheme?"
Logical Validity
When we think, we bring a variety of thoughts together into some order. The thinking is “logical” when the conclusion follows from the supporting data or propositions. The conclusion is “illogical” when it contradicts proffered evidence, or the arguments fail to cohere." Does this really make sense?" "How does that follow from what you said?" "But before you implied this and now you are saying that, I don’t see how both can be true."
Fairness
Fairness is particularly at play where either a problem has multiple approaches (conflicting conceptual systems), or conflicting interests among stake-holders. Fairness gives all perspectives a voice, while recognizing that all perspectives may not be accurate or equally valuable.
The following three standards are not found either in Figure 1 above, nor in Paul and Elder’s writing. We have included them in our teaching because they have frequently caught our attention as defects in the work of our undergraduates.
Concision
The days are well past when great oratory meant hours, or great literature necessarily included chapter-long depictions of the field at
Suitability-
Suitability applies largely to our written and oral communications, seeking to be “fitting”, “appropriate”, or “suited to the purpose.” Suitability entails selecting right tone and presentation for the audience. It is seldom easy to craft our speech or writing to squarely address the interests, knowledge, and abilities of our audience/readers.
The
The general facts surrounding the loss of the space shuttle
Unfortunately, the board’s findings on organizational behavior have not been as broadly discussed. The technical story is fascinating; the CAIB’s discussion of organizational behavior is heart-rending. The real meat lies here for those who lead or will lead technical organizations, because it’s a tragic story of bright, devoted, hard-working professionals whose leaders allowed the team’s thinking to stray adrift, killing seven of their friends and scattering an irreplaceable national asset across the
The CAIB’s most severe criticism of NASA sprang from their observation of the strong similarity between the loss of
Surely in these grand tragedies we have the grist of poignant lessons for future leaders. Our issue as engineering educators and leaders is modeling consideration of the board’s findings in such a way that students can extract lessons about how to think about thinking in organizational contexts, rather than simply reiterating criticism of the actors’ mistakes. We want them to portage worthwhile, generalizable lessons from situation to situation, much as a canoe might be portaged from one body of water to another.
The pages that follow are extracted directly from the CAIB Report, Chapter 6, “Decision Making at NASA.” They summarize a very lengthy section 6.3, “Decision-Making During the Flight of STS-107,” which detailed the substance of multiple meetings and extensive correspondence within and between program teams as decisions were made regarding the condition of
We’ve chosen this section for emphasis because it describes the dysfunction of a specific team, involving small meetings and personal communications, rather than the report’s broader treatment of the dysfunction of an entire agency or
Summary: Mission Management Decision Making [Gehman, 2003, pp. 166-170]
Discovery and Initial Analysis of Debris Strike
In the course of examining film and video images of Columbia’s ascent, the Intercenter Photo Working Group identified, on the day after launch, a large debris strike to the leading edge of Columbia’s left wing. Alarmed at seeing so severe a hit so late in ascent, and at not having a clear view of damage the strike might have caused, Intercenter Photo Working Group members alerted senior Program managers by phone and sent a digitized clip of the strike to hundreds of NASA personnel via e-mail. These actions initiated a contingency plan that brought together an interdisciplinary group of experts from NASA, Boeing, and the United Space Alliance to analyze the strike. So concerned were Intercenter Photo Working Group personnel that on the day they discovered the debris strike, they tapped their Chair, Bob Page, to see through a request to image the left wing with Department of Defense assets in anticipation of analysts needing these images to better determine potential damage. By the Board’s count, this would be the first of three requests to secure imagery of Columbia on-orbit during the 16-day mission. [Clear recognition of the need for better data.]
Upon learning of the debris strike on Flight Day Two, the responsible system area manager from United Space Alliance and her NASA counterpart formed a team to analyze the debris strike in accordance with mission rules requiring the careful examination of any “out-of-family” event. Using film from the Intercenter Photo Working Group, Boeing systems integration analysts prepared a preliminary analysis that afternoon. (Initial estimates of debris size and speed, origin of debris, and point of impact would later prove remarkably accurate.) [Excellent initial inferences based upon scant preliminary data.]
[“Out-of-family” meant out of NASA’s experience base.]
As Flight Day Three and Four unfolded over the Martin Luther King Jr. holiday weekend, engineers began their analysis. One Boeing analyst used Crater, a mathematical prediction tool, to assess possible damage to the Thermal Protection System. Analysis predicted tile damage deeper than the actual tile depth, and penetration of the RCC coating at impact angles above 15 degrees. This suggested the potential for a burn-through during re-entry. Debris Assessment Team members judged that the actual damage would not be as severe as predicted because of the inherent conservatism in the Crater model and because, in the case of tile, Crater does not take into account the tile’s stronger and more impact-resistant “densified” layer, and in the case of RCC, the lower density of foam would preclude penetration at impact angles under 21 degrees. [Gut-based judgment replaces engineering analysis. Inaccurate inference based on invalid logic, and unsubstantiated assumptions.]
[RCC= Reinforced Carbon-Carbon, from which the wing leading edges were made.]
On Flight Day Five, impact assessment results for tile and RCC were presented at an informal meeting of the Debris Assessment Team, which was operating without direct Shuttle Program or Mission Management leadership. Mission Control’s engineering support, the Mission Evaluation Room, provided no direction for team activities other than to request the team’s results by January 24. As the problem was being worked, Shuttle managers did not formally direct the actions of or consult with Debris Assessment Team leaders about the team’s assumptions, uncertainties, progress, or interim results, an unusual circumstance given that NASA managers are normally engaged in analyzing what they view as problems. At this meeting, participants agreed that an image of the area of the wing in question was essential to refine their analysis and reduce the uncertainties in their damage assessment.
