Pseudo Critical Thinking in the Educational Establishment

Introduction

Sometimes when people think poorly, they do so out of simple ignorance. They are making mistakes, they don’t know they are making mistakes, but they would willingly correct their mistakes if they were pointed out to them. Often mistakes in thinking are quite humble. No one is apt to take them for models of how to think.

Such thinking may be quite uncritical, but is not pseudo critical thinking. Pseudo critical thinking is a form of intellectual arrogance masked in self-delusion or deception, in which thinking which is deeply flawed is not only presented as a model of excellence of thought, but is also, at the same time, sophisticated enough to take many people in. No one takes a rock to be a counterfeit diamond. It is simply other than diamond. But a zircon mimics a diamond and is easily taken for one and hence can be said to be a pseudo diamond.

There is much “sophisticated” but deeply flawed thinking which is presented as a model for thought. This is nothing new in the history of thought and knowledge.

Medieval philosophy and theology, for example, was used as a sophisticated tool to resist, quite unknowingly of course, the advance of science. When deeply flawed thinking is embedded in teaching, then the development of thought and knowledge in the student is retarded or arrested. Teachers at every level of education, for example, tell students how to think. They point out thinking which they in effect encourage students to emulate. When what they point out as a model is deeply flawed, and yet sophisticated enough to take many in, it is a form of destructive pseudo critical thinking.

Pseudo critical thinking is everywhere in the world, for everywhere there are people who take themselves to be models of good thinking and who are engaged in influencing others by their model. Sometimes they foster an approach to thinking quite explicitly — by, for example, designing a program that purports to foster critical thinking. But more often they simply implicitly propagandize for a form of flawed thinking, not aware of the thinking that they are modeling. In any case, it is a rare person, one who really does think critically, who recognizes fundamental flaws in his or her own thinking. Most people are victims of their bad thinking. They do not know how to analyze and assess thinking. Consequently, most believe that their thinking is instinctively and naturally of good quality. Most believe, in other words, that his or her own thinking is that of a fairminded person who judges persons and events in an impartial and accurate way. Often people, then, inadvertently buy into one or more kinds of pseudo critical thinking: in business, in politics, and, of course, in personal, emotional, and family life. The pseudo critical thinking that I propose to concentrate on in this chapter is pseudo critical thinking in the educational establishment.

I will use as my major illustration, the California State Department of Education’s new assessment tool for reading and writing. Its development and nature provide an illuminating example of how deeply-flawed thinking is generated and worked into the system, from the statewide to the classroom level. Of course, we must remember that there is local and statewide bureaucracy and that they exist in symbiosis, each feeding the other. And teachers themselves have learned to think the way they do in bureaucratic settings, so very often they are in effect asking for, from the system, what the system by its nature is ready to give them. It is therefore somewhat misleading to say that the flawed thinking at the statewide bureaucratic level is the cause — it is rather a cause — of flawed thinking in the classroom.

Before we proceed to our “exemplar,” I would like to set the stage for what we shall do by providing the reader in advance with one — hopefully intuitive — example of why it might be that flawed thinking is regularly generated in the educational establishment. It is important that the reader comes to see why the blunders and mistakes of the California reading and writing assessment, which I shall presently document, are not exceptions in a generally good record, but rather representative examples of a typically bad product in a system that, like many others, typically generates bad products.

Consider one way in which the educational environment invites flawed thinking. It is an environment in which many whose education may in fact have been quite narrow and flawed, (see “Research Findings,” p. 19) take themselves to be experts in one form of knowledge or another, and of course, not only in a form of knowledge per se but in the kind of thinking that has created or discovered the knowledge.

These experts — called teachers and administrators — are presumed to be qualified to tell the young not only what to think but how to think about mathematical, scientific, social, and literary questions, for example. It would be odd for someone to say, “I’m a qualified teacher but my thinking is deeply flawed.” That is to say, most people recognize that there is something incoherent about saying that one is well educated but thinks poorly. Imagine someone saying, “Jack is very well educated, but with just one minor exception; his thinking is unclear, imprecise, inaccurate, irrelevant, narrow, insignificant, and shallow. Other than that, he is well educated.” Clearly this would be absurd. Hence to believe oneself an educator is pretty much tantamount to believing oneself a critical thinker, at least in some academic domain. Chemistry teachers take themselves to be experts in sound chemical thinking. Math teachers take themselves to be experts in sound mathematical thinking, and so on.

Yet many educators have been miseducated. Many are poor reasoners. Many confuse issues and questions, are easily diverted from the relevant to the irrelevant. Many lack a comprehensive educational philosophy. Many do virtually no serious reading. Many cannot speak knowledgeably outside a narrow field. And many are not even up-to-date in their own field.

Furthermore, the educational environment dominant in the schools is not traditionally conducive to critical thinking or to the development of further learning on the part of teachers and administrators. Much of the in-service is episodic, intellectually unchallenging, and fragmented. At most schools there is very little discussion on or about serious educational issues, and when there is such discussion it is often simplistic. And that is not all. The kind of instruction that is prevalent at all levels is didactic instruction. The kind of testing that is prevalent is multiple-choice focused on recall. Most students pass their courses by relying on rote memorization. Most teachers, even college professors, passed most of their courses in the same way. (see “Research Findings,” p. 19) It is the thesis of this chapter that the models for thinking and the assessment of thinking presented in the schools are generally deeply flawed, and that the reason why this is so is systemic. I will also make recommendations at the end of this chapter as to the kind of action that is called for.

The Bureaucracy Ignores Reasoning & Intellectual Standards

 Much of the pseudo critical thinking derives from the lack of a coherent understanding of the role of reasoning and intellectual standards in disciplined thought. What do I mean by this? Consider that as soon as we set our minds to the task of figuring anything out — a poem, a book, our bank account, a problem in our personal relationships, whatever — we are engaged in the task of reasoning, and reasoning can be done well or poorly. It can be assessed. And to assess it, we need intellectual standards.

The California Department of Education English Language Arts Assessment (ELAA) committees are not clear about the role of reasoning in reading and writing, and therefore they are not clear about the role of intellectual standards in the assessment of reasoning in reading and writing. Unfortunately, when one is confused on a basic point such as this, the confusion inevitably spreads to other matters as well. And so we should not be surprised to find a variety of confusions in their work.

I will enumerate for your convenience some of the major ones just below. In the next section, I list flaws characteristic of the educational establishment in general. Each item in this second list I analyze in detail, to provide a background set of understanding in preparation for an in-depth analysis of the California reading and writing assessment.

The reason for this is simple. If one understands the general pattern of misunderstanding, then specific instances of the pattern are much easier to see. A third list of flaws follows the analysis of the test. This final list makes clear the significance and instructional implications of the flawed character of the test. The general point, running throughout, is that the ill-constructed California reading and writing assessment is not an anomaly. The mistakes it makes are painfully predictable, mistakes being made all over the country in any number of ill-designed tests, in any number of ill-conceived curricula, in any number of ill-thought-through assignments.

You shall read, then, three lists of flaws. Remember that each has a somewhat different, but related, purpose. Now, the first list.

The California Department of Education English Language Arts Assessment materials, as we shall show below, contain all of the following flaws:

• Its treatment of intellectual standards is confused and erroneous.
• It confuses recall with knowledge.
• It confuses subjective preference with reasoned judgment.
• It confuses irrational with rational persuasion.
• Its key terms are often vague.
• Some key terms are dangerously ambiguous.
• It inadvertently encourages “subjectivism.”
• Its scoring is arbitrary.
• It is both invalid and unreliable.

But before we look at the detail of these manifestations of pseudo critical thinking in the California Department of Education’s assessment materials, let’s make clearer what some of the common confusions of pseudo critical thinking amount to in the domain of educational assessment and why they occur. With these understandings in hand, it will be easier to explain what precisely is wrong with California’s reading and writing assessment. 

What Does Pseudo Critical Thinking Look Like in Educational Assessment?

The advance of knowledge has been achieved not because the mind is capable of memorizing what teachers say but because it can be disciplined to ask probing questions and pursue them in a reasoned, self-critical way. Scholars pursuing knowledge submit their thinking to rigorous discipline, just as the discipline within which they think must itself submit to the broader discipline of more encompassing intellectual standards. Each academic discipline, in other words, develops special standards in virtue of its specialized concepts, procedures, and assumptions, but each also must submit to general standards that enable it to share its knowledge with all disciplines and enable all genuine knowledge to be integrated comprehensively and tested for coherence.

All research must be put, therefore, into a form of reasoning taken seriously in a field and the reasoning must then submit to the reasoned critique of others, both within and ultimately without the field, who share not only its standards but the standards of good thinking generally. Every field must be intellectually accountable to every other field by demonstrating its commitment to clarity, precision, accuracy, relevance, consistency, depth, and coherence.

Pseudo critical thinking is revealed in educational assessment when the assessment theory or practice — or the approaches to teaching, thinking, or knowledge that follows from it — fails to take into account fundamental conditions for the pursuit or justification of knowledge. The result is the unwitting or unknowing encouragement of flawed thinking. What are some of the common ways, then, that the assessment of thinking or, indeed, any approach to the teaching of thinking might be flawed?

Here are three. These are not by any means the only ones, but they are very common, very basic, and very important.

First Basic Flaw
The Lack or Misuse of Intellectual Standards

This is one of the most common flaws. It derives from the fact that though all of us think, and think continually, we have not been educated to analyze our thinking and assess it. We don’t have explicit standards already in mind to assess our thinking.

We may then fall back on “mental process words” to talk about good thinking, words such as analyzing, identifying, classifying, and evaluating.

These are words that name some of what thinking does. We use our thinking to identify things, to classify them, to analyze them, to apply them, and to evaluate them. It is tempting, then, to think of critical thinking as merely thinking engaged in identification, classification, analysis, application, evaluation, and the like. But it is important to remember that responsible critical thinking requires intellectual standards. Hence, it is not enough to classify, one must do it well, that is, in accord with the appropriate standards and criteria. Misclassification, though a form of classification, is not an ability. The same goes for analysis, application, and evaluation.

The standards used in these examples are “relevance,” “accuracy,” “plausibility,” and “significance.” Each of these standards would, needless to say, have to be contextualized. Nevertheless — and this is the key point — there can be no critical thinking without the use of intellectual standards.

Hence, if an approach to teaching or thinking focuses on the use of mental processes without a critical application of standards to that use, and persuades many to do the same, then, it is an example of pseudo critical thinking.

There are in fact many such approaches in use in education today.

Second Basic Flaw
Misconceptions Built Into the System

Flaws occur when thinking or an approach to thinking embodies a misconception about the nature of thinking or about what makes for excellence in it. I will explain just two of the most common misconceptions. The first involves confusing reasoned judgment (which is one of the most important modes of thinking leading to the possibility of knowledge) with subjective preference (which is not a basis for attaining knowledge). The second misconception involves confusing recall (which is a lower order use of the mind) with knowledge (which requires higher order thinking). Here are the explanations in brief. See if you can follow the examples and relate them to your experience.

Reasoned Judgment Confused with Subjective Preference

Many pseudo critical thinking approaches present all judgments as falling into two exclusive and exhaustive categories: fact and opinion. Actually, the kind of judgment most important to educated people and the kind we most want to foster falls into a third, very important, and now almost totally ignored category, that of reasoned judgment. A judge in a court of law is expected to engage in reasoned judgment; that is, the judge is expected not only to render a judgment, but also to base that judgment on sound, relevant evidence and valid legal reasoning. A judge is not expected to base his judgments on his subjective preferences, on his personal opinions, as such. You might put it this way, judgment based on sound reasoning goes beyond, and is never to be equated with, fact alone or mere opinion alone.

Facts are typically used in reasoning, but good reasoning does more than state facts. Furthermore, a position that is well-reasoned is not to be described as simply “opinion.” Of course, we sometimes call the judge’s verdict an “opinion,” but we not only expect, we demand that it be based on relevant and sound reasoning.

Here’s a somewhat different way to put this same point. It is essential when thinking critically to clearly distinguish three different kinds of questions: 1) those with one right answer (factual questions fall into this category), 2) those with better or worse answers (well-reasoned or poorly reasoned answers), and 3) those with as many answers as there are different human preferences (a category in which mere opinion does rule).

Here are examples of the three types: 1) What is the boiling point of lead? 2) How can we best address the most basic and significant economic problems of the nation today? and 3) Which would you prefer, a vacation in the mountains or one at the seashore? Only the third kind of question is a matter of sheer opinion. The second kind is a matter of reasoned judgment — we can rationally evaluate answers to the question (using universal intellectual standards such as clarity, depth, consistency and so forth).

When questions that require better or worse answers are treated as matters of opinion, pseudo critical thinking occurs. Students come, then, to uncritically assume that everyone’s “opinion” is of equal value. Their capacity to appreciate the importance of intellectual standards diminishes, and we can expect to hear questions such as these: What if I don’t like these standards? Why shouldn’t I use my own standards? Don’t I have a right to my own opinion? What if I’m just an emotional person? What if I like to follow my intuition? What if I don’t believe in being “rational?” They then fail to see the difference between offering legitimate reasons and evidence in support of a view and simply asserting the view as true. The failure to teach students to recognize, value, and respect good reasoning is one of the most significant failings of education today.

Recall Confused With Knowledge

A second common confusion which leads directly to pseudo critical thinking is recall confused with knowledge. As I suggested above this confusion is deeply embedded in the minds of many “educators.” It results from the fact that most instruction involves didactic lectures and most testing relies fundamentally on recall. Educators confuse students recalling what was said in the lecture with knowing the how and the why behind what was said. For example, a teacher might give you information, some of which is true and some of which is not, and you may not know which is which. Another way to see this point is to figure out why we don't think of parrots as gaining any knowledge when they learn to repeat words. Tape recorders get no credit for knowledge either. Do you see the point?

We tend to assume, to carry the point a bit further, that all information in a textbook is correct. Some, of course, is not. We attain genuine knowledge only when the information we possess is not only correct but, additionally, we know that it is and why it is. So, strictly speaking, I don’t know that something is true or correct if I have merely found it asserted to be so in a book. I need to have a greater understanding — for example, I need to know what supports it, what makes it true — to properly be said to know it.

So if someone tells me Jack has flown to Paris for the weekend, I don’t know if he actually did. I might believe that he had (because I trust the person who told me) and my belief might even be correct (through happen stance), but still I don’t yet know for sure that he did. I am operating on the basis of “hearsay.”

The failure to appreciate the significance of this distinction causes a lot of problems in schooling because many who teach do not really know their own subjects well enough to explain clearly why this or that is so, and why this or that is not so. They know what the textbook says, certainly, but not why the textbook says what it says, or whether what it says is so or not. Having knowledge (for such confused persons) is nothing other than remembering what the textbook said. 

Third Basic Flaw
The Misuse of Intellect

“Skilled” thinking can easily be used to obfuscate rather than to clarify, to maintain a prejudice rather than to break it down, to aid in the defense of a narrow interest rather than to take into account the public good. If we teach students to think narrowly, without an adequate emphasis on the essential intellectual traits of mind (intellectual humility, intellectual honesty, fairmindedness, etc.) the result can then be the inadvertent cultivation of the manipulator, the propagandist, and the con artist. We unknowingly end up, then, undermining the basic values of education and public service, properly conceived.

It is extremely important to see that intelligence and intellect can be used for ends other than those of gaining “truth” or “insight” or “knowledge.” One can learn to be cunning rather than clever, smooth rather than clear, convincing rather than rationally persuasive, articulate rather than accurate. One can become judgmental rather than gain in judgment. One can confuse confidence with knowledge at the same time that one mistakes arrogance for self-confidence. In each of these cases a counterfeit of a highly desirable trait is developed in place of that trait.

There are many people who have learned to be skilled in merely appearing to be rational and knowledgeable when, in fact, they are not. Some of these have learned to be smooth, articulate, confident, cunning, and arrogant. They lack rational judgment, but this does not dissuade them from issuing dogmatic judgments and directives. They impress and learn to control others, quite selfishly. Unless we carefully design schooling to serve the “higher” ends of education, it can easily, as it now often does, degenerate into merely serving “lower” ends. When this happens, schooling often does more harm than good. It spreads the influence and resultant harm of pseudo critical thinking.

With the above understandings in mind, we are prepared to examine the new California Assessment Program and its evaluation of reading and writing. 

The California Assessment Program
English Language Arts Assessment

Introduction

California has developed the reputation of being a leader in educational reform. It was the first state to mandate critical thinking instruction at all educational levels.

However, it is now becoming apparent that at the K–12 level at least, the mandate is not on solid ground, for pseudo critical thinking approaches, and the misunderstandings that underlie them, are becoming rampant in the state. The jargon of reform is everywhere, but substance is virtually nowhere. Unfortunately, the California Department of Education is oblivious of the danger, in fact, is very much part of the problem. Not only is it failing to provide sound leadership in integrating critical thinking into instruction, it is developing an assessment program which is shot-through with pseudo critical thinking confusions.

It is now deeply involved in developing what it calls “authentic” assessment that focuses on student “performances” found in the student “construction of meaning” in language arts and social studies. Now all of these terms — “authentic” and “performance” and “construction of meaning” — are part of the buzz words of the day in educational circles. Of course, the theoretical insights that led to emphasis on these words are important, so let’s briefly review them.

Testing and assessment in this country has come under increasing fire, and for good reason. Much of what has traditionally been tested in the popular, machine scorable, multiple choice tests has contributed to little more than trivial pursuit, more and more emphasis in instruction on the lowest order of thinking: rote memorization.

Growing numbers of critics have pointed out that the items on which we have been testing students do not involve reasoning and have little relationship to the kinds of tasks that students will later be called upon to “perform.” The tests fail, in other words, to “authentically” test higher order “performances.” The reform of assessment has increasingly looked to an increased emphasis on “authentic” items that involve “performances” of a “higher order.”

Furthermore, research by cognitive psychologists and others have clearly established the fact that when humans deeply learn something — in contrast to, say, storing it temporarily in short-term memory — that learning involves the “construction of meaning.”

Here’s how you might look at it:  In order to get about successfully in the world in which we live we have to continually “make sense” of things, to give a meaning to what is surrounding us. As we do this we develop networks, systems of meanings that enable us quickly to size up what’s in front of us. The result is we don’t see “meaningless” colors and shapes and sounds. We see trees, and people, and dogs, and speeding cars, and smog. We immediately construct “meaning” out of our experience. Our experience is made by our minds to “fit into” meanings we have already constructed, or, if we cannot do this, we set about constructing a new meaning out of the old ones.

Now, what puzzled educational researchers was what has come to be called the problem of “transfer.” Why don’t students take what they are studying at school and use it in their daily acts of “constructing meanings?” Why don’t they use scientific concepts when they make everyday predictions or form everyday theories about people and events? Why don’t they use concepts from their social studies textbooks when they go about interpreting social situations and trying to figure out solutions to their social problems? Their conclusion was that the students don’t use what they study in school in their everyday life because they are not engaged in the construction of meaning in class. In class, they are merely, or at least mainly, memorizing, not constructing meaning, not integrating school learning with everyday life.

Now we are ready to bring the three theoretical concepts together — “authentic,” “performance,” and the “construction of meaning.” In authentic performances students construct meaning. They do not simply memorize. So why not focus school instruction on just such matters? Why not give them tasks that are “authentic?” Why not help them, in “performing” those tasks, to actually “construct meaning,” in other words, to integrate what they are learning into the network of meanings they are already using to make sense of the world. This is the basic theoretical idea behind the ELAA materials, and, as far as it goes, there can be no objection. But as one wise person once said, “The important truths are in the details.” And the details of the ELAA materials are horrendous.

A close examination of the details of the California Language Arts Assessment reveal that it is flawed in all of the following ways:

• The overall conception is not theoretically coherent. It is filled with vagueness and confusion. This is probably the result of the committee adopting the key buzz words without clearly understanding the theory underlying them. The buzz words are then used vaguely and the details are filled out with terms from the agendas of the various stake-holders.
  
• It does not provide a realistic model of reasoning, of critical reading and writing. Indeed, it is clear that the developers of the assessment do not realize that both reading and writing intrinsically involve the use of reasoning and that reasoning can be done well or poorly.
  
• The overall conception does not call attention to definite and clear intellectual standards. The criteria given are typically vague and applied inconsistently. Important intellectual standards are missing.

• There is no way that a teacher might grasp an organized and systematic approach to the role of reasoning in reading and writing by studying the materials being disseminated. The test, in fact, leads the teachers in the direction of malpractice, that is, into the systematic misassessment of reading and writing, leading the students in turn to become inaccurate, imprecise, and undisciplined readers and writers.

A Tangle of Confusions

Let’s now look at the details and shed light on some of the theoretical confusions that undermine the approach.

First of all, the ELAA commentators open by confusing knowledge with recall and “constructing meaning” with “reasonably constructing meaning.” Since these confusions are basic and lead to indiscriminate scoring, let’s look at how this occurs.

California’s New English-Language Arts Assessment: An Integrated Look, begins by announcing a paradigm shift. As the English Language Arts Assessment (ELAA) document explains, “At the heart of the framework is a paradigm shift in which ‘constructing meaning’ replaced ‘gaining knowledge’ as the primary goal.” Or, as it says later, “Since the construction of meaning is the essence of both reading and writing, the new assessment allows students to shape the outcome rather than to identify correct meanings that test makers have posited.”

It is clear that the writers of the assessment are either not clear about the difference between recall and knowledge, or they are wrongly assuming that the attainment of knowledge is not intrinsically connected to the construction of meaning, or both. Briefly, let’s make these relationships clear.

Rational and Irrational Constructions

“Constructing meaning” is a process that is common to all learning which becomes deep-seated in the mind of the learner. It applies, however, just as much to the formation of flawed, irrational meanings as it does to the formulation of defensible, rational meanings. Deep-seated irrational fears, for example, result as much from the personal construction of meaning as do insights and understandings.

Knowledge, on the other hand, though also the result of the construction of meaning, requires a clear-cut exercise of the rational faculties of the mind. For example, to appropriately judge a person accused of murder to be guilty or innocent, one’s thinking must be guided by a careful and rational use of evidence, legal criteria (the criteria for “murder”), and the canons of sound reasoning. When a jury appropriately attains the knowledge of guilt or innocence, that knowledge, expressed in their verdict, is a product of a rational, a reasoned, construction of meaning. Of course, a jury may not function as it ought. It may be irrational and prejudiced, and the judge may overturn its verdict precisely because it did not properly discharge its responsibility to be “rational.” There is nothing wrong, therefore, with focusing attention on the need of students to “construct meaning” but it must be underscored that the mere construction of meaning, as such, is not a significant achievement, since it is done as much by Archie Bunker as by Einstein.

But the authors of The California Student Assessment System are confused on this point, for they talk as if the construction of meaning is an
end in itself. They forget that “prejudice,” “stereotypes,” “misconceptions,” “illusions,” “delusion,” “self-deceptions,” “false beliefs,” and all manner of other intellectually flawed creations of the mind, are just as much “constructions” and as “meaningful” constructions as ones more insightful and discerning.

We should rather be interested in fostering in children adherence to those intellectual standards that maximize their construction of genuine “knowledge,” for otherwise they are likely to engage in a great deal of “irrational” construction and they will not know they are doing so.

Education must discriminate between the quality of students’ constructions of meaning, both in their reading and in their writing. But this can only defensibly be done by judging them by means of those intellectual standards common to educated thought. A construct that is unclear is not to be confused with one that is clear. One that is inaccurate is not to be confused with one that is accurate. One that is relevant to an assigned task is not to be confused with one that is irrelevant. One that is superficial is not to be confused with one that is deep.

Hence we do not need to decide between emphasizing the construction of meaning and the goal of attaining knowledge. If we properly understand the “dual” character of “meaning construction,” we will immediately recognize the need to focus on the “reasoned” and “reasonable” construction of meaning, and not indiscriminately credit any construction of meaning.

To underscore the point, the human mind naturally and inevitably constructs meaning. The mere fact that students construct meanings tells us nothing about the quality of those constructs. For example, in the extended example on reading as a form of thinking which we cited (pp. 24–27), both readers, Stephen and Colleen, constructed meanings. But you will remember that the meanings constructed by Colleen were absurd. Both students reasoned about the text’s meaning, but there was a stark contrast in the quality of reasoning in the two cases.

The point should now be clear. We want to work with our students’ capacity to construct meaning from a text, but we want to do this while teaching them to discipline their reading, to learn how to fit their interpretations to the logic of the words of the text. We want them to develop definite intellectual standards for their reading and not feel free to treat a text as if it were “silly putty,” to be shaped into any “meaning” they choose.

In passing, in the workplace, there is no economic value in constructing irrational, fanciful, lively, and entertaining meaning if it is irrational and reflects a flawed understanding of the text. Employers are not looking for a flashy, individualized response to a piece of writing, but rather a solid grasp of the meaning intended by the author. This is a fundamental premise of written communication! 

We should therefore continually underscore the pivotal role of intellectual standards, not only in assessment but in any form of intellectual work whatsoever, including of course, reading and writing.

A Pseudo Commitment to Intellectual Standards

Intellectual standards are essential to the appropriate assessment of reading and writing. At some level, the assessment authors are aware of this necessity. Their description of their own criteria imply both impartiality and commitment to intellectual standards. For example, the authors of the English-Language Arts Assessment often speak of their commitment to “encourage students to read widely and in depth.” (Depth implies criteria for distinguishing “deep” from “shallow” readings.) Secondly, they imply impartial assessment when they state the three-fold purpose of the new English-Language Arts Assessment (p. I-2):

They use language that implies a concern with rational judgment: for example, the ELAA report says when speaking of “meaning-making,” that,

We want students to think critically as they explore interests, clarify values, solve problems, resolve conflicts, generate new ideas, synthesize/apply learnings, set goals, and make decisions in response to the literature they read. (p. I - 2)
Now, the processes of thinking critically, clarifying values, solving problems, setting goals, and making decisions all presuppose the importance of rationality, of engaging in sound reasoning. To clarify values, for example, requires that we rationally analyze them.

It becomes clear, from here on, that the California testing experts’ glossy, global statements about critical thinking and meaning-making and standards are losing their luster.

Solving problems is not the product of an arbitrary construction of meaning, but requires, amongst other things, an objective and accurate analysis of the nature of the problems, of the information relevant to the problems, and such like. Effective goal setting requires that we accurately identify possible competing goals and reasonably assess which make most sense. And certainly, decisions can reasonably or unreasonably be arrived at. It becomes clear, from here on, that the California testing experts’ glossy, global statements about critical thinking and meaning-making and standards are losing their luster. Empty platitudes and vacuous ideals are a specialty of virtually all bureaucracies. So common are they now that they are hardly noticed anymore.

The Problem of Scoring

As you might expect, all of the confusions above come home to roost in the design for scoring student “performances” in reading and writing. For example, in explaining the design with respect to assessing elementary reading, the authors introduce us to a 15-point list under the head of the reading performances of effective readers.

Effective readers connect with, reflect on, and challenge the text. Readers do not need to show evidence of all the performances listed here. The discerning and insightful reader may display a broad spectrum of reading behaviors or may investigate a few selected behaviors in great depth. The exemplary reader may show variety, complexity, breadth, and/or depth. Through their writing and graphics, these readers show convincing evidence of their ability to construct meaning. They may:

1. Experiment with ideas; think divergently; take risks; express opinions (e.g., speculate, hypothesize, explore alternative scenarios; raise questions; make predictions; think metaphorically).
2. Explore multiple possibilities of meaning; see cultural and/or psychological nuances and complexities in the text.
3. Fill in gaps; use clues and evidence in the passage to draw conclusions; make plausible interpretations of ideas, facts, concepts, and/or arguments.
4. Recognize and deal with ambiguities in the text.
5. Revise, reshape and/or deepen early interpretations.
6. Evaluate; examine the degree of fit between the author’s ideas or information and the reader’s prior knowledge or experience.
7. Challenge the text(s) by agreeing or disagreeing, arguing, endorsing, questioning, and or wondering.
8. Demonstrate understanding of the work as a whole.
9. Show sensitivity to the structure of the text(s): how the parts work together; how characters and/or other elements of the work(s) change.
10. Show aesthetic appreciation of the text(s); see linguistic and structural complexities.
11. Allude to and/or retell specific passages(s) to validate and/or expand ideas.
12. Make connections between the text(s) and their own ideas, experiences, and knowledge.
13. Demonstrate emotional engagement with the text(s)
14. Retell, summarize, and/or paraphrase with purpose.
15. Reflect on the meaning(s) of the text(s), including larger or more universal significance; express a major understanding about or insight into a subject, an aspect of self, or of life in general.

The Escape Hatch

First off, it is clear that since each scorer can pick and chose from such a wide variety of criteria (a number of which as we shall see are extremely vague), the impartiality of application of the criteria is suspect. Also note that any and all of these activities can be done either defensibly or indefensibly. Hence, if these are to be assessed in order to be credited or discredited, criteria must be provided for each of the individual “performances” cited (over 50 are buried in the list). We need some explanation of how they expect someone assessing student reading to apply them. Consider each of the following. The assessor is left to her own intuitions in determining whether or not a student’s reading:

• is insightful; discerning; perceptive;
• is sensitive to linguistic, structural, cultural and psychological nuances and complexities;
• entertains challenging ideas; grounds meaning in acute perceptions of textual and cultural complexities.

Do the assessors really know how to impartially assess whether or not a student reader is being “sensitive to a psychological nuance” or to a “structural nuance” or a “cultural nuance?” Or can they impartially determine whether or not the student reader is entertaining a “challenging” idea? Or whether a student perception is “acute” or not? Isn’t it highly probable that different assessors are going to have somewhat different conceptions of each of these matters, for example, one thinking a given idea is “quite challenging” and another thinking it is not? Surely this much is clear! No criteria, however, are provided, and this invalidates any attempt to use the results to assess one student’s performances year after year, as well as to assess all California students collectively, year after year. Instead, having given us an array of vague descriptors, the authors now largely set them aside and focus instead on a six-point “Scoring Guide” that is to be used in distinguishing student reading into the following categories:

• Exemplary Reading Performance (Six Points)
• Discerning Reading Performance (Five Points)
• Thoughtful Reading Performance (Four Points)
• Literal Reading Performance (Three Points)
• Limited Reading Performance (Two Points)
• Minimal Reading Performance (One Point)

These general descriptors are of very little use. For example, consider the words “discerning” and ”thoughtful.” It is not obvious that one is better off being “discerning” than being “thoughtful.” It is also not obvious why “literal” is above “limited.” It is certainly not clear why the lowest score is “minimal” reading. What ever happened to just plain “poor” reading? Has it disappeared or is it one of many forms of “minimal” reading. (See Colleen’s reading on pp. 24–27.) Is it “minimal” or just plain “poor?”)

But that is not all. Each of the terms listed in each of the six point scoring guide create further problems for the conscientious scorer. Consider the terms in the first category alone, that of “exemplary reading performance.” A person who takes seriously the characterizations of this first category should be prepared to notice and assess whether or not the student is:

The State Department Criteria for an Exemplary Reading Performance

1) An exemplary reading performance is insightful, discerning and perceptive as the reader constructs and reflects on meaning in a text. Readers at this level are sensitive to linguistic, structural, cultural, and psychological nuances and complexities. They fill in gaps in a text, making plausible assumptions about unstated causes or motivations, or drawing meaning from subtle cues. They differentiate between literal and figurative meanings. They recognize real or seeming contradictions, exploring possibilities for their resolution or tolerating ambiguities. They demonstrate their understanding of the whole work as well as an awareness of how the parts work together to create the whole.

Readers achieving score point six develop connections with and among texts. They connect their understanding of the text not only to their own ideas, experience, and knowledge, but to their history as participants in a culture or larger community, often making connections to other texts or other works of art. Exceptional readers draw on evidence from the text to generate, validate, expand, and reflect on their own ideas.

These readers take risks. They entertain challenging ideas and explore multiple possibilities of meaning as they read, grounding these meanings in their acute perceptions of textual and cultural complexities. They often revise their understanding of a text as they re-read and as additional information or insight becomes available to them. They sometimes articulate a newly developed level of understanding.

Readers demonstrating a score point six performance challenge the text. They carry on an internal dialogue with the writer, raising questions, taking exception, agreeing, disagreeing, appreciating or objecting to text features. They may test the validity of the author's ideas, information, and/or logic by considering the authority of the author and the nature and quality of the author's source(s). They frequently suggest ways of rewriting the text, speculating about the ideology or cultural or historical biases that seem to inform a text, sometimes recognizing and embracing and sometimes resisting the ideological position that a text seems to construct for its reader.

Must the Student Just Do It or Do It Well?

Even more problematic than the likely disagreement among assessors as to the application of the vague standards provided is, as I suggested above, the problem of the assessors being given many criteria that name processes that can, in principle, be done well or poorly. The directions do not explain whether, to be credited, the student is obliged to use the cited processes well or simply use them in any way whatsoever. That is, there is no indication as to whether the assessor is to evaluate the “quality” of the way the student is

doing these things or simply certify the fact of doing these things — however poorly. Remember, a student who is drawing an absurd meaning is still drawing a meaning. A student who is making a trivial connection is still making a connection. A student who is drawing on irrelevant evidence is still drawing on evidence. A student who is raising a silly or superficial question is still raising a question. And so forth, and so on.

Unfortunately, once we examine the actual student writing examples along with the commentary provided, it becomes painfully clear that the assessors were simply looking to see if the student in any sense used the process and did not have, or did not use, criteria to assess how well the students used the processes credited. The California State Department of Education falls directly into the trap of failing to discriminate between these crucial differences. The assessors are left to their own devices. They can draw these distinctions or fail to draw them. It is clear that most failed to draw them. This is a fatal flaw in the assessment. It renders the results of the assessment virtually useless.

The Misassessment of Elementary Reading

The only elementary reading passage which is given with scored examples is from a story by John Gardiner called Stone Fox. It is an emotionally explosive story, one chosen perhaps to ensure an emotional response. The children are simply asked to give some of their “thoughts, feelings, and questions” about what they are reading. No other kind of writing is given as an example.

Insufficient Directions Are Given

Now the first remarkable feature of the Stone Fox reading prompt is that the student readers are given no indication whatsoever of the purpose for which they are reading the story. This is ironic in the light of the fact that the CAP materials emphasize the fact that one can read and write “for different purposes” (p. I-3).

And yet, here, the students are asked to read for no particular purpose. Are they to read in a casual fashion, simply for amusement? Or are they to do a “close reading” for detail? Are they to be analytical and reflective, or not? The students are given no indication of how they will be assessed. Hence, they are asked to engage, at best, in an ill-defined “performance.” Nothing is given in the way of directions to the students except “Read to see what happens” and a place to the right in which the student may write notes under a column head titled, “My thoughts, feelings, and questions about what I am reading.” The students were apparently not even told that they should try to write out as much of their thoughts, feelings, and questions as they could. Some readers might presumably have thoughts, feelings, and questions they would not bother to express.

Consider that you are a student reading the story. You see a column at the right which says, “My thoughts, feelings, and questions about what I am reading.” How should you understand it? Wouldn’t you wonder which thoughts ... which feelings ... based on what? What am I to think about? Why am I to think about it?

In any case, since the student reader is not told that she is going to be evaluated on what she writes and is told none of the criteria, why should she be motivated to fully express her thoughts and feelings? Consider, on the other hand, what the directions might have said.

Possible Directions

When we read a story we have to try to understand and follow its meaning. We try to figure out what it is saying and we try to connect it with our own life in some way. We want you to read this story and see what it means to you. Why do you think it was written? Do you think that it was written well? Do you think it is true to life? Does it illustrate anything that you believe is important? Please write in as many of your thoughts on these questions as possible as you read. Help us to understand what is going on in your mind as you are reading and trying to relate this story to your life.

Can One Evaluate Purely Subjective Responses to Stories?

When the student reads that she is directed to express her “feelings,” with no explanation given why she is so directed, then how can we legitimately go on to judge those feelings and “score” them one through six? Suppose the student quite sincerely said, “In my view this is a sentimental story that insults my intelligence. I feel disgusted when I read it and bored silly.” How should the assessor evaluate that “feeling” response? Is there any way to discredit it according to the directions? Certainly not. It is as good a response to a request for “feelings” as any other. To put the point succinctly, either we help students understand the difference between a request for a purely subjective response, (How do you feel when you read this?) and a request for a more reasoned response (What feeling do you judge the author wants you to feel and why? In your judgment is the author successful? Tell us the reasons why you think so.) or we must not indulge in any assessment of the students “feelings.” We have no legitimate grounds for doing so. The student can legitimately take the request to be one that asks for a subjective response and a purely subjective response cannot be impartially assessed or scored.

The Misassessment of Writing at the Elementary Level

Different kinds of writing will be assessed at different levels: Elementary: persuasive writing ... Middle School: problem solution, evaluation and speculation about cause and effect ... High School: evaluation, speculation about cause and effect, interpretation and controversial issues plus a reflective essay. (p. I–4)

The grade four writing assessment is designed to reflect a variety of purposes for which children write:

1. Expressive writing
   (“This is what I see, think, and feel . . .”)
   
2. Persuasive writing
   (“This is what I believe and why I think you should believe it . . .")
   
3. Narrative writing
   (“This is what happened . . .")
   
4. Informational writing
   (“This is what I know and how I know it . . .") 

Criteria For Persuasive Writing

Persuasive writing is explained in the following terms (p.lll — 12)

Persuasive writing requires students to choose positions, to make judgments, to offer proposals, and to argue convincingly for their beliefs and ideas. However, some students may choose to explore both sides of an issue and then offer a compromise, . . . Effective writers use evidence such as examples or anecdotes to support their arguments. Convincing arguments may appeal to logic, emotions, and/or philosophical beliefs.

Persuasive writers establish themselves as informed, knowledgeable individuals. They orient readers . . . More than any other kind of writing, argument requires writers to consider their audience . . . The best persuasive writers systematically develop arguments with a strong sense of coherence and movement throughout the piece.

California’s Standards for Exceptional Writing (Six Points)

An exceptional score (6 points) must meet the following standards:

Focus/Coherence

Position. Writers of six-point papers usually assert and maintain a clear position throughout the piece; they present evidence and explanations in a purposeful way.

Occasionally these writers will effectively evaluate both sides of an issue and offer a reasonable compromise; or they may conclude that neither position is preferable, or they may suggest a third position.

Organization. Writers arrange reasons, examples, information and/or personal anecdotes in a discernible and effective pattern resulting in an overall persuasive effect.

Coherence. Writers provide overall links or transitions; they present arguments, evidence, and reasons logically so that the overall effect is one of coherence.

Elaboration

Depth/Density of Arguments. Writers thoroughly develop and elaborate their reasons, examples, information and/or anecdotes. Some writers may develop only one reason or example, but they do so in depth; others may choose to develop several appropriate reasons, examples, and so forth.

Relevance of Arguments. Writers choose and present appropriate reasons, examples, information, and so forth to support their argument(s). They show their arguments are valid based on prior knowledge, personal experience and reflection.

Audience Awareness. Writers choose and present arguments with a clear awareness of reader needs. They often show credibility and a sense of authority by revealing source(s) of information. They may anticipate possible reader response by including some counter-arguments.

Style

Word Choice. These writers use lively, interesting concrete language that carries precise meanings and emotions. Word choice is appropriate to the writer's purpose.

Sentence Variety. Writers vary sentence length and type, making the writing interesting and readable.

Voice. Writers evidence confidence, conviction, belief and sometimes enthusiasm.

Problems With the Criteria In Assessing Writing

Once again there are a host of problems with the criteria, the most serious being that, just as in the criteria for reading, there is no indication of whether the students are expected to do any of the above well, or simply do them in any form whatsoever. For example, consider the claim that “convincing arguments may appeal to logic, emotions, and/or philosophical beliefs.” Suppose a student uses “convincing” but “fallacious” logic — is the student to be credited? Or suppose the student appeals to the emotions of the reader by engaging in name-calling (“This stupid communist idea...!”) — is the student to be credited? The formulators of the criteria are seemingly oblivious of the problem.

A Grade Four Essay Judged to Illustrate High-Range Achievement

The CAP Commentary

From the opening . . . this writer exudes confidence, focuses on an audience and takes a firm stand on the issue. These attributes of persuasive writing are maintained throughout the piece.

Support for the writer’s credible arguments in favor of a park versus a mall are drawn from personal experiences. She uses examples: “We already have over five stores in our plaza,” and notes that the park is a “perfect place to play, take your dog for a walk.” She includes reflection, “Do you honestly think that children at the age of two or three are going to enjoy a mall ...?” Her arguments are appropriate and appeal to reason (safety) and emotions (“I didn’t come here to watch those C.A.T.s tare up that park!”)

This writer is consistently aware of and appeals to her audience with appropriate tone and lively language.

The Problem of Subjectivity, Once Again

The problem of subjectivity that was so apparent in the elementary reading assessment reappears again in the writing assessment. It is clear in this, and in other examples, that the commentators are using the criteria for good persuasive writing literally and not making any real judgments of reasonability. Hence, if the student says anything which can be construed as falling under one of the criteria then that is credited. For example, if she uses an emotional appeal then credit the emotion criterion (however irrelevant or inappropriate it might be); if she says something that can be described as “reflection,” then credit the reflection criterion (however irrelevant or inappropriate the “reflection” might be). Any kind of a move that would work with an audience, in the view of the commentator, is credited as good (regardless of how irrational it might be).

But the significant question for anyone concerned with the traditional values of education is whether a reasonable audience should be persuaded or “moved.” The important distinction for students to grasp is that between what might be called “low level rhetorical appeals” and those appeals which would convince or “move” a reasonable audience. No such distinction is recognized by ELAA. The criteria are used crudely, without any important intellectual distinctions in evidence.

The implication of this is that if a student wrote a very rational appeal, which recognized the weaknesses in her position and the need to qualify her claims, she would be downgraded because she would not be “exuding confidence,” “taking a firm stand on the issue,” etc. Indeed, though her presentation might appeal to a “jury of reasonable persons,” the graders of the ELAA would not be impressed.

In this particular essay, for example, though it is good for children to display confidence, etc., the issue of park versus mall should be decided on rational grounds. Yes, it is important that the park is a favorite place for children and that they are safe there, but this is really a decision about options, and alternatives, and relative costs and benefits. This issue should be decided by looking at these factors and weighing them as rationally as possible, not on the basis of emotional appeals, like that of the child. To suggest that this is good persuasive writing is to teach children exactly the wrong lesson. It fails to show them the vital distinction between reason and its counterfeit.

Suppose the child has to present a case before an audience she knows to be racist; if one reads and applies the criteria presented for good persuasive writing in ELAA, the student who plays on those racist sentiments will score higher than one who opposes them.

What apparently matters in the mind of the ELAA assessors is that one successfully persuades, not that one argues reasonably. But it is harder to imagine a distinction which it is more vital for the educated person to grasp. Indeed, one might almost regard it as a criterion of being educated that one sees the difference between fairminded, reasonable argument, and self-seeking, low-level, persuasive rhetoric. The demagogues may often win the day, but do we want to use public monies to generate armies of demagogues, all having mastered the art of demagogery at the public expense?

In another example, which contains some very good reasoning about why another child should feel good about himself, the commentators say, “She arranges her reasons and evidence in a sophisticated pattern.” There is no mention of whether the reasons and evidence are relevant or irrelevant, true or false, good or bad.

Yet, again, isn't this what really matters? It is not that the reasons are arranged in a sophisticated pattern, but that they are good reasons!

In another, mid-range, example, about how to solve problems between children and parents, the commentators say (among other things), “Audience awareness is evident throughout, although appeals lack the vigor and exactness of higher score point papers. The use of bullets to summarize the writer’s arguments is an effective tool and adds to her sense of conviction.” Why is the writer criticized for a lack of vigor and praised for conveying a sense of conviction? Neither is a virtue in itself. What matters is that one exhibits the appropriate degree of vigor and conviction, depending on the strength of one’s case. Vigorously arguing a weak case and displaying conviction despite poor supporting reasons ought to be marked down, not praised.

The Young Hitler Scores High on the CAP Test

We can now make our point dramatically by considering how the following piece of persuasive, but highly irrational, writing should be graded according to the ELAA criteria.

This piece asserts and maintains a clear position throughout and presents evidence and explanations in a purposeful way; the writer arranges reasons, examples, information and/or personal anecdotes in a discernible and effective pattern resulting in overall persuasive effect; the overall effect is one of coherence; the writer shows his arguments are valid based on prior knowledge, personal experience and reflection; the writer chooses and presents arguments with a clear awareness of reader needs; the writer evidences confidence, conviction, belief, and sometimes enthusiasm.

Clearly, for all these reasons we have to give this piece of Hitler’s writing a Point 6 score! It meets the CLAS criteria, as does much of his writing. Hitler’s writing was widely recognized to meet the needs of his audience, to exude confidence, etc. Is that really what we want to praise? Is that the model of persuasive writing that we want to hold up to our children? If so, shame on the California Learning Assessment System!

The Misassessment of Writing at the High School Level

Introduction — Same Problem: Next Level

At the high school level the Writing Assessment assesses four types of writing, Autobiographical Incident, Interpretation, Reflective Essay, and Speculation About Causes and Effects, some of which are successors to the elementary level Persuasive writing.

The same general faults that were mentioned earlier in connection with the elementary level are to be found at this level too. For example, the scoring guide for interpretation makes it clear that only subjective reactions to and subjective interpretation of fiction are really being considered. But interpretation is important in many other contexts, e.g. history, and in history it is crucial to distinguish between subjective response and objective interpretation, between reasons that persuade irrational audiences and reasons which persuade rational and fairminded persons, between something which is rhetorically convincing and something which is true. None of these distinctions is recognized in the CLAS.

To be specific in our criticisms, consider the scoring guide for Speculation About Causes and Effects; once again writing of this kind is judged almost entirely by subjective standards, by standards appropriate to one’s response to fiction.

Here is what is said about writing of this kind which should score Point 6, for Exceptional Achievement:

A six-point essay engages the reader immediately. It seems purposeful. The writer seems aware of reader’s questions and needs throughout the essay. The essay seems to be not just written but written to particular readers. The writer convinces the readers of the plausibility of the speculation.

A six-point essay demonstrates qualities all readers admire: conviction, enthusiasm, freshness. These essays may use an unconventional rhetorical approach. A six-point essay may take chances and succeed.

Presenting the Situation. The six-point essay writer clearly defines, identifies, or describes the situation to be speculated about. Though it does not dominate the essay at the expense of speculation, the situation is nevertheless presented fully and precisely. The writer limits the occasion appropriately, focusing reader attention on just those aspects of the situation that the writer will speculate about.

Writers of six-point essays may describe or detail the situation that is established in the prompt, or they might create the situation by using narrative or anecdotal techniques. In either case, they will use concrete language, rich in sensory detail.

The writer of the six-point essay acknowledges readers’ concerns. For real world situations, the writer of the six-point essay acknowledges the reader’s experience or familiarity with a situation and, using narrative or descriptive strategies, builds on this awareness to focus reader attention on a comparable situation. ...

Whether the essay arises from a factual assessment of a real situation or from a fanciful guess about a fanciful situation, the writer consistently demonstrates broad knowledge and clear understanding of the situation. In this way the writer establishes authority.

Logic and Relevance of Causes and Effects. In the six-point essay, the proposed causes and effects are clearly related to the particular situation that the writer has defined. Writers use imaginative, inventive argument to convince the reader of the logic of their speculation. The best writers are clearly considering possibilities and are seeing multiple perspectives.... Because speculation is essentially a persuasive type of writing, the best writers will be continually aware of readers’ needs. They might refer to the readers directly, trying to enlist their support ...

Elaboration of Argument. The six-point essay provides substantial elaboration, convincing the reader that the writer’s conjectures are valid for the situation. These writers elaborate their speculated causes and effects with carefully chosen evidence that is logically and fully developed. Such evidence is chosen because it is relevant and convincing. It is developed fully with precise, explicit detail to convince the reader both of the logic and the authenticity of the proposed cause and effect.

Some strategies writers may use to develop their arguments are the following:

• Cite facts, opinions, projections, and personal experiences or observations (anecdotes) to explain or validate a cause or an effect.
• Elaborate on possibilities arising from proposed causes and effects, showing possible “domino effects” that might determine the direction of the developing situation.
• Give specific examples of comparable causes and effects that have arisen in analogous situations. (p. III–64)

How may we best make the point that these criteria again fail to recognize the crucial distinctions of which we have been speaking — between subjective responses and good reasons, etc? Perhaps the simplest way is to look at another example of persuasive writing, this time Speculating About Causes and Effects, and consider how we should grade it according to the CLAS criteria.

Just as every people, as a basic tendency of all its earthly actions possesses a mania for self-preservation as its driving force, likewise is it exactly so with Jewry too. Only here, in accord with their basically different dispositions, the struggle for existence of Aryan peoples and Jewry is also different in its forms. The foundation of the Aryan struggle for existence is the soil, which he cultivates and which provides the general basis for an economy satisfying primarily its own needs within its own orbit through the productive forces of its own people. Because of the lack of productive capacities of its own the Jewish people cannot carry out the construction of a state, viewed in a territorial sense, but as a support of its own existence it needs the work and creative activities of other nations. Thus the existence of the Jew himself becomes a parasitical one within the lives of other people. Hence the ultimate goal of the Jewish struggle for existence is the enslavement of productively active peoples. In order to achieve this goal, which in reality has represented Jewry’s struggle for existence at all times, the Jew makes use of all weapons that are in keeping with the whole complex of his character. Therefore in domestic politics within the individual nations he fights first for equal rights and later for super-rights. The characteristics of cunning, intelligence, astuteness, knavery, dissimulation, etc, rooted in the character of his folkdom, serve him as weapons thereto. They are as much stratagems in his war of survival as those of other peoples in combat. In foreign policy he tries to bring nations into a state of unrest, to divert them from their true interests, and to plunge them into reciprocal wars and in this way gradually rise to mastery over them with the help of the power of money and propaganda. His ultimate goal is the denationalization, the promiscuous bastardization of other peoples, the lowering of the racial level of the highest peoples as well as the domination of this racial mish-mash through the extirpation of the folkish intelligentsia and its replacement by the members of his own people. The end of the Jewish world struggle therefore will always be a bloody Bolshevization. In truth this means the destruction of all the intellectual upper classes linked to their peoples so that he can rise to become the master of a mankind become leaderless. — (Hitler’s Secret Book, pp. 212-213)

Hitler’s Assessment Based on CAP Criteria

• This piece of writing certainly engages the reader immediately;
• it seems purposeful;
• the writer seems aware of reader’s questions and needs throughout;
• the piece seems not just to be written but written to particular
  readers;
• the writer convinced those readers of the plausibility of the
  speculation;
• the writing shows conviction and enthusiasm;
• the writer clearly defines the situation to be speculated about, acknowledges the reader’s experience or familiarity with the situation, and consistently demonstrates broad knowledge and clear understanding of the situation;
• the writer uses imaginative, inventive argument to convince the reader of the logic of his speculation;
• the writer elaborates on possibilities arising from the proposed causes and effects, showing possible “domino effects” that might determine the direction of the developing situation; etc., etc.

Can there be any doubt that Hitler’s writing in the category merits a Point 6 grade?! If that is so then once again, shame on the California Learning Assessment System, which has again failed to see the difference between a proper and improper use of rhetoric and reason.

Summary Judgment on the California Assessment of Reading & Writing

An assessment of reading and writing should not only underscore the role of reasoning in both, but also firmly establish defensible intellectual standards, appropriately and specifically explained and consistently and appropriately applied.

No assessment of intellectual work, nor foundation for teaching, should be based on an approach in which intellectual standards are confused and erroneous, confusing recall with knowledge, subjective preference with reasoned judgment, irrational with rational persuasion. No assessment of intellectual work should use its key terms vaguely or oscillate between two significant uses of a term or score in an arbitrary manner. And, most important of all, no intellectual assessment should encourage irrational subjectivism.

For example, it is striking that in the context of reading and listening, there is little discussion of the need to create an accurate interpretation: there are many contexts in which it is not appropriate for the readers to “create their own meaning” and where accuracy is what is required. This may be equally true if the author is addressing a particular question or problem, or using particular basic concepts; very often the good response to what is said or written is good precisely because it is based on an accurate construal of the text. Nothing in this assessment mentions the virtue of accuracy.

Most importantly however, this approach is flawed again and again because what gets credited is anything that could be construed as fulfilling one of the criteria and the criteria are the wrong ones for the purpose. If the writer is using any emotion (however discreditable) then it is marked positively; if she is giving anything that could be called evidence — even if it is bad evidence — it is credited; any kind of a move that would work with an audience, in the view of the reader is credited as good (regardless of how irrational it might be).

The end result is that if a student wrote a very rational appeal, one that would persuade a rational audience, it is going to be graded down because it wouldn’t appeal to an irrational judge (because it won’t be maintaining a strong line, it won’t necessarily be persuasive; it will be putting in qualifications; it will be speaking in terms of greys and greys don’t persuade; it will not be engaging in hyperbole and hyperbole is effective; it won’t be trying to negate everything about the other side, it will be recognizing reasonable objections; it will express the degree of confidence that is appropriate but no more). Given a list of what a rational person would do, you will be able to see that the criteria for success under CLAS are negating these rational qualities and therefore encouraging irrational beliefs about how you communicate to people, indeed encouraging people to become manipulators!

Underneath all of this is a question of values. We are obliged to educate our students, not simply to shape them.

The educated person is reasonable; the educated person isn’t simply concerned with winning. The educated person wants to win when winning is the appropriate thing. However, when the other side is more reasonable, then the educated person, who is more interested in getting at the truth than in winning, will want to make concessions.

If the goal of education is simply to enable people to get what they want, then we should teach them tools of manipulation, ways to win battles, ways to undermine positions whether those positions are rational or not, ways to just get what they want, irrespective of fairness, and of the evidence. Inadvertently and unknowingly, this is what CLAS is supporting. CLAS does not have in mind a clear difference between the educated, fairminded, and rational person, on the one hand, and the person who is simply good at manipulating, winning, and defeating others, on the other.

In effect what CLAS has said is that the name of the game is to persuade the audience by whatever methods work and we’ll credit anything that works. Shouldn’t they instead have said, “Since we are obliged and committed to educating children, and since this requires they learn to reason well as readers, writers, and thinkers, we will not credit flawed reasoning. We will only credit well-reasoned responses.”

To Summarize Some of Our Criticisms of the CLAS Approach

1) The overall conception fails to capture the practices of critical readers, writers, and thinkers the world over.

2) The overall conception does not call attention to definite and clear intellectual standards, and without them, it becomes impossible for both teacher and student to engage in “objective” assessment.

3) The overall conception does not provide an organized and systematic approach to posing, analyzing, and reasoning through problems embedded in everyday personal and professional reading and writing.

4) The teacher who takes this approach seriously will misteach reading and writing.

5) The student who learns through this approach will mislearn the art of reading and writing.

Given these failings, it is clear that the California Learning Assessment System falls into the category of pseudo critical thinking, and will not help students and teachers to develop their critical thinking abilities, but will hinder this process. Indeed, when classroom teachers receive copies, as they surely will, of test items, sample answers, commentary, and scores, they will use them as a guide for instruction. Thousands of school children will lose an opportunity to begin to become critical readers and writers. Thousands of school children will themselves learn to confuse recall with knowledge, subjective preference with reasoned judgment, irrational with rational persuasion. They will learn to use language vaguely and to think that their subjective pronouncements are not to be criticized. Their reasoning skills will remain abysmally low.

Is The California Assessment Fiasco a Fluke?
The Educational Bureaucracy and Self-Deception

One of the most significant facts about the California language arts test fiasco is that it is not a fluke. But of course, neither is it a plot to undermine education. The situation is worse than either. As a fluke it could be corrected. As a plot the perpetrators could be severely dealt with when exposed. No, the pathetic side of the case is that there are systemic reasons why educational bureaucracies, framed as they are, will continue to generate just such fiascos regularly and predictably. And predictably, many will be taken in. Furthermore, because classroom teachers have emerged from a long-term training that reflects a similar background to that of the test designers, they also will fall easily into line with the flawed thinking passed down to them.

To deal with the problem at its roots, we must own the fact that there are significant problems in education due to its wide-spread and large-scale bureaucratization.

Large-scale bureaucratization entails, or at least makes highly probable, a high degree of narrow specialization — and specialization tends to bring fragmentation, narrowness of vision, politicization, and self-deception in its wake. The fragmentation and narrowness of vision makes it difficult to effect fundamental changes because the parts do not work together in a rational way and no one sees clearly that this is so, since each element in the structure becomes an end in itself, to itself.

It is almost impossible for the most pressing problems of education to become “issues” in educational bureaucracies because the focus is inevitably on the political, the narrow, the fragmented part or parts. With each part serving itself as an ultimate end — including those on the top — the whole is left to take care of itself. No one is left responsible for it.

The executive wing is also focused on itself and typically is satisfied with or driven to manufacture an illusion of serving the announced or official goals and ends. Meanwhile, the politicization and self-deception helps hide those realities most unpleasant to think about, and to have to face, and consequently those realities most in need of change.

This includes, of course, the most significant one today: the fact that modern American bureaucratic schooling is a system that preserves at its heart a mode of instruction that is a hold-over from the 19th Century and whose consistent effect is a superficial one. Most students in most classes most of the time are not actively engaged in learning what is worth learning. Most students are, on their side, not taking their education seriously. On the teaching side, they are not given challenging instruction. They are not engaged in genuine intellectual work. They are not developing intellectual standards or discipline. And, most assuredly, they are not learning to reason scientifically, mathematically, geographically, economically, sociologically, or morally. (See “Research Findings,” p. 19.)

Modern educational bureaucracy has developed multiple ways to appear to be, and to appear to be doing, what it is not. In the K–12 domain especially, the history of education in the 20th Century is a triumph of propaganda and self-deception. This is documented in story after story of wave after wave of pseudo reform following pseudo reform, of new buzz words and new jargon replacing old buzz words and old jargon — each set of new words serving as a new mask to obscure the one-and-the-same consistent lower order face. (Cf “Critical Thinking in Historical Perspective chapter.) Unfortunately, virtually everyone in the game has a stake in making their playing of it look more honorable, more lofty, more noble and effective than it really is.

You may remember that it is only some three years now since every state in the union, through the massaging and manipulation of statistics by its own state department of education, proudly announced that its students had scored above the national average!!! This is the kind of self-serving propaganda and trickery that is the daily fare of educational reality.

State departments of education, as I have suggested, are a particularly interesting manifestation of the workings of the educational establishment. Each consists of huge bureaucracies, interlaced with committees that are in turn tied into networks of teachers and administrators spread across their states. The microcosm we analyzed in this chapter (the California State Department of Education’s new English-Language Arts Assessment materials) is still officially in draft stage, but already is being highly touted as a refined, future-oriented, testing instrument. We’ve seen what the ELAA has done. Let’s see what CLAS and the Department of Education says it has done. The test is being represented to the public and to teachers within the state and the nation as having,

The goal ... to evaluate students’ capacities for insightful, productive thinking with tests that support the finest curriculum and instructional programs in the language arts ... [California’s Learning Assessment System, CDE Publication]

The most important single component of the new assessment system will be the statewide performance standards, and the most important outcome of the assessment process will be the internalization of those standards in the thinking and work of teachers, students, and parents. The performance standards will undergird all aspects of the educational enterprise; serving as the center of the seamless web of teaching, learning and assessment. [Some Principles and Beliefs about the Role of Assessment in California's School Reform Plan, February 15, 1993, CDE]

Let’s see what the legislature mandated, Senate Bill 662 (Hart)

Develop a system for producing valid, reliable individual scores and to develop and implement common statewide performance standards of student achievement as a basis for reporting all test results and setting targets for improvement.

It sounded simple enough but it provided us with a classic model of pseudo critical thinking in the educational establishment. The manner in which it is structured provides a textbook case. By mirrors, illusion, and standards self-deception, it creates the

appearance of substantial change and reform. In fact, nothing is really being changed. The result is that the most fundamental problem in education today — that students are not learning to reason well — is not only ignored, it is intensified.

Fragmentation and Vested Interests

There are a number of reasons why it is unlikely that fundamental reforms will be effected by state departments of education or that this critique, by itself, will bring about fundamental change. In the first place, most of the positions within the state departments of education are for specialists, for example, positions for those specializing in nutrition, for those specializing in transportation, for those specializing in the laws regarding education, for those specializing in learning disorders, for those specializing in a and b and c and d and e and f and g. Virtually no one, however, has a responsibility directly connected to the fundamental goals of education (except possibly the director of instruction). Each specialist has his or her own special interest to focus on and a special group of stake-holders to represent.

When there is a need to develop an assessment instrument, like the one we examined of the California Department of Education, a large group of teachers and administrators from around the state are appointed. For example, there are 33 members of the CAP English-Language Arts committee. In addition to the main committee there is a supporting committee (“Reserve Team”) consisting of 46 additional members. According to the California State Department: “These development team teachers have been responsible for shaping the test format, developing prompts for the assessment, and constructing scoring rubrics.” (p. I-2)

Many political considerations go into the selection of the members of the development teams; most of the members, for example, are not scholars with publications that could be used as the basis of selection. There is an effort made to balance the committee by region, gender, race, and ethnicity. In addition, many members will have personal agendas to advance.

There are usually three “Consultants/Advisors” selected from universities to bring in the over-arching theoretical framework. These consultants are usually the pipeline to the latest buzz words and to the theory behind them. The consultants concede to each other the right to get their favored terms into the language of the test materials.

The result of this process, as we have seen, is that the official “standards” embodied in the test become extraordinarily numerous. Many of them remain vague and ill-defined. Others take on a dangerous ambiguity. The diverse criteria and the open-ended nature of the directions combined with the ill-defined nature of the terms, opens the way to arbitrary and inconsistent grading of student responses. The test becomes, then, both invalid and unreliable. In fact, it becomes a hot-bed of pseudo critical thinking, with a variety of misconceptions and flaws emerging. But while its intellectual value is low, its political value is high. The various political interests around the state are served. The media has a simplistic event to cover. Parents can delight in the fact that the scores will go up. (How can they go down when anything can count as a good answer?) The politicians will gain because they can speak of their state as in the vanguard. And so it goes. A new pseudo reform is put in place and the educational bureaucracy grinds on until the next wave of public criticism requires it to generate a new and fresh illusion of change, a new catalogue of counterfeit, bogus, and superficial “reforms.”

So What Can We Do? Recommendations

There is a pressing need to develop networks of educators, parents, politicians, and business people who see the need for truly fundamental reform. That reform must be advanced simultaneously on many levels, for it is not going to result from action on one level alone. Because it must go to the roots of things, because it must be substantial, because it involves deep understandings, it must be incremental, evolutionary, and long-term. Everyone with the insight to see the problem comprehensively should act within the sphere of his or her greatest influence. There is a role for everyone concerned to exercise influence for the better: for parents, for public citizens, for business people, for civic leaders, for superintendents, for teachers, for college professors, and ... yes, even for those in state educational bureaucracies. Let us consider each briefly in turn.

What Can Parents Do?

Insightful parents can make the case for an emphasis on intellectual discipline and reasoning in the school curriculum. They can ask whether there is any long-term in-service in critical thinking and reasoning. They can ask what intellectual standards the students are being taught and how they are being taught them. They can make the case to other parents. They can write letters to the local papers. They can organize groups of parents who petition the school board. And most important they can develop a home environment in which the reasonability and intellectual discipline of their children is fostered, in which both they and their children routinely ask and give good reasons in support of their decisions and reason together about issues of importance not only to the family but to the broader society as well.

What Can Citizens Do?

Insightful public citizens can make the case for an emphasis on intellectual discipline and reasoning in the school curriculum in virtue of the need to develop voters who will help the country maintain a democratic form of government. They, too, can go to the local school board and ask whether there is any long-term in-service in critical thinking and reasoning. They, too, can make the case to parents and other citizens. They can contact civic groups. They can write letters to the local papers. They can organize groups of interested citizens to petition the school board.

What Can Business People Do?

Insightful business people can use the respect that their success commands to exercise influence, alone or in concert with others, over educational decisions about what to teach and how to teach it. Since their success will be increasingly dependent upon their bringing critical thinking into the inner workings of their own businesses, on workers learning how to continually relearn and improve in their performances and in the systems they use, they will have ready access to models and paradigms that can be used to illuminate what should be happening in the classroom. Increasingly, cutting-edge businesses are moving away from an emphasis on hierarchy to an emphasis on group problem solving. Since critical thinking is essential to effective group problem solving, progressive business people will be able to talk intelligibly with educators and other citizens about how problem solving structures function in business and how parallel classroom problem solving groups might be set up. And, certainly, there are any number of civic groups that business people with insight might address on the problem of educational reform, putting emphasis, of course, on the missing foundation: the failure of teachers to learn how to think critically themselves and to teach for that thinking in their instruction, the failure to focus education, in other words, on “carefully-reasoned” problem solving. Finally, insightful business people can form alliances with insightful educators, to create symbiotic, reflective, mutually useful dialogues on what each group can learn from the other and how each can profit by working together.

What Can Civic Leaders Do?

Insightful civic leaders can draw public attention to the need for intellectual discipline and reasoning in instruction. They can articulate publicly the key links to developing responsible citizens, moral persons, and workers on the cutting edge of development. They can use their access to a more public forum by focusing the discussion of educational reform on the historical problem of the educational bureaucracy and its tendency to generate pseudo reform. They can create a public awareness of the importance of reasoning, critical thinking, and problem solving. They can help organize civic groups. They can use their superior access to other persons of leadership and influence to facilitate significant pressure on the educational bureaucracies. They can make contact with insightful and responsible politicians who are in a position to facilitate appropriate legislation.

What Can Superintendents Do?

Insightful superintendents can make the case for an emphasis on intellectual discipline and reasoning in the school curriculum to the school board, administration members, teachers, and parents. They can ensure that there is long-term in-service in critical thinking and reasoning. They can ensure that students are being taught intellectual standards in depth. They can create incentives to teachers motivated to move in this direction. They can make the case to civic groups. Most importantly they can model reasonability and help create an atmosphere conducive to making the school a network of communities of inquiry.

What Can Teachers Do?

Insightful teachers can make the case for an emphasis on intellectual discipline and reasoning in the school curriculum. They can request and help design long-term in-service in critical thinking and reasoning. They can bring intellectual standards into the classroom. They can make the case to parents. They can work with other teachers to foster a school environment in which reasonability and intellectual discipline are accepted school norms. Most importantly, they can routinely ask for and give good reasons in the classroom. They can foster student reasoning in history, science, math, and so forth. They can ensure that students must regularly assess their own work using intellectual standards.

What Can College Professors Do?

Insightful college professors can make the case for an emphasis on intellectual discipline and reasoning in the college curriculum. They can request and help design long-term faculty development in critical thinking and reasoning. They can bring intellectual standards into the classroom. They can do research on the significance of critical thinking and reasoning in their discipline. They can work with schools and departments of education to ensure that those studying to become teachers take classes that require reasoning and disciplined thought. They can articulate the need for prospective teachers to learn how to design assignments that require reasoning and critical thinking. Most importantly, they can routinely foster reasoning in their own classrooms and ensure that their students must regularly assess their own work using intellectual standards.

What Can Those in State-wide Bureaucracies Do?

Insightful members of state-wide bureaucracies (who recognize the systemic ways that educational bureaucracies have fostered pseudo reforms and constructed ill-designed assessments) can play a number of significant roles. They can inform themselves and others they work with of the fundamental changes that are being made in businesses adopting structures contrary to those of traditional bureaucratic organization. They can foster movement toward problem-solving teams. They can raise broader and deeper issues. They can recommend hiring people with broader vision and more developed reasoning abilities. They can help to work against narrow specialization. At the same time, they can argue for more appropriate use of experts, so that those who lack expertise in a subject will not become, for example, principal designers in tests or assessment instruments in that subject. They can argue for the construction of assessment instruments that assess reasoning in every subject area and so help to integrate emphases across subject areas.

Caveat

Doubtless you noticed my emphasis on “insightful” in characterizing those who can make important contributions to reform. It is important to underscore the problem of pseudo reform, which emerges when well-meaning persons use their intelligence inadvertently to re-duplicate an old problem in new form, creating the illusion of change. Many persons today are unwilling to think through the implications of accelerating change and intensifying complexity. Many are subconsciously wedded to rigid ideas and a static way of thinking. Many are taken in by their own platitudes and high-sounding words. These facts guarantee that a long struggle will be required to work through the superficial and work into the substantial.

Final Conclusion

Pseudo critical thinking is more or less inevitable in the educational bureaucracies, given the way we have traditionally arranged and ordered things. This is illustrated, as we have seen, in the way the American educational establishment goes about designing assessment. Unfortunately, faulty assessment leads to faulty teaching, which leads to more faulty thinking in society, in business, in politics, and in everyday social life. The California Department of Education is a model case of American educational bureaucracy at work and the new California reading and writing assessment instrument is the typical resultant bad practice. Good thinking is now a fundamental human need. And though it will take generations to fully evolve from a society in which pseudo critical thinking is dominant to one in which sound, fairminded, ethically-informed reasoning is dominant, every step in that direction will reduce the amount of suffering and injustice that exists and increase, by degrees, human well being and quality of life. It is our intellectual and moral responsibility to make some contribution to this evolution. Though we are only at the beginnings of this evolution, the irresistible dynamic of accelerating change and intensifying complexity will eventually force it upon us. I hope we learn our lessons sooner rather than later, that the price of waste and unnecessary human misery may be as little as possible.

References

All of the references in this chapter (unless otherwise noted) are from the Samplers for English Language Arts Assessment, for Elementary and High School, disseminated statewide in the Spring, 1993, by the California Department of Education.

Back to top