Guidelines for Submissions

APA NEWSLETTERS
    American Indians
        Viola F. Cordova &
        Anne Waters, Co-Editors
    Black Experience
        Jesse Taylor, Editor
    Philosophy and Computers
        Jon Dorbolo, Editor
    Feminism and Philosophy
        Joan Callahan, Editor
    Hispanic/Latino Issues in
    Philosophy
        Eduardo Mendieta, Editor
    Philosophy and Law
        Richard Nunan, Editor
    Philosophy and Lesbian,
    Gay, Bisexual and
    Transgender Issues
        Timothy Murphy, Editor
    Philosophy and Medicine
        Rosamond Rhodes, Editor
    Teaching Philosophy
        Tziporah Kasachkoff &
        Eugene Kelly, Co-Editors

Navigation
    Newsletters Index (00:2)
    apaOnline Home Page

APA Newsletters

Spring 2001
Volume 00, Number 2

Newsletter on Teaching in Philosophy

Articles

Previous Article | Index | Next Article

Analyzing the Scope of Critical Thinking Exams1

Don Fawkes
Fayetteville State University, Fayetteville, North Carolina

This paper reports the results of an analysis of the scope of major commercially available machine scoreable critical thinking exams. It examines the range of competencies that these critical thinking (CT) exams attempt to measure. No assessment of the quality of the exam questions is given here. The tests examined are: The Watson Glaser Critical Thinking Appraisal; The California Critical Thinking Skills Test; The Cornell Critical Thinking Test, Level X; The Cornell Critical Thinking Test, Level Z; and, The California Critical Thinking Disposition Inventory.2 Of these instruments, only the last attempts to measure CT dispositions like clarity, accuracy, curiosity, etc. Since a disposition is roughly a skill plus a propensity or intent to use it, dispositions can be stated as skills, e.g., to exhibit clarity, to exhibit curiosity, and so on.

As the political process in higher education continues to lead institutions to seek ways to evaluate results of instruction, CT tests are often proposed and used. Hence, becoming "test savvy" is useful. By making a careful listing of the skills addressed by these tests, we can come to an appraisal of their usefulness in terms of the range of skills they address. A fairly comprehensive inventory of more than 250 basic CT skills is posted at the website http://smartthink.webjump.com. Though the inventory was originally compiled based on the author's knowledge of standard texts, teaching experience, and experience working with diverse faculty on critical thinking projects, it is also informed by commercially available CT tests and by three interdisciplinary models of critical thinking.

In the presentation below the three CT models are crosschecked with the analysis, that is, if a skill is found in one of the three CT models this is indicated by a letter designation: D, for the Delphi model; E, for the United States National Educational Goals model; and S, for the Sonoma model. (Please see references.) As the Delphi model is likely the best and most widely known, the inventory is listed under the six main headings of that model: 1. Interpretation; 2. Analysis; 3. Evaluation; 4. Inference; 5. Explanation; and 6. Self-Regulation. (Self-Regulation is the Delphi category for dispositions.) The CT skills listed in the analysis are stated as objectives; each completes the phrase, "A critical thinker is able to…". For dispositions the phrase is, "A critical thinker is able to and has a propensity to…".

The reader is invited to participate in this process of analysis and appraisal, to obtain copies of the exams, to make independent analyses, and to suggest additions, criticisms, modifications, etc. The reader may also wish to respond through the website or directly to me at fawkesdx@yahoo.com.

Some Qualifications and Related Matters

This section provides a bit of context by discussing: (1) Ranges of skills and competency within them; (2) statement and interpretation of skills; (3) critical thinking and critical apprehension; and (4) critical thinking is nothing new.

(1) Ranges of skills and competency within them. The skills listed under the six major headings may not all be equally important to being generally competent in critical thinking. But though it might be thought that this means that some sort of ranking of the skills is needed, this is probably illusory, for different skills will be important in different contexts. So, a reasonable general ranking is unlikely. Furthermore, though evidence of mastery of a wide range of skills may serve as an indication of general CT competency, it is not a sufficient indication. As one reviewer of this paper noted, "Coverage of the range [of skills] is not a measure of how one works within it, much in the way that a large vocabulary is not the same as a good vocabulary." Hence, a demonstrated competency over a fairly wide range of CT skills is necessary but not sufficient for general critical thinking competency. (Here traditional essay test questions and other writing assignments have the potential to be better measures of abilities within a range of skills.) These points also hold with regard to the testing at the collegiate level of virtually any subject. So, these points have broad import, and should raise notes of caution with regard to all machine scoreable testing and all testing of specific skills. Insofar as such testing can address a range of combinations of skills, then so far these limitations can be ameliorated. But as is shown below, the amelioration of these limitations by current CT tests is at best minimal; and, it is also worth noting that time and space limits of testing make it quite unlikely that any machine scoreable or other skills test can be designed to reduce significantly these severe shortcomings. So, for virtually any collegiate subject these points show limitations on what machine scoreable tests or other specific skills tests are able to measure.

None of these points however obviates the need to state the skills properly to be expected of a competent critical thinker-we still need to be able to state what it is that a competent critical thinker is able to do. Otherwise, we are unable to say with any clarity, accuracy, or precision what critical thinking is. If we can't state the components, we can't meaningfully discuss combinations of the components that would demonstrate general competency.

(2) Statement and interpretation of skills. At least with regard to critical thinking (and likely with regard to many collegiate subjects), the statement of the skills themselves is no easy matter, and the skills require considerable subject matter expertise to interpret. For example, consider the skill, "interpret and apply complex texts, instructions, illustrations, etc." Having stated the skill we need to know how interpreting or applying are to be measured, and what sort and what degree of complexity is involved. These questions ultimately will have to be answered with examples. Or consider the skill, "evaluate whether an inductive argument is strong or weak." One of the reviewers of this paper commented,

Since inductive arguments vary enormously with regard to the number of stated and implicit premises, their relationship to each other and to the conclusion, and the structure of the various [implications] within, evaluating some inductive arguments as strong or weak may be an easy task while evaluating other inductive arguments may task the skills of even the strongest critical thinkers.

Once again, we will need examples to make clear the level of challenge to be met. But again, none of these points argues against stating the skills or identifying the skills tested by typical exams, but they do show that there is a good deal more reasoning to be done once the skills are stated, and they do raise further cautionary notes with regard to the general usefulness of such tests.

(3) Critical thinking and critical apprehension. The skills found within the general categories of 1. Interpretation; and 2. Analysis may be seen as preliminary to skills at the core of critical thinking. Fisher and Scriven (1997) do not consider these skills to be properly critical thinking skills but as preliminary to CT. One reviewer of this paper makes essentially the same point, considering them to be closer to "critical reading comprehension" skills. Whether these skills are considered proper parts of CT or not seems to be primarily a matter of emphasis. But in testing CT, if test takers do not have a fairly good grasp of these skills, it is unlikely that they will have any chance of performing well on other CT skills. The same point applies for the teacher working in the classroom. Further, as is shown below, the current exams mostly attempt to test skills in these categories 1 and 2. Perhaps the best way to handle this matter is to consider these skills as the critical apprehension portion of CT.

(4) Critical thinking is nothing new. If we were to list the topics addressed by the whole panoply of CT textbooks that have come onto the commercial academic market since the 1970's we would find virtually nothing that is not addressed in Mill's System of Logic of 1859. "Logic" is the ancient name for the content of "critical thinking," a term that began to appear in the 1970's. By that time the term "logic" had become mostly identified with deductive reasoning. But logic has always included both inductive and deductive reasoning, a division that is exhaustive and mutually exclusive. Nevertheless, quite typically nowadays the term "logic" is mentioned in a phrase such as "formal logic," or "deductive logic," or "formal logical systems," or "formal deductive logic." The formal here is not honorific nor is it the formal of social relations to be contrasted perhaps with casual and compared with ceremonious. The formal in these phrases is the formal of form, the form of ordinary statements. If the statement "All idiots are vociferous" is true, then it follows as a matter of the form of statements that if someone is not vociferous then she or he is not an idiot. Recognition of such facts of form appears to be somehow deep in the nature of human language and naturally available to humans. But formal consequences turn out to be much more subtle, complex, and extensive3 than these natural first steps. And the less obvious facts of form appear quite immediately. For example, it takes most beginning students a little while to see that from the statement "All idiots are vociferous," it does not follow as a matter of form that if someone is vociferous then he or she is an idiot.4 Or again, (though the distinction is almost as obvious as any basic fact of form) any informed and watchful review of the deliverances of news media will disclose that otherwise well educated public officials and representatives of the media routinely confuse contrary and contradictory claims.5 Hence, deductive logic beyond the most naturally obvious parts must be studied and learned in order to grasp more of it. These inescapable facts of form are naturally part of human language; they are in us as much as verbs and nouns. They are readily found in our intellectual products from popular literature like mysteries and romances to theories like the general theory of relativity in physics. And insofar as we are ignorant of them we are ignorant of ourselves, ignorant at our own risk for (1) we live and think and believe and act by reasons we give ourselves and others, (2) the reasons we give are inevitably affected by facts of form, and (3) we live and think and believe and act this way quite independently of whether or not we recognize this fact or care to recognize it. An unexamined life, here as in other respects, is not worth living.

And if we wish to find out whether or not all idiots are vociferous or anything else beyond the facts of form, we need to turn to observation and experiment and inductive reasoning. When the products of these are combined with the facts of form, then we sometimes succeed in figuring things out. It is in this combination that our most valued intellectual products are found. Deduction and induction form a thinking collaboration, a collaboration without which we can come to figure out almost nothing. So, an intellectual rejection of either is simply silly.

In this context it seems somewhat odd that certain remarks occasionally appear in serious professional discussions of critical thinking, an oddity to be explained. The sorts of remarks I have in mind are ones like the following: "One particularly astute critical thinker of my acquaintance claims to know nothing about deductive logic." Or, "These materials seem to be too much like traditional logic." Or, "The materials appear to be primarily those of the philosophy discipline." Or, "The reviewers did not feel that the materials would appeal to groups outside the discipline of philosophy." Or, "This approach is not sufficiently interdisciplinary, depending too much on logical thinking." I have encountered these and similar remarks often over more than twenty years of work in critical thinking. Such remarks deserve to be addressed and accounted for. Though space is limited here, the more salient points seem to be as follows.

First, one of the motivations for advancing the "new" academic subject of critical thinking in the 1970's was a kind of intellectual rebellion against the academic subject of logic. By that time logic as typically taught at college seemed to many as having become nothing but the study of formal deductive systems. And philosophers generally acted in accord with this perception, usually sticking to the deductive half of their logic textbooks in teaching introductory logic courses, and usually thinking that there was just too much deductive logic to cover to have any time for the inductive parts of the logic textbooks. So, at the collegiate level introductory logic courses tended (and likely still tend) to be too deductive. There was even a standard joke of the time among philosophers that was likely to be badly misunderstood by those outside philosophy and perhaps by some within. The joke went like this: All logic textbooks have two parts, the deductive part wherein the fallacies are identified, and the inductive part wherein the fallacies are committed.

Second, with regard to remarks like the first one quoted above, one has to wonder whether any accomplished critical thinker would be willing claim something like this: "One particularly astute critical thinker of my acquaintance claims to know nothing about deductive logic and her reasonings are filled with deductive errors and omissions."

Third, one has to wonder what such remarks reveal about general perceptions of philosophy in the academic world. Philosophy by its very nature takes all matters of interest as its subject matter. The subject matter of philosophy ranges from ordinary everyday practical questions, to ethics, to law, to mathematics, to religion, to psychology, to anthropology, to literature, to artificial intelligence, to space-time-and-motion theory, to evolutionary theory, to theology, to politics, to thermodynamics, to quantum mechanics, to semantics, to rhetoric …and so on. So philosophy by its very nature is interdisciplinary. Of course philosophers also work in fields like epistemology that are mostly only studied by philosophers, but even here the study of knowledge and belief is done with a wide range of fields in view, otherwise the study would lack the generality required.

Fourth, critical thinking is not and cannot be persuasion. When we think critically, we are not trying to persuade ourselves or anyone else; instead, we are trying to figure things out. That is why it is possible to think critically about rhetoric or persuasion, just as we can think critically about anything else. When we consider, for example, how persuasion functions we are interested in getting clear, accurate, and precise information about it, and if we try to approach it rhetorically here, that can only get in the way of figuring out these things. We're not trying to persuade in thinking critically, we're trying to understand. So, although there is a critical thinking approach to rhetoric, there cannot be a rhetorical approach to critical thinking. Any attempt at such an approach will only confuse practicing persuasion with understanding it, or understanding anything else. But none of this is to argue that rhetoric and persuasion are somehow not important. Obviously they are important. We live much of our lives in relations with others wherein persuasion is essential, and if we are lucky we live with representative government, wherein persuasion is the mode of government. So persuasion is important, but that does not make it critical thinking, and no amount of rhetoric, no matter how skilled, can make it critical thinking. And a few further qualifications are in order here: Expressions of critical thinking are quite often persuasive, and these can be combined usefully and consistently with rhetorical and persuasive skills (skills that are a valuable part of any good education); and yet also, each of us knows that obfuscation, distraction, vagueness, outright lies, repetition, childhood training, money, advertising, favors, and many other things are often persuasive too. Persuasion is not and cannot be critical thinking.

Fifth, one has to wonder what can be the relevance or importance of the remarks quoted above in reviewing critical thinking materials. Relevant considerations include ones like these: Do the materials address what claims follow from (deductively or inductively) what other claims, and why? Do the materials address how deliberately, carefully, effectively, and efficiently to determine to accept, or reject, or suspend judgment on a wide variety of claims? Do the materials have to do with matters like clarity, accuracy, and precision? Do the materials deal with matters such as noncircularity, relevance, testability, etc. in the assessment of explanations? Do the materials deal with dispositions like curiosity, reflectiveness, open-mindedness, self-criticality, etc.?…and so on. Against the background of such questions, remarks like the ones quoted can be seen as at best trivial. Finally, it is my long, personal, direct experience that such remarks have more to do with "turf battles" in academia than they do with understanding critical thinking.6 And perhaps the time is right to drop altogether the term "critical thinking" and return to the ancient term, "logic." Perhaps future historians will find this to be a terminological curiosity worthy of description and explanation. Perhaps. But whether we call it logic, or critical thinking, or reasoning, or something else, the elimination or minimization or marginalization of either deductive or inductive reasoning is pernicious. It fails the nature of our claims, it fails the nature of our questions, it fails the nature of our speculations, it fails the nature of our arguments, it fails the nature of our explanations, it fails the nature of our languages, it fails the nature of much but not all of our thinking, it fails the nature of our intellectual products, and (likely more important than any of the rest) it fails the nature of our curiosity. In short, it fails us. For the nature we are talking about is in us, and that nature cannot be fooled, though we surely can.7

Summary of Findings for the Analysis

• A comparison of the summaries for the skills exams shows that using the Cornell Z and the California skills exams together "covers" the same skills as using all of the skills exams; this coverage cannot be achieved by using any other combination of skills exams, short of using them all.
  
• Only the California Critical Thinking Disposition Inventory attempts to measure CT dispositions.
  
• "Coverage" of critical thinking skills is extremely limited. Using all of the nationally available skills exams would address fewer than 35 of more than 250 basic critical thinking skills inventoried thus far (and there are surely more than these).
  
• "Coverage" of combinations of critical thinking skills is extremely limited. Again, using all of the nationally available skills exams would address fewer than 35 skills, and the few that are combined are repeatedly combined in very similar ways.
  
• Of six major categories of skills (1. Interpretation; 2. Analysis; 3. Evaluation; 4. Inference; 5. Explanation; 6. Self-Regulation), the skills exams attempt to measure only skills in the first three categories, with a preponderance of questions in the first two categories and evaluation of only simple fallacies and simple deductive logic. Yet the skills of the first two categories represent only the most elementary steps of critical thinking, not the core of critical thinking.8
  
• Skills are addressed only at the most rudimentary level. For example, most critical thinking skills require a critical thinker to supply an action (distinguish, develop, identify, interpret…etc.), not to select a response from a given list of four or five alternatives. Hence, no indication of competence in a skill tested is given for a student who responds correctly. Assuming that such tests contain questions of good quality,9 the only thing that such tests can indicate is that wrong answers show a lack of even the most rudimentary familiarity with the relevant skills.
  

Consequences of the Analysis

No standard multiple-choice exam10 can hope to capture the range or depth of CT basic skills, but that is not an argument against such testing. It is instead an argument in favor of understanding the results of any such testing: For only the specific skills tested, and assuming that the test questions are of good quality,8 such results can give an indication of familiarity with the skills addressed (a kind of minimal familiarity, below which there can be fairly high confidence that remediation for those skills is in order). This means that no such testing can give any indication of competence, under any standard dictionary understanding of that term. Furthermore,

(1) since most CT skills involve a "supply" response rather than a "select" response (that is, most CT skills involve initiating responses rather than making a selection from given alternatives); and,

(2) since most CT skills involve reflection on these "supply" responses themselves (thinking about thinking); and,

(3) since many CT skills involve originating thought and then carefully examining it, rather than making any response at all,

such testing, assuming good quality questions are used,8 only can provide indicators at the most rudimentary level. For these reasons any attempt to use such testing to grant any form of credit by exam, or to waive any CT requirement, or to make any positive claim about scores on such exams as indicators of competence is sheer folly. The better place for both the acquisition and the assessment of CT skills is the traditional classroom (with small class size,11 without multiple-choice testing, and with the requirement that students explain every answer to a teacher competent in CT skills who cares enough and has time enough to read and listen and respond to every response and every explanation). There are no short cuts. Nature cannot be fooled.

The Analysis

The Watson Glaser Critical Thinking Appraisal (Form A)

This exam consists of five "tests" of 16 questions for a total of 80 questions. The skills addressed by each test are as follows.

"A critical thinker is able to…"

Test 1

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• distinguish supporting, conflicting, and compatible claims D

• assess the relevance of claims to other claims D E

• evaluate claims and arguments in terms of criteria such as:

• consistency D E S

• relevance E S

• support

Test 2

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• assumptions (stated and unstated) D E S

• assess the relevance of claims to other claims D E

Test 3

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• assess the relevance of claims to other claims D E

• evaluate whether a deductive argument is valid or invalid (logical form) D E

Test 4

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• assess the relevance of claims to other claims D E

• evaluate whether an inductive argument is strong or weak D E

Test 5

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• assess the relevance of claims to other claims D E

• evaluate whether an inductive argument is strong or weak D E

  Summary for The Watson Glaser Critical Thinking Appraisal (Form A)

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• assumptions (stated and unstated) D E S

• distinguish supporting, conflicting, and compatible claims D

• assess the relevance of claims to other claims D E

• evaluate claims and arguments in terms of criteria such as:

• consistency D E S

• relevance E S

• support

• evaluate whether a deductive argument is valid or invalid (logical form) D E

• evaluate whether an inductive argument is strong or weak D E

The California Critical Thinking Skills Test

This exam consists of 34 questions grouped by tested skills as indicated below. The skills addressed are as follows.

"A critical thinker is able to…"

Questions 1-4

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• distinguish supporting, conflicting, and compatible claims, arguments, explanations, descriptions, representations, etc. D

• assess the relevance of claims to other claims D E

• evaluate whether a deductive argument is valid or invalid (logical form) D E

• evaluate whether an inductive argument is strong or weak D E

• evaluate claims and arguments in terms of criteria such as:

• consistency D E S

• relevance E S

• support

Questions 5-9

• recognize ambiguity and unclarity in claims, arguments, and explanations D E

• interpret and apply complex texts, instructions, illustrations, etc. D E

• distinguish supporting, conflicting, compatible, and equivalent claims, arguments, explanations, descriptions, representations, etc. D

Question 10

• recognize and clarify issues, claims, arguments, and explanations D E

• interpret and apply complex texts, instructions, illustrations, etc. D E

Questions 11-13

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• explanations D E S

• assumptions (stated and unstated) D E S

Questions 14-19

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether a deductive argument is valid or invalid (logical form) D

Questions 20-21

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether an inductive argument is strong or weak D E

Questions 22-23

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether a deductive argument is valid or invalid (logical form) D

Questions 24-27

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether an inductive argument is strong or weak D E

Questions 28

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether an inductive argument is strong or weak D E

• identify and avoid errors in reasoning: D

• informal fallacy:

• post hoc, ergo propter hoc (after that, therefor because of that)

Question 29

• interpret and apply complex texts, instructions, illustrations, etc. D E

• assess the relevance of claims to other claims D E

• discern whether pairs of claims are consistent, contrary, contradictory, or paradoxical D E

Question 30

• interpret and apply complex texts, instructions, illustrations, etc. D E

• identify and avoid errors in reasoning: D

• informal fallacy:

• begging the question

Questions 31-34

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether an inductive argument is strong or weak D E

• identify and avoid errors in reasoning: D

• informal fallacy:

• smokescreen/red herring/rationalizing

Summary for The California Critical Thinking Skills Test

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• explanations D E S

• assumptions (stated and unstated) D E S

• distinguish supporting, conflicting, and compatible claims, arguments, explanations, descriptions, representations, etc. D

• assess the relevance of claims to other claims D E

• evaluate whether a deductive argument is valid or invalid (logical form) D E

• evaluate whether an inductive argument is strong or weak D E

• evaluate claims and arguments in terms of criteria such as:

• consistency D E S

• relevance E S

• support

• recognize ambiguity and unclarity in claims, arguments, and explanations D E

• distinguish supporting, conflicting, compatible, and equivalent claims, arguments, explanations, descriptions, representations, etc. D

• recognize and clarify issues, claims, arguments, and explanations D E

• identify and avoid errors in reasoning: D

• informal fallacy:

• post hoc, ergo propter hoc (after that, therefor because of that)

• begging the question

• smokescreen/red herring/rationalizing

Cornell Critical Thinking Test, Level X

This exam consists of 76 questions addressing CT skills as indicated below. The test designers indicate that this test is designed for grade levels 4-14. The skills addressed are as follows.

"A critical thinker is able to…"

Questions 3-25, 48, 50

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether an inductive argument is strong or weak D E

Questions 52-65, 67-76

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether a deductive argument is valid or invalid (logical form) D

Questions 27-50

• interpret and apply complex texts, instructions, illustrations, etc. D E

• assess the relevance of claims to other claims, and to questions, descriptions, representations, procedures, information, directives, rules, principles, etc. D E

• evaluate whether an inductive argument is strong or weak D E

Questions 67-76

• interpret and apply complex texts, instructions, illustrations, etc. D E

• distinguish:

• assumptions (stated and unstated) D E S

• assess the relevance of claims to other claims, and to questions, descriptions, representations, procedures, information, directives, rules, principles, etc. D E

Summary for Cornell Critical Thinking Test, Level X

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether an inductive argument is strong or weak D E

• evaluate whether a deductive argument is valid or invalid (logical form) D

• assess the relevance of claims to other claims, and to questions, descriptions, representations, procedures, information, directives, rules, principles, etc. D E

• distinguish:

• assumptions (stated and unstated) D E S

Cornell Critical Thinking Test, Level Z

This exam consists of 52 questions addressing CT skills as indicated below. The test designers indicate that this test is designed for "advanced and gifted high school students, college students, and other adults." (Test Manual p3) The skills addressed are as follows.

"A critical thinker is able to…"

Questions 17, 26-42

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether an inductive argument is strong or weak D E

Questions 1-10, 39-52

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether a deductive argument is valid or invalid (logical form) D

Questions 22-25

• interpret and apply complex texts, instructions, illustrations, etc. D E

• assess the relevance of claims to other claims, and to questions, descriptions, representations, procedures, information, directives, rules, principles, etc. D E 3-1

• evaluate whether an inductive argument is strong or weak D E

Questions 43-52

• interpret and apply complex texts, instructions, illustrations, etc. D E

• distinguish:

• assumptions (stated and unstated) D E S

• assess the relevance of claims to other claims, and to questions, descriptions, representations, procedures, information, directives, rules, principles, etc. D E

Questions 11-21

• interpret and apply complex texts, instructions, illustrations, etc. D E

• identify and avoid errors in reasoning: D

• informal fallacies:

• equivocation

• appeal to ridicule/sarcasm

• false dilemma/false dichotomy fallacy (line drawing fallacy, perfectionist fallacy)

• hasty generalization

Questions 43-46

• interpret and apply complex texts, instructions, illustrations, etc. D E

• distinguish supporting, conflicting, compatible, and equivalent claims, arguments, explanations, descriptions, representations, etc. D

Summary for Cornell Critical Thinking Test, Level Z

• interpret and apply complex texts, instructions, illustrations, etc. D E

• evaluate whether an inductive argument is strong or weak D E

• evaluate whether a deductive argument is valid or invalid (logical form) D

• assess the relevance of claims to other claims D E

• distinguish:

• assumptions (stated and unstated) D E S

• identify and avoid errors in reasoning: D

• informal fallacies:

• equivocation

• appeal to ridicule/sarcasm

• false dilemma/false dichotomy fallacy (line drawing fallacy, perfectionist fallacy)

• hasty generalization

• distinguish supporting, conflicting, compatible, and equivalent claims, arguments, explanations, descriptions, representations, etc. D

Summary of Skills Addressed by All Nationally Available CT Skills Exams

• interpret, and apply complex texts, instructions D E

• distinguish:

• conclusions D E S

• premises (reasons) D E S

• explanations D E S

• implications S

• assumptions (stated and unstated) D E S

• distinguish supporting, conflicting, and compatible claims, arguments, explanations, descriptions, representations, etc. D

• assess the relevance of claims to other claims D E

• evaluate whether a deductive argument is valid or invalid (logical form) D E

• evaluate whether an inductive argument is strong or weak D E

• recognize ambiguity and unclarity in claims, arguments, and explanations D E

• evaluate claims and arguments in terms of criteria such as:

• consistency D E S

• relevance E S

• support

• recognize and clarify issues, claims, arguments, and explanations D E

• identify and avoid errors in reasoning: D

• informal fallacy:

• post hoc, ergo propter hoc (after that, therefor because of that)

• begging the question

• smokescreen/red herring/rationalizing

• equivocation

• appeal to ridicule/sarcasm

• false dilemma/false dichotomy fallacy (line drawing fallacy, perfectionist fallacy)

• hasty generalization

The California Critical Thinking Disposition Inventory

This exam consists of 75 questions addressing CT dispositional skills as indicated below. The exam's terminology for the dispositions is shown below inside braces {}. The skills addressed are as follows.

"A critical thinker is able to and has a propensity to …"

QUESTIONS 5, 12, 19, 23, 35, 39, 43, 50, 62, 70, 72, 75 {truth-seeking}

• exhibit truth-seeking candor D E

Questions 1, 8, 13, 20, 24, 30, 36, 41, 45, 48, 64, 73 {open mindedness}

• exhibit open-minded empathy D E S

Questions 2, 15, 26, 34, 44, 47, 51, 55, 59, 65 {inquisitiveness}

• exhibit curiosity D E S

Questions 4, 9, 17, 22, 25, 29, 33, 37, 58, 68, 74 {systematicity}

• exhibit clarity (including organization) D E S

Questions 3, 7, 11, 14, 28, 32, 53, 61, 67, 71
{maturity}

• exhibit skepticism D

• exhibit the settled disposition to employ and regularly re-apply CT skills and virtues (self-direction, self-correction, self-regulation, self-confidence, maturity) D E

Questions 10, 16, 18, 27, 40, 46, 49, 52, 56 {confidence}

• exhibit self-criticality D

• exhibit integrity S

Questions 6, 21, 31, 38, 42, 54, 57, 60, 63, 66, 69 {analyticity}

• exhibit accuracy D E S

• exhibit precision D E S

• exhibit consistency D E S

References

Crosschecked Models of Critical Thinking:

D the Delphi model: Facione, Peter A. Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction, "The Delphi Report", Milbrae, California: The California Academic Press, 1990.

E the U. S. National Educational Goals model: Click, Benjamin A. L. & Hoffman, Steven & Jones, Elizabeth & Moore, Lynne M. & Ratcliff, Gary & Tibbitts, Stacy, National Educational Goals, National Assessment of College Student Learning: Identifying College Graduates' Essential Skills in Writing, Speech and Listening, and Critical Thinking, Final Round Consensus of Faculty, Employers, and Policymakers, United States Department of Education, 1996.

S the Sonoma model: Paul, Richard, et al, Center for Critical Thinking, Sonoma State University, Critical Thinking: Basic Theory & Instructional Structures, Rohnert Park, California: The Foundation for Critical Thinking, 1998.

General References:

Facione, Peter A. & Facione, Noreen C. & Gainen, Joanne & Sanchez, Carol A "The Disposition Toward Critical Thinking" The Journal of General Education, Vol.44, No.1, p1-25, 1995.

Fawkes, Don, "On Method" American Philosophical Association Newsletter on Teaching Philosophy, Vol.95, No.1. December, 1995.

Fisher, Alec & Scriven, Michael, Critical Thinking Its Definition and Assessment, Point Reyes, California: Edgepress, 1997.

Flew, Antony, Thinking Straight, Buffalo, New York: Prometheus Books, 1977.

Mill, John Stuart, A System of Logic, 8th ed. 1874 (first ed. London: Blackwell, 1859), reprint, New York: Harper Brothers, 1961.

Moore, Brook, and Parker, Richard, Critical Thinking, 5th ed. Mountain View, California: Mayfield Publishing Company, 1998.

Murphy, Arthur E. The Uses of Reason, Westport, Connecticut: Greenwood Press, 1972.

Paul, Richard & Elder, Linda & Bartel, Tedd, California Teacher Preparation for Instruction in Critical Thinking: Research Findings and Policy Recommendations, Sacramento, California: California Commission on Teacher Credentialing, 1997.

Paul, Richard, Critical Thinking, How To Prepare Students For A Rapidly Changing World, Santa Rosa, CA: Foundation For Critical Thinking, 1995a.

Scriven, Michael, The Logic of Evaluation, Point Reyes, California: Edgepress, 1981.

Scriven, Michael, Reasoning, New York: McGraw-Hill, 1976.

Toulman, Stephen, The Uses of Argument, New York: Cambridge University Press, 1976.

Endnotes

1. Thanks to my colleagues Dan Flage and Bill O'Meara at James Madison University and Henry Byerly, emeritus at the University of Arizona, for comments and suggestions on earlier drafts. Thanks also to the editors and blind reviewers of the APA Newsletter on Teaching Philosophy for a very thorough and helpful review.

2. Other more general aptitude or achievement exams (e.g. ACT, SAT, various "Rising Junior" exams, etc.) may purport to test CT skills, but this paper addresses exams specifically designated as tests of critical thinking. Also, each of the tests addressed here is in current use for the collegiate level of instruction, though most purport to apply to some other levels as well.

3. In fact, in the twentieth century consequences so subtle and complex were reached (by persons with names like Gödel and Turing and Church and Skidmore) that only those who have completed advanced study can understand the conclusions, let alone the arguments. And these consequences not only demonstrate startling limits on deductive reasoning and on all of mathematics, but they also led quite directly to the development of the computers by which this essay was written and transmitted and printed.

4. Most, but not all beginning students "take a while" here, and professors in many disciplines experience that such as this is so for students in many other subjects. Natural human talents are enormously diverse in range, diversity, sophistication, and potential. Teachers of logic encounter a Gödel now and then, just as teachers of music encounter a Mozart. And alas, despite all their efforts professors sometimes miss a Mozart or a Gödel who is there before them.

5. Two claims are contrary if and only if they both cannot be true, but they can both be false under the same conditions. Two claims are contradictory if and only if they both cannot be true, and they both cannot be false under the same conditions.

6. But where turf battles are what are being pursued, one relevant question is whether it is appropriate that there are required introductory writing and communication courses at college and required writing and communication courses in K through 12, but no required philosophy courses at virtually any college, and virtually none in K through 12?

7. The nature, for example, of the distinction between contradictory and contrary claims cannot be avoided by any ruse, but we surely can be fooled by ruses about it, whether they are intended or not.

8. Fisher and Scriven (1997 p 87-114) do not consider skills in the first two categories to be critical thinking skills, essentially considering them as generally preliminary to critical thinking. This is primarily a matter of emphasis, but it does serve to highlight just how little the skills exams attempt to measure. Please see also the discussion of this matter herein under the heading Some Qualifications and Related Matters, (3) Critical thinking and critical apprehension.

9. Preliminary examination of several of these exams by this author and several colleagues calls this assumption into question, but that is a topic for another day.

10. Fisher and Scriven (1997 p 173-203) offer a nonstandard testing procedure that may hold promise.

11. It may be appropriate to quantify the meaning of "small class size." Perhaps it would be best simply to note that at this writing the fairly widely publicized national goal of the U. S. Department of Education for K through 12 education is a class size of 18. Is collegiate education less demanding?

Previous Article | Index | Next Article