A Taxonomy of Intelligence Variables ^[1]

Science is organized knowledge. — Herbert Spencer [2] 

Aristotle may be the father of scientific classification, but it was biologist Carolus Linnaeus who introduced the first formal taxonomy—kingdom, class, order, genera, and species—in his Systema Naturae in 1735. By codifying the naming conventions in biology, Linnaeus’s work provided a reference point for future discoveries. Darwin’s development of an evolutionary theory, for example, benefited greatly from Linnaeus’s creation of a hierarchical grouping of related organisms. The Systema Naturae taxonomy was not a fixed product but rather a living document. Linnaeus himself revised it through 10 editions, and later biologists have continued to modify it.[3]

In response to new discoveries and the development of new research methods in other domains, taxonomies were created to help organize those disciplines and to assist researchers in identifying variables that required additional study. The development of specific taxonomies—from highly structured systems, such as the periodic table of chemical elements, to less structured approaches, such as Bloom’s Taxonomy[4]—is a key step in organizing knowledge and furthering the growth of individual disciplines. A taxonomy differentiates domains by specifying the scope of inquiry, codifying naming conventions, identifying areas of interest, helping to set research priorities, and often leading to new theories. Taxonomies are signposts, indicating what is known and what has yet to be discovered.

This chapter, to which more than 100 individuals contributed their time and advice, proposes a taxonomy for the field of intelligence. It is my hope that the resulting organized listing of variables will help practitioners strengthen their understanding of the analytic process and point them in directions that need additional attention.

Intelligence Analysis

We could have talked about the science of intelligence, but . . . the science of intelligence is yet to be invented.— Charles Allen[5] 

Developing an intelligence taxonomy is complicated by the fact that the literature in the field is episodic and reflects specialized areas of concern. Perhaps it is best to begin with what appears to be a key distinction between general analysis and intelligence analysis, that of solving a problem in the public domain, and solving a problem in a private or secret domain.

Ronald Garst articulates two arguments that are used to support this distinction: intelligence analysis is more time sensitive than analysis in other domains and it deals with information that intentionally may be deceptive.[6] The notion that intelligence is uniquely time sensitive is questionable, however. Intelligence is not the only domain where time constraints can force decisions to be made before data are complete. Time is always a key variable, whether one is in an operating room or in a cockpit. To be sure, intelligence is a life and death profession, but so are medicine and mass transportation. In each instance, failure can mean casualties.

Garst’s point about intentional deception is more germane. With the possible exception of business and financial markets, analysts in other fields seldom deal with intentional deception. As discussed in Chapter One, Michael Warner makes a good case for secrecy being the primary variable distinguishing intelligence from other such activities.[7] He argues that the behavior of the subject of intelligence changes if the subject is aware of being observed or analyzed. As discussed earlier, Warner’s argument is supported by a long history of psychological research, beginning with an experimental program between 1927 and 1930 at Western Electric’s Hawthorne Works in Chicago.[8]

Intentional deception can occur outside intelligence—in connection with certain law enforcement functions, for example—but most of the professional literature treats this as the exception rather than the rule. In the case of intelligence analysis, deception is the rule; the validity of the data is always in doubt. Moreover, intelligence analysts are specifically trained to take deception into account as part of the analytic process—to look for anomalies and outliers instead of focusing on the central tendencies of distribution.

The taxonomy being developed here requires a definition of intelligence analysis that is specific to the field. Intelligence pioneer Sherman Kent, who saw intelligence as a “special category of knowledge,” laid the foundation for understanding the activities inherent in intelligence analysis by demonstrating that the analytic process itself was subject to being analyzed.[9] Kent’s approach to analysis was to reduce the process to smaller functional components for individual study.[10] For example, he described intelligence analysis as having a basic descriptive element, a current reporting element, and an estimative element.

Following suit, other authors focused attention on the process or methodological elements of intelligence analysis. In Intelligence Research Methodology, Jerome Clauser and Sandra Weir followed Kent’s three functional areas and went on to describe basic research foundations and the inductive and deductive models for performing intelligence analysis.[11] Garst’s Handbook of Intelligence Analysis contains less background in basic research methods than Clauser and Weir’s book, but it is more focused on the intelligence cycle.[12]

Bruce Berkowitz and Allan Goodman highlight the process of strategic intelligence and define intelligence analysis as: “[T]he process of evaluating and transforming raw data into descriptions, explanations, and conclusions for intelligence consumers.”[13] Lisa Krizan, too, focuses on process. She writes that, “At the very least, analysis should fully describe the phenomenon under study, accounting for as many relevant variables as possible. At the next higher level of analysis, a thorough explanation of the phenomenon is obtained, through interpretation of the significance and effects of its elements on the whole.”[14] In addition, several authors have written about individual analytic approaches.[15]

Although the referenced works focus on methods and techniques, they do not suggest that analysis is limited to these devices. The view that analysis is both a process and a collection of specific techniques is explicit in the above definitions. Analysis is seen as an action that incorporates a variety of tools to solve a problem. Different analytic methods have something to offer different analytic tasks.

Although largely implicit in the above definitions, analysis is also seen as a product of cognition, and some authors directly link the two. Robert Mathams defines analysis as: “[T]he breaking down of a large problem into a number of smaller problems and performing mental operations on the data in order to arrive at a conclusion or generalization.”[16] Avi Shlaim writes: “Since the facts do not speak for themselves but need to be interpreted, it is inevitable that the individual human propensities of an intelligence officer will enter into the process of evaluation.”[17] Yet others describe analysis as a process whereby: “[I]nformation is compared and collated with other data, and conclusions that also incorporate the memory and judgment of the intelligence analyst are derived from it.”[18]

Several authors make the case that analysis is not just a product of cognition but is itself a cognitive process. J. R. Thompson and colleagues write that “[I]ntelligence analysis is an internal, concept-driven activity rather than an external data-driven activity.”[19] In his Psychology of Intelligence Analysis, Heuer observes: “Intelligence analysis is fundamentally a mental process, but understanding this process is hindered by the lack of conscious awareness of the workings of our own minds.”[20] Ephraim Kam comments: “The process of intelligence analysis and assessment is a very personal one. There is no agreed-upon analytical schema, and the analyst must primarily use his belief system to make assumptions and interpret information. His assumptions are usually implicit rather than explicit and may not be apparent even to him.”[21]

These definitions reflect the other end of the spectrum from those concerned with tools and techniques. They suggest that the analytic process is a construction of the human mind and is significantly different from individual to individual or group to group. Certainly, Kam goes farthest along this path, but even he does not suggest that one forgo tools; rather, he says that the process of choosing the tool is governed by cognition as well.

Recognizing that the scope of intelligence analysis is so broad that it includes not only methods but also the cognitive process is a significant step. Viewing analysis as a cognitive process opens the door to a complex array of variables. The psychology of the individual analyst must be considered, along with individual analytic tools. In the broadest sense, this means not merely understanding the individual psyche but also understanding the variables that interact with that psyche. In other words, intelligence analysis is the socio-cognitive process,[22] occurring within a secret domain, by which a collection of methods is used to reduce a complex issue to a set of simpler issues.

Developing the Taxonomy

The first step of science is to know one thing from another. This knowledge consists in their specific distinctions; but in order that it may be fixed and permanent distinct names must be given to different things, and those names must be recorded and remembered.

Carolus Linnaeus

My research was designed to isolate variables that affect the analytic process. The resulting taxonomy is meant to establish parameters and to stimulate dialogue in order to develop refinements. Although a hierarchic list is artificial and rigid, it is a first step in clarifying areas for future research. The actual variables are considerably more fluid and interconnected than such a structure suggests. Once the individual elements are refined through challenges in the literature, they might be better represented by a link or web diagram.[23]

To create this intelligence analysis taxonomy, I used Alexander Ervin’s applied anthropological approach, which employs multiple data collection methods to triangulate results.[24] I also drew on Robert White’s mental workload model, David Meister’s behavioral model, and the cognitive process model of Gary Klein and his colleagues.[25] Each model focuses on a different aspect of human performance: White’s examines the actual task and task requirements; Meister’s looks at the behavior of individuals performing a task; and Klein’s uses verbal protocols to identify the cognitive processes of individuals performing a task.

Surveying the literature. My research began with a review of the literature, both for background information and for the identification of variables. The intelligence literature produced by academics and practitioners tends to be episodic, or case-based. This is not unique to the field of intelligence. A number of disciplines—medicine, business, and law, for example—are also case-based. Many of the texts were general or theoretical rather than episodic. Again, this is not an uncommon phenomenon. The review yielded 2,432 case studies, journal articles, technical reports, transcripts of public speeches, and books related to the topic. I then narrowed the list to 374 pertinent texts on which a taxonomy of intelligence analysis could be built, and I analyzed them to identify individual variables and categories of variables that affect intelligence analysis.[26]

Using a methodology known as “Q-Sort,” by which variables are sorted and categorized according to type, I read each text and recorded the variables that each author identified.[27] These variables were then sorted by similarity into groups. Four broad categories of analytic variables emerged from this process.[28]

Refining the prototype. Next, I used the preliminary taxonomy derived from my reading of the literature to structure interviews with 51 substantive experts and 39 intelligence novices. In tandem, I conducted two focus group sessions, with five individuals in each group. As a result of the interviews and focus group discussions, I added some variables to each category, moved some to different categories, and removed some that appeared redundant.

Testing in a controlled setting. Finally, to compare the taxonomy with specific analytic behaviors, I watched participants in a controlled intelligence analysis–training environment. Trainees were given information on specific cases and directed to use various methods to analyze the situations and to generate final products. During the training exercises, the verbal and physical behavior of individuals and groups were observed and compared with the taxonomic model. I participated in a number of the exercises myself to gain a better perspective. This process corroborated most of the recommendations that had been made by the experts and novices and also yielded additional variables for two of the categories.

The resulting taxonomy is purely descriptive. It is not intended to demonstrate the weight or importance of each variable or category. That is, the listing is not sufficient to predict the effect of any one variable on human performance. The intention of the enumeration is to provide a framework for aggregating existing data and to create a foundation for future experimentation. Once the variables have been identified and previous findings have been aggregated, it is reasonable to consider experimental methods that would isolate and control individual variables and, in time, indicate sources of error and potential remediation

Systemic Variables

The column of Systemic Variables incorporates items that affect both an intelligence organization and the analytic environment. Organizational variables encompass the structure of the intelligence organization; leadership, management, and management practices; history and traditions; the working culture, social practices within the organization, and work taboos; and organizational demographics. They also include internal politics, the hierarchical reporting structure, and material and human resources. Industrial and organizational psychology, sociology, and management studies in business have brought attention to the importance of organizational behavior and its effect on individual work habits and practices. The works of Allison, Berkowitz and Goodman, Elkins, Ford, Godson, and Richelson, among others, examine in general the organizational aspects of intelligence.[29]

Systemic Variables [PDF 69.8KB*]

The Systemic Variables category also focuses on environmental variables. These include such external influences on the organization as consumer needs and requirements, time limitations, and methods for using the information; and the consumer’s organization, political constraints, and security issues. The works of Betts, Hulnick, Hunt, Kam, and Laqueur address the environmental and consumer issues that affect intelligence analysis.[30]  Case studies that touch on various systemic variables include: Allison, on the Cuban missile crisis; Betts, on surprise attacks; Kirkpatrick, on World War II tactical intelligence operations; Shiels, on government failures; Wirtz, on the Tet offensive in Vietnam; and Wohlstetter, on Pearl Harbor.[31

Systematic Variables

The Systematic Variables are those that affect the process of analysis itself. They include the user’s specific requirements, how the information was acquired, the information’s reliability and validity, how the information is stored, the prescribed methods for analyzing and processing the information, specific strategies for making decisions about the information, and the methods used to report the information to consumers.

Systematic Variables [PDF 69.3KB*]

A number of authors have written about the analytic tools and techniques used in intelligence, among them Clauser and Weir, on intelligence research methods; Jones, on analytic techniques; and Heuer, on alternative competing hypotheses. Comparatively little work has been done comparing structured techniques to intuition. Robert Folker’s work is one of the exceptions; it compares the effectiveness of a modified form of alternative competing hypotheses with intuition in a controlled experimental design. His study is unique in the field and demonstrates that experimental methods are possible. Geraldine Krotow’s research, on the other hand, looks at differing forms of cognitive feedback during the analytic process and makes recommendations to improve intelligence decisionmaking.[32]

Idiosyncratic Variables

Variables in the third column are those that influence individuals and their analytic performance. These include the sum of life experiences and enculturation—familial, cultural, ethnic, religious, linguistic, and political affiliations—that identify an individual as a member of a group. I have used the German word Weltanschauung (customarily rendered in English as “world view”) to denote this concept. These idiosyncratic variables also encompass such psychological factors as biases, personality profiles, cognitive styles and processing, cognitive loads,[33] expertise, approach to problem-solving, decisionmaking style, and reaction to stress. Finally, there are such domain variables as education, training, and the readiness to apply knowledge, skills, and abilities to the task at hand.

Idiosyncratic Variables [PDF 66.7KB*]

The relevant psychological literature is extensive. Amos Tversky and Daniel Kahneman began to examine psychological biases in the early 1970s.[34] Their work has found its way into the intelligence literature through Butterfield, Davis, Goldgeier, and Heuer, among others.[35] Decisionmaking and problem-solving have been studied since the early 1920s, and these topics are reflected in Heuer’s work as well.[36] Personality-profiling, too, is well understood and has had an impact on recent intelligence practices and theory.[37]

Other well-researched areas, however, have yet to be studied in the context of intelligence. Acculturation and enculturation, educational factors, and training strategies, for example, may yet yield interesting results and insights into the field of intelligence.[38]

Communicative Variables

Communicative Variables [PDF 70.2KB*]

The fourth category contains variables that affect interaction within and among groups. Because communication is the vital link within the system—among processes and among individuals—this group of variables logically could be included in each of the other three categories. Its broad relevance, however, makes it seem reasonable to isolate it as a distinct area of variability. The Communicative Variables include formal and informal communications within an organization (from products to e-mails), among organizations, and between individuals and the social networks they create. In his essay on estimative probability, Kent highlights this area by describing the difficulty that producers of intelligence have in communicating the likelihood of an event to their consumers.[39] In addition to addressing organizational issues, case studies by Wohlstetter and others touch on communication and social networks and the impact that communication has on the analytic process.[40] This is an area that could benefit from additional study.

Conclusion

There is rarely any doubt that the unconscious reasons for practicing a custom or sharing a belief are remote from the reasons given to justify them.

—Claude Levi-Strauss[41]

As it is now practiced, intelligence analysis is art, tradecraft, and science. There are specific tools and techniques to help perform the tasks, but, in the end, it is left to individuals to use their best judgment in making decisions. This is not to say that science is not a part of intelligence analysis. Science is born of organized knowledge, and organizing knowledge requires effort and time. The work on this taxonomy is intended to help that process by sparking discussion, identifying areas where research exists and ought to be incorporated into the organizational knowledge of intelligence, and identifying areas where not enough research has been performed.

There are a number of parallels in the field of medicine, which, like intelligence, is art, tradecraft, and science. To solve problems, practitioners are trusted to use their best judgment by drawing on their expertise. What is important to remember is that there are numerous basic sciences driving medical practice. Biology, chemistry, physics, and all of the subspecialties blend together to create the medical sciences, the foundation on which modern medicine rests. The practice of medicine has been revolutionized by the sciences that underpin its workings.

Intelligence analysis has not experienced that revolution. Unlike medicine, the basic sciences that underpin intelligence are the human sciences, which are considerably more multivariate and more difficult to control. Because of these factors, it is a more complex task to measure “progress” in the human sciences. Even so, there are numerous domains from which intelligence may borrow. Organizational behavior is better understood today than ever before. Problem-solving and decisionmaking have been researched since the 1920s. Structural anthropology addresses many of the enculturation and identity issues that affect individual behavior. Cognitive scientists are building models that can be tested in experimental conditions and used for developing new tools and techniques. Sociology and social theory have much to offer in studying social networks and communication.

The organization of knowledge in intelligence is not a small task, but I believe that the effort should be undertaken for the betterment of the profession. The taxonomy proposed here could serve as a springboard for a number of innovative projects, for example: development of a research matrix that identifies what is known and how that information may be of use in intelligence analysis, setting a research agenda in areas of intelligence that have been insufficiently studied, application of research from other domains to develop additional training and education programs for analysts, creation of a database of lessons learned and best practices to build a foundation for an electronic performance support system, integration of those findings into new analytic tools and techniques, and development of a networked architecture for collaborative problem-solving and forecasting. It is my hope that this taxonomy will help intelligence practitioners take steps in some of these new directions.

Footnotes:

[1] A version of this chapter, “Developing a Taxonomy of Intelligence Analysis Variables,” originally appeared in Studies in Intelligence 47, no. 3 (2003): 61–71.

[2] Herbert Spencer’s The Study of Sociology, published in 1874, set the stage for the emergence of sociology as a discipline.

[3]Ernst Haeckel introduced phylum to include related classes and family to include related genera in 1866. The Linnaeus taxonomy is currently being revised to accommodate genomic mapping data.

[4] See Benjamin S. Bloom, Taxonomy of Educational Objectives. Bloom’s taxonomy is a classification of levels of intellectual behavior in learning, including knowledge, comprehension, application, analysis, synthesis, and evaluation.

[5] Comment made by the Associate Director of Central Intelligence for Collection at a public seminar on intelligence at Harvard University, spring 2000.

[6] Ronald Garst, A Handbook of Intelligence Analysis.

[7] Michael Warner.

[8] The Hawthorne Effect. See footnote 5 in Chapter Two.

[9] Sherman Kent, Strategic Intelligence for American World Policy.

[10] See Chapter Seven for a fuller discussion of this approach, now usually referred to as meta-analysis.

[11] Jerome K. Clauser and Sandra M. Weir, Intelligence Research Methodology.

[12]See also: Morgan Jones, The Thinker’s Toolkit. Jones’s book is a popular version of the work of Garst and Clauser and Weir in that it describes a collection of analytic methods and techniques for problem-solving; however, the methods are not necessarily specific to intelligence.

[13] Bruce D. Berkowitz and Allan E. Goodman, Strategic Intelligence for American National Security, 85. See Chapter Four for more on the intelligence cycle.

[14] Lisa Krizan, Intelligence Essentials for Everyone.

[15] See the apprendix for a listing of the literature.

[16] Robert Mathams, “The Intelligence Analyst’s Notebook.”

[17] Avi Shlaim, “Failures in National Intelligence Estimates: The Case of the Yom Kippur War.”

[18] John Quirk et al., The Central Intelligence Agency: A Photographic History.

[19] J. R. Thompson, R. Hopf-Weichel, and R. Geiselman, The Cognitive Bases of Intelligence Analysis.

[20] Richards J. Heuer, Jr., Psychology of Intelligence Analysis.

[21] Ephraim Kam, Surprise Attack. The Victim’s Perspective, 120

[22] That is, analysis does not occur in a vacuum. It is socially constructed. See Lev Vygotsky, Mind and Society.

[23] See Chapter Four for Judith Meister Johnston’s systems analysis approach to describing the fluidity of the intelligence process.

[24] Alexander Ervin, Applied Anthropology. See Chapter One, note 4 for a definition of triangulation.

[25] Robert White, Task Analysis Methods; David Meister, Behavioral Analysis and Measurement Methods; G. Klein, R. Calderwood, and A. Clinton-Cirocco, Rapid Decision Making on the Fire Ground.

[26] A copy of the list and search criteria is available from the author.

[27] William Stephenson, The Study of Behavior: Q-Technique and its Methodology. See Chapter Eleven for additional information on this methodology.

[28] I would like to credit Dr. Forrest Frank of the Institute for Defense Analyses for his suggestions regarding the naming convention for the categories of variables in the accompanying chart.

[29] Graham T. Allison, Essence of Decision; Bruce D. Berkowitz and Allan E. Goodman, Best Truth; Dan Elkins, An Intelligence Resource Manager’s Guide; Harold Ford, Estimative Intelligence; Roy Godson, Comparing Foreign Intelligence; Jeffrey Richelson, The U.S. Intelligence Community.

[30] Richard K. Betts, “Policy-makers and Intelligence Analysts: Love, Hate or Indifference”; Arthur S. Hulnick, “The Intelligence Producer-Policy Consumer Linkage: A Theoretical Approach”; David Hunt, Complexity and Planning in the 21st Century; Kam, Surprise Attack; Walter A. Laqueur, The Uses and Limits of Intelligence.

[31] Allison; Richard K. Betts, Surprise Attack; Lyman B. Kirkpatrick, Jr., Captains Without Eyes: Intelligence Failures in World War II; Frederick L. Shiels, Preventable Disasters: Why Governments Fail; James J. Wirtz, The Tet Offensive: Intelligence Failure in War; Roberta Wohlstetter, Pearl Harbor: Warning and Decision.

[32] Geraldine Krotow, The Impact of Cognitive Feedback on the Performance of Intelligence Analysts, 176.

[33] “Cognitive loads” are the amount/number of cognitive tasks weighed against available cognitive processing power.

[34] Amos Tversky and Daniel Kahneman, “The Belief in the ‘Law of Small Numbers’” and “Judgment Under Uncertainty: Heuristics and Biases.”

[35]Alexander Butterfield, The Accuracy of Intelligence Assessment; Jack Davis, “Combating Mindset”; James M. Goldgeier, “Psychology and Security”; Heuer.

[36] Frank H. Knight, Risk, Uncertainty and Profit.

[37] Caroline Ziemke, Philippe Loustaunau, and Amy Alrich, Strategic Personality and the Effectiveness of Nuclear Deterrence.

[38] Acculturation is the cultural change that occurs in response to extended firsthand contact between two or more previously autonomous groups. It can result in cultural changes in groups as well as individuals.

[39] Sherman Kent, “Words of Estimative Probability.”

[40] Wohlstetter.

[41] Claude Levi-Strauss wrote Structural Anthropology in 1958, setting the stage for structuralism to emerge as an analytic interpretive method. Broadly, structuralism seeks to explore the inter-relationships (the “structures”) through which meaning is produced within a culture. This meaning, according to structural theory, is produced and reproduced through various practices, phenomena, and activities that serve as systems of “signification.” A structuralist studies activities as diverse as food preparation and serving rituals, religious rites, games, literary and non-literary texts, and forms of entertainment to discover the ways in which cultural significance develops.

* Adobe® Reader® is needed to view Adobe PDF files. If you don't already have Adobe Reader installed, you may download the current version at www.adobe.com (opens in a new window). [external link disclaimer]