Introductio
In August 2001, I accepted a Director of Central Intelligence postdoctoral research fellowship with the Center for the Study of Intelligence (CSI) at the Central Intelligence Agency. The purpose of the fellowship, which was to begin in September and last for two years, was to identify and describe conditions and variables that negatively affect intelligence analysis. During that time, I was to investigate analytic culture, methodology, error, and failure within the Intelligence Community using an applied anthropological methodology that would include interviews (thus far, there have been 489), direct and participant observation, and focus groups.
I began work on this project four days after the attack of 11 September, and its profound effect on the professionals in the Intelligence Community was clearly apparent. As a whole, the people I interviewed and observed were patriotic without pageantry or fanfare, intelligent, hard working, proud of their profession, and angry. They were angry about the attack and that the militant Islamic insurgency about which they had been warning policymakers for years had murdered close to 3,000 people in the United States itself. There was also a sense of guilt that the attack had happened on their watch and that they had not been able to stop it.
Having occurred under the dark shadow of that attack, this study has no comparable baseline against which its results could be tested, and it is difficult to identify biases that might exist in these data as a result of 11 September. In some ways, post-9/11 data may be questionable. For example, angry people may have an ax to grind or an agenda to push and may not give the most reliable interviews. Yet, in other ways, post-9/11 data may be more accurate. When people become angry enough, they tend to blurt out the truth—or, at least, their perception of the truth. The people I encountered were, in my judgment, very open and honest; and this, too, may be attributable to 9/11. In any case, that event is now part of the culture of the Intelligence Community, and that includes whatever consequences or biases resulted from it.
Background
The opportunity to do this research presented itself, at least in part, as a result of my participation in a multiyear research program on medical error and failure for the Defense Advanced Research Projects Agency (DARPA).[1] The DARPA research focused on team and individual error in minimally invasive or laparoscopic surgical procedures. This research revealed that individual errors were cognitive rather than purely psychomotor or skill-based. For example, some surgeons had trouble navigating three-dimensional anatomical space using the existing laparoscopic technology, with the result that these surgeons would identify anatomical structures incorrectly and perform a surgical procedure on the wrong body part.
Other individual errors were discovered during the DARPA studies, but, for the most part, these were spatial navigation and recognition problems for which there were technological solutions. Team errors, unlike individual errors, proved to be more challenging. The formal and informal hierarchical structures of operating rooms did not lend themselves to certain performance interventions. Generally, junior surgical staff and support personnel were not willing to confront a senior staff member who was committing, or was about to commit, an error.
The culture of the operating room, coupled with the social and career structure of the surgical profession, created barriers to certain kinds of communication. For a surgical resident to inform a senior surgeon in front of the entire operating room staff that he was about to cut the wrong organ could result in career “suicide.” Such a confrontation could have been perceived by the senior surgeon as a form of mutiny against his authority and expertise and a challenge to the social order of the operating room. Although not universal, this taboo is much more common than surgeons would care to admit. Unlike individual errors, purely technological solutions were of little value in trying to solve team errors in a surgical environment.
The DARPA surgical research was followed up by a multiyear study of individual and team performance of astronauts at the National Aeronautics and Space Administration’s (NASA) Johnson Space Center. Results of the NASA study, also sponsored by DARPA, were similar to the surgical study with regard to team interactions. Although, on the face of it, teams of astronauts were composed of peers, a social distinction nevertheless existed between commander, pilots, and mission specialists.
As with surgery, there was a disincentive for one team member to confront or criticize another, even in the face of an impending error. Eighty percent of the current astronauts come from the military, which has very specific rules regarding confrontations, dissent, and criticism.[2] In addition to the similarities in behavior arising from their common backgrounds, the “criticism” taboo was continually reinforced throughout the astronaut’s career. Virtually any negative comment on an astronaut’s record was sufficient for him or her to be assigned to another crew, “washed out” of an upcoming mission and recycled through the training program, or, worse still, released from the space program altogether.
Taboos are social markers that prohibit specific behaviors in order to maintain and propagate an existing social structure. Generally, they are unwritten rules not available to outside observers. Insiders, however, almost always perceive them simply as the way things are done, the natural social order of the organization. To confront taboos is to confront the social structure of a culture or organization.
I mention the surgical and astronautical studies for a number of reasons. Each serves as background for the study of intelligence analysts. Astronauts and surgeons have very high performance standards and low error rates.[3] Both studies highlight other complex domains that are interested in improving their own professional performance. Both studies reveal the need to employ a variety of research methods to deal with complicated issues, and they suggest that there are lessons to be learned from other domains. Perhaps the most telling connection is that, because lives are at stake, surgeons and astronauts experience tremendous internal and external social pressure to avoid failure. The same often holds for intelligence analysts.
In addition, surgery and astronautics are highly selective and private disciplines. Although their work is not secret, both groups tend to be shielded from the outside world: surgeons for reasons of professional selection, training, and the fiscal realities of malpractice liability; astronauts because their community is so small and the selection and training processes are so demanding.[4] Intelligence analysts share many of these organizational and professional circumstances.
The Intelligence Community is relatively small, highly selective, and largely shielded from public view. For its practitioners, intelligence work is a cognitively-demanding and high-risk profession that can lead to public policy that strengthens the nation or puts it at greater risk. Because the consequences of failure are so great, intelligence professionals continually feel significant internal and external pressure to avoid it. One consequence of this pressure is that there has been a long-standing bureaucratic resistance to putting in place a systematic program for improving analytical performance. According to 71 percent of the people I interviewed, however, that resistance has diminished significantly since September 2001.
It is not difficult to understand the historical resistance to implementing such a performance improvement program. Simply put, a program explicitly designed to improve human performance implies that human performance needs improving, an allegation that risks considerable political and institutional resistance. Not only does performance improvement imply that the system is not optimal, the necessary scrutiny of practice and performance would require examining sources and methods in detail throughout the Intelligence Community. Although this scrutiny would be wholly internal to the community, the concept runs counter to a culture of secrecy and compartmentalization.
The conflict between secrecy, a necessary condition for intelligence, and openness, a necessary condition for performance improvement, was a recurring theme I observed during this research. Any organization that requires secrecy to perform its duties will struggle with and often reject openness, even at the expense of efficacy. Despite this, and to their credit, a number of small groups within the Intelligence Community have tasked themselves with creating formal and informal ties with the nation’s academic, non-profit, and industrial communities. In addition, there has been an appreciable increase in the use of alternative analyses and open-source materials.
These efforts alone may not be sufficient to alter the historical culture of secrecy, but they do reinforce the idea that the Intelligence Community itself has a responsibility to reconsider the relationship between secrecy, openness, and efficacy. This is especially true as it relates to the community’s performance and the occurrence of errors and failure. External oversight and public debate will not solve these issues; the desire to improve the Intelligence Community’s performance needs to come from within. Once the determination has been found and the necessary policy guidelines put in place, it is incumbent upon the Intelligence Community to find and utilize the internal and external resources necessary to create a performance improvement infrastructure.
Scope
This project was designed explicitly as an applied research program. In many respects, it resembles an assessment of organizational needs and a gap analysis, in that it was intended to identify and describe conditions and variables that affect intelligence analysis and then to identify needs, specifications, and requirements for the development of tools, techniques, and procedures to reduce analytic error. Based on these findings, I was to make recommendations to improve analytic performance.
In previous human performance-related research conducted in the military, medical, and astronautic fields, I have found in place—especially in the military—a large social science literature, an elaborate training doctrine, and well-developed quantitative and qualitative research programs. In addition to research literature and programs, these three disciplines have substantial performance improvement programs. This was not the case with the Intelligence Community.
This is not to say that an intelligence literature does not exist but rather that the literature that does exist has been focused to a greater extent on case studies than on the actual process of intelligence analysis.[5] The vast majority of the available literature is about history, international relations, and political science. Texts that address analytic methodology do exist, and it is worth noting that there are quantitative studies, such as that by Robert Folker, that compare the effectiveness of different analytic methods for solving a given analytic problem. Folker’s study demonstrates that objective, quantitative, and controlled research to determine the effectiveness of analytic methods is possible.[6]
The literature that deals with the process of intelligence analysis tends to be personal and idiosyncratic, reflecting an individualistic approach to problem solving. This is not surprising. The Intelligence Community is made up of a variety of disciplines, each with its own analytic methodology. The organizational assumption has been that, in a multidisciplinary environment, intelligence analysts would use analytic methods and tools from their own domain in order to analyze and solve intelligence problems. When interdisciplinary problems have arisen, the organizational assumption has been that a variety of analytic methods would be employed, resulting in a “best fit” synthesis.
This individualistic approach to analysis has resulted in a great variety of analytic methods—I identified at least 160 in my research for this paper—but it has not led to the development of a standardized analytic doctrine. That is, there is no body of research across the Intelligence Community asserting that method X is the most effective method for solving case one and that method Y is the most effective method for solving case two.[7]
The utility of a standardized analytic doctrine is that it enables an organization to determine performance requirements, a standard level of institutional expertise, and individual performance metrics for the evaluation and development of new analytic methodologies.[8] Ultimately, without such an analytic baseline, one cannot assess the effectiveness of any new or proposed analytic method, tool, technology, reorganization, or intervention. Without standardized analytic doctrine, analysts are left to the rather slow and tedious process of trial and error throughout their careers.
Generally, in research literature, one finds a taxonomy, or matrix, of the variables that affect the object under study. Taxonomies help to standardize definitions and inform future research by establishing a research “road map.” They point out areas of interest and research priorities and help researchers place their own research programs in context. In my search of the intelligence literature, I found no taxonomy of the variables that affect intelligence analysis.
Following the literature review, I undertook to develop working definitions and a taxonomy in order to systematize the research process. Readers will find the working definitions in the first chapter. The second chapter highlights the the broader findings and implications of this ethnographic study.
Because the first two chapters contain many quotes from my interviews and workshops, they illustrate the tone and nature of the post-9/11 environment in which I worked.
The taxonomy that grew out of this work was first described in an article for the CSI journal, Studies in Intelligence, and is presented here as Chapter Three. In addition to the normal journal review process, I circulated a draft of the taxonomy among 55 academics and intelligence professionals and incorporated their suggestions in a revised version that went to press. This is not to assert that the taxonomy is final; the utility of any taxonomy is that it can be revised and expanded as new research findings become available.
The chapter by Dr. Judith Meister Johnston that follows offers an alternative model—more complex and possibly more accurate than the traditional intelligence cycle—for looking at the dynamics of the intelligence process, in effect the interrelationships of many elements of the taxonomy
The following chapters, prepared by me and other able colleagues, were developed around other variables in the taxonomy and offer suggestions for improvement in those specific areas. One of them—Chapter Five, on integrating methodologists and substantive experts in research teams—also appeared in Studies in Intelligence. Chapter Nine contains several broad recommendations, including suggestions for further research.
To the extent possible, I tried to avoid using professional jargon. Even so, the reader will still find a number of specific technical terms, and, in those cases, I have included their disciplinary definitions as footnotes.
A Work in Progress
In some respects, it may seem strange or unusual to have an anthropologist perform this type of work rather than an industrial/organizational psychologist or some other specialist in professional performance improvement or business processes. The common perception of cultural anthropology is one of fieldwork among indigenous peoples. Much has changed during the past 40 years, however. Today, there are many practitioners and professional associations devoted to the application of anthropology and its field methods to practical problem-solving in modern or postindustrial society.[9]
It is difficult for any modern anthropological study to escape the legacy of Margaret Mead. She looms as large over 20th century anthropology as does Sherman Kent over the intelligence profession. Although Franz Boas is arguably the father of American anthropology and was Margaret Mead’s mentor, hers is the name everyone recognizes and connects to ethnography.[10] Chances are, if one has read anthropological texts, one has read Mead.
I mention Mead not only because my work draws heavily on hers, but also because of her impact on the discipline and its direction. She moved from traditional cultural anthropological fieldwork in the South Pacific to problem-oriented applied anthropology during World War II. She was the founder of the Institute for Intercultural Studies and a major contributor to the Cold War RAND series that attempted to describe the Soviet character. She also pioneered many of the research methods that are used in applied anthropology today. I mention her work also as an illustrative point. After two years of field research in the South Pacific, she wrote at least five books and could possibly have written more.
As I look over the stacks of documentation for this study, it occurs to me that, given the various constraints of the fellowship, there is more material here than I will be able to address in any one text. There are the notes from 489 interviews, direct observations, participant observations, and focus groups; there are personal letters, e-mail exchanges, and archival material; and there are my own notes tracking the progress of the work. Moreover, the fieldwork continues. As I write this, I am scheduling more interviews, more observations, and yet more fieldwork.
This text, then, is more a progress report than a final report in any traditional sense. It reflects findings and recommendations to date and is in no way comprehensive. Finally, based as it is on my own research interests and research opportunities, it is but one piece of a much larger puzzle.
Footnotes:
