Testing the Intelligence Cycle Through Systems Modeling and Simulation
Judith Meister Johnston[1]
Rob Johnston
Throughout the Intelligence Community, the process of analysis is represented conventionally by a model known as the Intelligence Cycle (See next page). Unfortunately, the model omits elements and fails to capture the process accurately, which makes understanding the challenges and responsibilities of intelligence analysis much more difficult. It also complicates the tasks of recognizing where errors can occur and determining methods for change based on accurate predictions of behavior. Our analysis of the Intelligence Cycle, employing a systems approach and a simulation created to represent it, demonstrated these shortcomings.[2] Because of its wide acceptance and use in training and in discussions of the analytic process, the traditional representation of the Intelligence Cycle will be closely considered in this chapter, especially with regard to its impact on analytic products, its effectiveness, and its vulnerability to error and failure.
The Traditional Intelligence Cycle
The Intelligence Cycle is customarily illustrated as a repeating process consisting of five steps.[3] Planning and direction encompasses the management of the entire effort and involves, in particular, determining collection requirements based on customer requests. Collection refers to the gathering of raw data to meet the collection requirements. These data can be derived from any number and type of open and secret sources. Processing refers to the conversion of raw data into a format analysts can use. Analysis and production describes the process of evaluating data for reliability, validity, and relevance; integrating and analyzing it; and converting the product of this effort into a meaningful whole, which includes assessments of events and implications of the information collected. Finally, the product is disseminated to its intended audience.[4]
The Traditional Intelligence Cycle [PDF 60KB*]
In some ways, this process resembles many other production cycles. It is prescriptive, structured, made up of discrete steps, and expected to yield a specific product. The traditional depiction of the process in the Intelligence Cycle, however, is not an accurate representation of the way intelligence is produced. The notion of a cycle assumes that the steps will proceed in the prescribed order and that the process will repeat itself continuously with reliable results. This type of representation gives the impression that all inputs are constant and flow automatically, but it does not address elements that may influence the movement of the cycle, positively or negatively.
The most significant assumption about the Intelligence Cycle model, that it provides a means for helping managers and analysts deliver a reliable product, should be examined at the outset. This can be accomplished through two types of analyses. The first is a systematic examination of the elements of the process, the inputs it relies on, and the outcomes that can be expected. The second uses a systemic approach to identifying the relationships of the elements in the process and their influence on each other.
Systematic Analysis
Many disciplines (for example, business process, organizational management, human performance technology, program evaluation, systems engineering, and instructional systems design) employ specific methods to analyze the effectiveness of products, programs, or policy implementation. Although they are often given different, domain-specific names and may involve varying levels of detail, these analytic methods involve the identification of inputs, processes, and outputs. Once these elements are identified, the evaluation process maps the relationships of the inputs, their implementation in processes, and their impact on intended—as opposed to actual—outputs.[5] The reasoning underlying this approach is that an effective product, result, or action is one that matches its objectives and that these objectives are reached by processes that logically lead from the objectives to results. Along the way, existing practices and barriers to reaching goals effectively can be identified. Finally, interventions, which can range in complexity from simple job aids to a complete restructuring of the process, can be proposed and implemented and their impacts assessed.[6]
This method of analysis has been employed successfully to evaluate processes that have characteristics similar to the Intelligence Cycle, and we use it here to examine the effectiveness of the Intelligence Cycle and its utility in representing the creation of sound analytic products while avoiding failure or error.
Findings Based on Systematic Analysis
The Intelligence Cycle is represented visually to provide an easy-to-grasp and easy-to-remember representation of a complex process. Although this type of representation may make the flow of information and the interrelationships of steps easy to identify, it does not indicate who or what may affect the completion of a step or the resources needed to begin the next step. In its concise form, then, the visual representation of the Intelligence Cycle is reduced to a map of information handling. Without explicit descriptions of the steps in the process or the benefit of prior knowledge, it can raise questions of accuracy and completeness and can occasion misconceptions, particularly concerning the roles and responsibilities of intelligence analysts.
Inputs Table [PDF 75.7KB*]
The table above depicts a more detailed input, process, and output analysis and makes some relationships clearer—for example, the steps that include two actions (planning and direction, analysis and production) have been separated into distinct processes—but it sill leaves a number of questions unanswered. It is difficult to see from this analysis specifically who is responsible for providing inputs, carrying out the processes, and producing outputs; and what requirements are expected of the inputs and outputs.
An important issue that this analysis only partly clarifies is the role of analysts. Nor does it demonstrate how great a burden the process places on them, an especially important point. Assuming that the actions identified in the “Processes” column are ultimately the responsibility of the intelligence analyst, the steps of the process move from a heavy reliance on information coming in from sources outside the analyst’s control to a heavy reliance on the analyst to produce and manage the final submission of the product.
Another important defect in this analysis is that steps in the cycle do not accurately represent the differences in the cognitive complexity involved in preparing a long-range assessment or a national intelligence estimate and that required for a two-paragraph brief on a current situation. The same can be said about the process required to develop each of the products.
The Intelligence Cycle depicts a sequential process and does not provide for iterations between steps. This is not an accurate reflection of what happens, particularly in the collection and production steps, where the challenges of defining policymaker needs and shaping collection necessitate repeated refinement of requirements by policymakers or of inferences by the Intelligence Community. A more accurate picture of the steps in the process and their iterative tendenciesmay be seen in Greg Treverton’s model, which he terms the “Real” Intelligence Cycle (above).[7]
Treverton's "Real" Intelligence Cycle [64.4KB*]
Mark Lowenthal proposes another model.[8] Although presented in a more linear fashion than Treverton’s, it focuses on the areas where revisions and reconsiderations take place, representing iteration in a slightly different light. Both models provide a more realistic view of the entire process. In addition, assuming that the analyst’s role is represented by the “Processing, Analysis” box, the Treverton model allows us to focus visually and conceptually on the demands that the process can place on the analyst. However, neither model provides an effective way of showing who is responsible for what, and neither reflects the impact of the work on the individuals responsible for producing the reports—particularly the analyst—nor the reliance of the analyst on a variety of factors beyond his or her control.
In sum, this brief evaluation of the Intelligence Cycle with respect to its inputs, processes, and outputs shows us that the traditional model:
-
assumes the process works the same way for all objectives, regardless of complexity and cognitive demands;
-
does not represent the iterative nature of the process required for meeting objectives;
-
does not identify responsibilities for completing steps and allows for misconceptions in this regard;
-
does not accurately represent the impact of resource availability on analysts.
To better understand these limitations and the relationships among elements in the process, it is necessary to step back and take a longer view of the process, using a different method of analysis.
Systemic Analysis
If we think of the phenomenon that is being described by the Intelligence Cycle as a system and perform a systems analysis, we may be able to derive a greater understanding of process relationships, a better representation of the variables affecting the process, and a greater level of detail regarding the process itself.
The premise that underlies systems analysis as a basis for understanding phenomena is that the whole is greater than the sum of its parts. A systems analysis allows for the inclusion of a variety of influences and for the identification of outliers that are obfuscated in other types of analyses but that often play major roles. A systems analysis is accomplished through the examination of phenomena as cause-and-effect patterns of behavior. This approach is called a “closed feedback loop” in systems analysis. It requires a close examination of relationships and their influences, provides a longer view of these relationships, and often reveals new insights based on trends rather than on discrete events.[9]
The systems model diagrammed below is a visual representation of the process. The elements of the Intelligence Cycle are identified in terms of their relationships with each other, the flow of the process, and phenomena that influence the elements and the flow. The model uses four icons to represent actions and relationships within the system: stocks, flows, converters, and connectors. The icons and their placement within the systems model show the relationships of the elements of the analyzed phenomenon.
The Components of the System Model [PDF 68.9KB*]
The systems model of the Intelligence Cycle provides insights into the process of analysis as well as other factors that can influence the successful and timely completion of an intelligence task. It also provides a way to understand the impact of change in any area of the Intelligence Cycle on other elements, either through reflection or by applying mathematical values to the influences and relationships and running simulations of the model.
Demand. As in the traditional Intelligence Cycle model, the systems model begins with requirements for information that generally come from policymakers. These requirements are represented by a stock (found in the upper left-hand quarter of the diagram) because they can increase or decrease based on the level of need for information (a flow). The change in level of need is influenced by national and world events, as well as by new questions or requests for clarification of items in previously delivered products. Each request does not contribute equally to the amount of work, which is influenced by the types of documents or products requested, the complexity of the products, and the turnaround time imposed. All of these factors determine the level of demand placed on the analyst.
Systems Model of the Intelligence Cycle [PDF 112KB*]
Production. This section focuses on the process of producing intelligence products. The elements described are tied, directly or indirectly, to the flow that represents changes in the analyst’s ability to produce. In turn, these changes cause products to be completed and requests of policymakers to be fulfilled. It is important to note that this portion of the model deals with factors that influence the act of analysis and does not attempt to address methods of analysis.
Factors that influence the ability of analysts to produce are numerous and complex, as shown. First and foremost are the capabilities an analyst brings to the task. This is represented by a stock—usually an increasing one—that derives from an analyst’s education, training, and experience.
Another influence is the number and frequency of evaluations and revisions imposed on a work in progress. That a draft of the product must be reviewed and edited by a number of others places variable constraints on the time available for creating the original draft. This factor increases in significance when the product requested has a short deadline.
Political and cultural values of the organization also have an influence, usually constraining. Strictly following traditional heuristics and methods and meeting organizational or management expectations may influence both an analyst’s ability to produce and the quality of the output. The weight of these influences will vary depending on the experience of the analyst.
Another factor that influences the analyst’s ability to produce is the amount of relevant, usable data (a stock) available. The term “relevant, usable data” describes all collected intelligence that is relevant to meeting the request and that exists in a format that can be used to develop the product. To become usable, the data must go through steps that are influenced by a variety of other people, organizations, systems, and technologies. This process is represented by the stock and flow chain that appears across the middle of diagram.
Data are collected from a variety of sources, represented by the INTs converter.[10] These data add to the stock of collected data. The ways in which accumulated collected data are converted to the stock of available data are influenced by internal research demands and specific collection requirements imposed by analysts, policymakers, and others. Once the data are processed and put into an agreed format for use by intelligence producers and consumers, they add to the accumulation of material that affects the ability of an analysts to produce.
Product Influences. The accumulation of completed intelligence products, which is represented as a stock, is not in practice an end-state for analysis. A customer may respond to a delivered product by levying additional or revised tasking. In all instances, this information influences the level of need for policymaker requirements and causes the process to begin again. Each iteration of the process is different, not because the steps in the process change, but because those responsible for carrying out the steps have changed as a result of their participation in the previous run. These changes can include a greater level of experience with the process, with the customer, with the topic area, or with the quirks of the organization and its processes. The changes are a manifestation of the concept that the system is greater than the sum of its parts.
Findings Based on Systems Analysis
Systems analysis clearly demonstrates the defects of the traditional Intelligence Cycle model. To recapitulate briefly, the traditional model merely represents a simple list of steps rather than a dynamic closed feedback loop. In addition, although the steps are meant to be performed by several different actors, the model does not provide useful information about what each actually contributes to the cycle, nor does it accurately represent the path a request takes as it is addressed. Another problem with the traditional model is that none of its features help identify ways of developing a consistent product. For example, there is no allowance for a statement of objectives or for any formative or summative evaluations to check that objectives have been met.
On the other hand, the model that resulted from a systems analysis provides a more complex view. That model shows cause and effect, and it shows what other elements have an impact on the development of intelligence products and how and why elements depend on other elements. These advantages of the systems model are clearly apparent in considering the role of analysts in production, a crucial element of the cycle that the traditional model all but ignores.
Impact on Production and Analyst’s Control. Study of the systems model shows that the “Analyst’s Ability to Produce” (upper right-hand quarter of the diagram) is the central factor in the production cycle and the driver of the feedback loop. The systems view also makes us aware of a less obvious fact that is critically important to a discussion of analytic failure.
A look at the entire system makes readily apparent the number of factors of varying complexity that influence an analyst’s ability to produce: the analyst’s capabilities; the product evaluation process; the political and cultural values of the organization; the amount of relevant, usable data and actions related to transforming collected data to relevant, usable data; and the level of demand on the analyst. Of these five factors, only one—the analyst’s capabilities—is an internal factor and somewhat under the analyst’s control.[11] Yet, even though the other factors are out of the analyst’s control, the analyst must rely on them to accomplish the goal and to meet the expectations of customers and the organization. When the proportion of external factors to internal factors is as unbalanced as the systems model of the Intelligence Cycle demonstrates, the causes of stress in the analytic environment increase, as does the possibility that stress will occur.
In such a high stress environment, where the critical person is responsible for delivering a product whose development relies on a great number of factors beyond his or her control, there is greater risk of error, with an increased likelihood of incomplete or incorrect products. Tendencies to use shortcuts, to avoid creative thinking, and to minimize the perceived impact of certain events or actions become more apparent in this situation, especially if their implementation means reducing the workload and the stressors. Results of working in such an environment can include increased personnel turnover, missed or undervalued information, lack of attention to detail, decreased motivation, and a lack of creativity in approaching analysis. Moreover, with analysts so central to the process, their actions may have a widespread and, thus, powerful influence on the entire system. This change can be positive or negative. Given the number of elements influencing the analyst that are out of his or her control, however, it is unlikely that the changes would positively affect the quality, accuracy, and number of intelligence products created.
Recommendations
Revisit the traditional intelligence model. The traditional Intelligence Cycle model should either be redesigned to depict accurately the intended goal, or care should be taken to discuss explicitly its limitations whenever it is used. Teaching with an inaccurate aid merely leads to misconceptions that can result in poor performance, confusion, and a need for unlearning and reteaching. If the objective is to capture the entire intelligence process, from the request for a product to its delivery, including the roles and responsibilities of Intelligence Community members, then something more is required. This should be a model that pays particular attention to representing accurately all the elements of the process and the factors that influence them.
Further Study. The use of simulation allows us to determine flaws in the system that basic informational models cannot address. A simulation moves the image of the Intelligence Cycle from a picture that selectively and indiscriminately illustrates a series of events to a holistic and realistic representation of events, responsibilities, processes, and their impact on each other. The simulation of the Intelligence Cycle developed for this analysis is merely a first step. Further work should be done with it to validate the representations, test for vulnerabilities, predict outcomes, and accurately recommend changes.
Lightening the Analyst’s Load. The systems model reveals a serious imbalance in the work processes analysts can and cannot control. It is unrealistic and unnecessary to consider reorganizing the process to correct this defect. However, there are actions that could be taken to provide analysts more control over external factors without significantly altering their roles. These actions would also reduce the amount of potential influence that one group could have over the entire process.
First, analysts might be designated as reports or research analysts. The former would prepare products that address short-term tasks, such as writing for the PDB. As the process of collection and analysis is different for short- and long-term products, this might be a responsibility assigned primarily to more junior analysts. Research analysts might be those with more experience. Freed from the obligation to prepare short-term reports, senior analysts would be available for more intense research efforts, such as those required for an NIE. In addition, cross-training or experience in creating both products and the flexibility to switch from one process to another would provide greater depth of personnel. If appropriate, movement to a long-term research position could be viewed as professional development.
Second, personnel responsible for formatting and processing raw data might be included on accounts. Through association with a particular group, people in this role would have a reasonable idea of analysts’ requirements. This would allow the preselection and preparation of data, so that analysts could focus on “connecting the dots.” The skills requirement for this role would be akin to those of a research librarian.
Third, tools to help the analyst identify, manage, and fuse relevant data could be identified and deployed. These tools, which need not be limited to those that are technology-based, should be used to support analysts’ labor-intensive tasks, thereby freeing them to focus on the analysis of data.
Employ alternative methods for examining work processes. Just as we used alternative methods to examine the Iintelligence Cycle, and as managers press analysts to use alternative analyses in assessing their targets, so should managers employ alternative methods for examining work processes. These methods should not simply test effectivenss; they should also identify vulnerabilities and potential sources of other problems in the community’s analytical methods.
Footnotes:
* Adobe® Reader® is needed to view Adobe PDF files. If you don't already have Adobe Reader installed, you may download the current version at www.adobe.com (opens in a new window). [external link disclaimer]
