Forecasting Counterfactuals in Uncontrolled Settings (FOCUS)

The FOCUS program seeks to develop and empirically evaluate systematic approaches to counterfactual forecasting and lessons learned processes. Counterfactual forecasts are statements about what would have happened if different circumstances had occurred. For example, a postmortem review of an analysis failure may lead to a conclusion that analysts would have avoided the failure if they employed better analytic tradecraft; perhaps by having double checked assumptions, perhaps by having considered a broader range of hypotheses, etc. Counterfactual forecasts about what would have worked in past circumstances are very often the basis for lessons learned for what to do in the future. And such lessons often evolve, over time, into best practices and tradecraft.

To date there has been little in the way of research that measures the extent to which different approaches to counterfactual forecasting yield accurate vs. inaccurate counterfactual forecasts. And there is a similar paucity of research on the accuracy of lessons drawn from different lessons learned approaches. As a result, there does not exist evidence-based guidance for approaching lessons learned activities or for developing the counterfactual forecasts that are the core of such activities; and also unfortunately there is correspondingly little evidence supporting a claim that the lessons learned from current lessons learned approaches are usually the right lessons. FOCUS will address this research gap by developing and empirically testing alternative approaches to structuring counterfactual forecasting and lessons learned processes in ways that (a) can be readily incorporated into lessons learned activities relevant to improving intelligence analysis and tradecraft, but also (b) are broadly relevant to any organizational lessons learned activity.

Research Area(s)

• Counterfactual reasoning
• Evidence-based approaches to lessons-learned analyses
• Analytic tradecraft
• Cognitive biases

Related Publications

To access FOCUS program-related publications, please visit Google Scholar.