Analysis & Projections
AEO Retrospective Review: Evaluation of 2011 and Prior Reference Case Projections
Release Date: March 16, 2012 | Next Release Date: March 2013 | Report Number: DOE/EIA-0640(2011)
The U.S. Energy Information Administration (EIA) produces projections of energy production, consumption and prices each year in the Annual Energy Outlook (AEO). Each year, EIA also produces this AEO Retrospective Review document, which presents a comparison between realized energy outcomes and the Reference case projections included in previous editions of the AEO. The purpose of the AEO Retrospective is to demonstrate how the AEO projections have tracked actual energy indicators, enable trend analysis, and inform discussions of potential improvements to the AEO.
The projections in the AEO are not statements of what will happen but of what might happen, given assumptions and methodologies. The AEO Reference case projection assumes trends that are consistent with historical and current market behavior, technological and demographic changes, and current laws and regulations. The potential impacts of pending or proposed legislation, regulations, and standards are not reflected in the Reference case projections 1. Thus, the AEO Reference case provides an unbiased baseline that can be used to analyze potential new policies or legislative initiatives.
The discussion and analysis in each year’s AEO primarily focus on a Reference case, lower and higher economic growth cases, and lower and higher oil price cases. In addition to these standard cases, numerous alternative cases have been included in recent AEOs. Readers are encouraged to review the full range of cases, which address many of the uncertainties inherent in long-term projections.
The AEO Retrospective comparison, by contrast, summarizes the differences between actual energy indicators and past AEO projections, by calculating the average absolute percent differences and the average absolute percent differences for several of the major variables for AEO1994 through AEO2011 2. The average absolute percent difference is the simple mean of the absolute values of the percentage difference between the Reference case projection and the actual value. The historical data are primarily from the Annual Energy Review (AER) 3. This edition includes tables for prices in constant dollars in addition to nominal-dollar tables. Projections for consumption concepts depend on real prices, so constant-dollar price comparisons give a clearer picture of the linkage between consumption projection differences and price projection differences.
The annual projection process begins with the development of assumptions for two key drivers – the world oil price and the macroeconomic growth baseline—that are determined exogenously to the National Energy Modeling System (NEMS) used to prepare the AEO. While the integrated nature of NEMS may result in some feedback that slightly modifies the initial assumptions about world oil price and the macroeconomic growth baseline, these feedbacks tend to be relatively small, so that the initial assumptions for these drivers largely determine the overall projection environment. To the extent that this general environment deviates from the initial assumptions, the NEMS projection results will also deviate.
Table 1 provides a summary for 25 comparisons based on 21 primary concepts – including both constant-dollar and nominal-dollar comparisons for each of the four price concepts projected. The detailed results that underlie Table 1 are presented in Tables 2 through 22. These tables show the Reference case projections, historical values and percent differences between projected and actual for all years available. They also provide the average absolute difference across all AEOs for each year of history. The tabulated results can change from one year’s evaluation to the next due to data revisions for prior years that can occur in the AER, the Monthly Energy Review (MER), and, occasionally, in gross domestic product (GDP).
The first column in Table 1 is the simple average of the absolute value of percent differences for each concept across all AEOs and across all available historical years. In general, energy consumption quantities tend to be less volatile and thus projected with greater accuracy than the relatively more volatile energy prices. This is because energy consumption trends reflect underlying sources of inertia in the energy-consuming capital stock, substantial lead times for capital purchase decisions, locked-in contract periods, and consumer perceptions of and responses to price changes. The average absolute percent differences for energy consumption, production, and carbon dioxide emissions range roughly between 4 and 11 percent. Net imports tend to exhibit somewhat larger average absolute differences ranging between 10 and 21 percent. The absolute percent differences for energy prices range between 10 and 33 percent.
The second column of Table 1 provides a summary of the percentage of cases in which a particular concept is overestimated relative to the total number of annual projections made across all AEOs. A zero would represent a concept that was always underestimated and 100 percent would represent a concept always overestimated. In an “unbiased” projection, one would expect the percentage of overestimates to be close to 50 percent when the sample size is large enough.
| Variable | Average Absolute Percent Differences | Percent of Projections Over- Estimated |
|---|---|---|
| Gross Domestic Product | ||
| Real Gross Domestic Product (Average Cumulative Growth)* (Table 2) | 0.9 | 40.4 |
| Petroleum | ||
| Imported Refiner Acquisition Cost of Crude Oil (Constant $) (Table 3a) | 33.3 | 20.5 |
| Imported Refiner Acquisition Cost of Crude Oil (Nominal $) (Table 3b) | 32.8 | 21.6 |
| Total Petroleum Consumption (Table 4) | 5.2 | 63.7 |
| Crude Oil Production (Table 5) | 6.2 | 62.0 |
| Petroleum Net Imports (Table 6) | 9.9 | 63.7 |
| Natural Gas | ||
| Natural Gas Wellhead Prices (Constant $) (Table 7a) | 31.4 | 24.0 |
| Natural Gas Wellhead Prices (Nominal $) (Table 7b) | 30.9 | 26.3 |
| Total Natural Gas Consumption (Table 8) | 7.5 | 69.6 |
| Natural Gas Production (Table 9) | 7.0 | 70.8 |
| Natural Gas Net Imports (Table 10) | 21.3 | 67.3 |
| Coal | ||
| Coal Prices to Electric Generating Plants (Constant $)** (Table 11a) | 17.6 | 43.3 |
| Coal Prices to Electric Generating Plants (Nominal $)** (Table 11b) | 20.2 | 47.4 |
| Total Coal Consumption (Table 12) | 6.5 | 55.6 |
| Coal Production (Table 13) | 5.1 | 60.2 |
| Electricity | ||
| Average Electricity Prices (Constant $) (Table 14a) | 9.6 | 40.9 |
| Average Electricity Prices (Nominal $) (Table 14b) | 12.4 | 45.6 |
| Total Electricity Sales (Table 15) | 4.0 | 42.1 |
| Total Energy, Carbon and Intensity | ||
| Total Energy Consumption (Table 16) | 5.0 | 76.0 |
| Delivered Residential Energy Consumption (Table 17) | 4.0 | 66.1 |
| Delivered Commercial Energy Consumption (Table 18) | 5.9 | 37.4 |
| Delivered Industrial Energy Consumption (Table 19) | 10.6 | 88.3 |
| Delivered Transportation Energy Consumption (Table 20) | 5.4 | 76.0 |
| Total Energy Related Carbon Dioxide Emissions (Table 21) | 5.5 | 57.3 |
| Energy Intensity (Energy Consumption / Real $ GDP) (Table 22) | 10.5 | 96.5 |
| AEO - Annual Energy Outlook Source: These statistics summarize the calculations in Tables 2 through 22. The data in Tables 2 through 22 are based on the 1994 through 2011 Annual Energy Outlook Reference case projections. Historical data are taken from the Annual Energy Review 2010, DOE/EIA-0384(2010) (Washington, DC, October 2011) and the Monthly Energy Review October 2011, DOE/EIA-0035(2011/10) (Washington, DC, October 27, 2011). * The basis for GDP comparison is the projection differences in the cumulative average growth rate of real GDP from the first year shown for each AEO. The summary information for projection differences given for GDP growth rates is absolute percentage point differences; for all other AEO concepts, the comparison basis is absolute percent differences. ** Coal prices to electric generating plants are from the Monthly Energy Review. Beginning in AEO2003, EIA electric generating projections incorporated combined heat and power (CHP) electricity generation in electricity generating plants. Prior to AEO2003, coal price projections reflected data collected, estimated, and reported to electric utilities and excluded CHP power generation. |
||
The underlying reasons for deviations between the AEO Reference case projections and actual outcomes are numerous and varied. In most cases, changes in economic, market, and technological trends led to the deviations. EIA monitors these developments and uses the best available data to continuously update NEMS and the AEO projections. Nonetheless, because the AEO Reference case is by definition a continuation of historic trends, breaks in trends lead to deviations between the projections and realized outcomes. EIA makes extensive use of scenarios in the AEO for precisely this reason; even though the precise timing and magnitude of a change in trend cannot be predicted, the likelihood that there will be some changes in trend is virtually certain, and the impact of such changes is estimated using the AEO scenarios. Thus, risks and opportunities presented by such potential outcomes cannot be evaluated with the AEO Reference case alone, but only in conjunction with the relevant AEO scenarios and other analysis as applicable to a specific situation.
Some notable factors contributing to deviations between actual outcomes and AEO projections are:
- Macroeconomic factors play a very important role in determining energy usage. The financial crisis and resulting economic weakness of 2007-2009 was not anticipated in AEO modeling as NEMS does not project future business cycles. AEO2001 through AEO2009 over-projected macroeconomic growth for 2007 (Table 2). Earlier AEOs systematically underestimated cumulative growth, even through 2007-2009, due in part to the robust 3.9% average growth in real GDP from 1993 through 2000 fueled by the “high tech bubble.” Post-2000 real GDP growth has decreased to an average of 1.6% and AEOs produced during this period have tended to over-project, even before the impacts of the latest recession. In general, the AEO tends to lag behind macroeconomic trends as a result of inherently conservative analytic framework which returns economic factors to long-run trends, rather than attempting to predict business cycles or other economic phenomena.
- The imported refiner acquisition cost of crude oil in 2008 was over 90 dollars per barrel and many of the AEOs prior to AEO2006 projected nominal results less than 30 dollars per barrel. Underestimation of crude oil prices since 2000 contributed to overestimation of petroleum consumption. This coupled with the effects of the 2007-2009 recession has resulted in overestimates of petroleum consumption for the years 2008 through 2010 (Table 4). Only since AEO2009 have the projection differences become more evenly distributed.
- Petroleum net imports peaked in 2005, falling successively in each year since. Before AEO2006, AEOs had projected rising net imports for this period. From AEO2006 on, projections for net imports have included years of decline (Table 6). This effect was caused primarily by the economic situation, with a lesser contribution from increasing vehicle fuel efficiency.
- Natural gas production has increased each year after 2005 after a long period of relatively flat production (Table 9). Natural gas net imports peaked in 2007, falling successively in each year since (Table 10). AEOs since AEO2006 have tended to underestimate production and consequently overestimate net imports.
- The fuel with the largest absolute percent difference between the projections and historical consumption has generally been natural gas. This can be partially traced to the performance of natural gas price projections. In earlier projections, the natural gas price was influenced by the changes in the world oil price, but this relationship has decoupled since 2000. Through AEO2005, the Reference case has generally overestimated natural gas consumption (Table 8), in part due to the tendency for significant underestimates of the natural gas wellhead price (Tables 7a, 7b). In addition, industrial natural gas usage is highly sensitive to business cycles, leading to greater volatility in overall natural gas consumption.
- Electricity prices were almost always underestimated in the Reference case projections from AEO2000 on (Tables 14a, 14b) following the pattern in the coal and natural gas price projections and the partially deregulated generation market. In deregulated markets, natural gas tends to determine marginal electricity prices and follow the changes in the delivered natural gas prices. The recent underestimation of natural gas wellhead prices contributes to underestimates in more recent electricity price projections.
- Total energy consumption is lower in 2010 than it was in 2000 (although 5 intervening years were higher than 2010). Most AEOs projected rising consumption with AEO2008 and AEO2009 being exceptions. This is the result of not anticipating the recent recession and slow recovery in the projections. Projection differences tend to be relatively small; however, some pattern of overestimation is evident (Tables 16 to 20).
- Projected energy intensity has been the concept most often overestimated (Table 22). Energy intensity for the retrospective comparisons is defined as the ratio of two other projected concepts: total energy consumption and real GDP. As a ratio, the projection differences are dependent upon relative differences in energy consumption and real GDP. The overestimates of energy intensity are the result of two “reinforcing” features of past projections. During the period through AEO2000, GDP tended to be underestimated, which imparts a tendency to overestimate energy intensity. Post-AEO2000, energy consumption has tended to be overestimated, once again imparting a tendency to overestimate intensity. The combination of these two occurrences has resulted in energy intensity being overestimated for nearly all years.