Evaluation of 2013 and Prior Reference Case Projections

The U.S. Energy Information Administration (EIA) produces projections of energy production, consumption and prices each year in the Annual Energy Outlook (AEO). Each year, EIA also produces an AEO Retrospective Review document, which presents a comparison between realized energy outcomes and the Reference case projections included in previous editions of the AEO. The purpose of the Retrospective is to show the relationship between past AEO projections and actual energy indicators, enable trend analysis, and inform discussions of potential improvements to the AEO.

The projections in the AEO are not statements of what will happen but of what might happen, given assumptions and methodologies. The AEO Reference case projection assumes trends that are consistent with historical and current market behavior, technological and demographic changes, and current laws and regulations. The potential impacts of pending or proposed legislation, regulations, and standards are not reflected in the Reference case projections¹.

The discussion and analysis in each year's AEO primarily focuses on a Reference case, lower and higher economic growth cases, and lower and higher oil price cases. In addition to these standard cases, numerous alternative cases have been included in recent AEOs. Readers are encouraged to review the full range of cases, which address many of the uncertainties inherent in long-term projections.

The AEO Retrospective comparison, by contrast, summarizes the differences between actual energy indicators and past AEO Reference case projections, by calculating the average absolute differences and the average absolute percent differences for several of the major variables for AEO1994 through AEO2013². The average absolute percent difference is the simple mean of the absolute values of the percentage difference between the Reference case projection and the actual value. The historical data are primarily from the Monthly Energy Review (MER)³. This edition includes tables for prices in constant dollars in addition to nominal-dollar tables. Projections for consumption concepts depend on real prices, so constant-dollar price comparisons give a clearer picture of the linkage between consumption projection differences and price projection differences.

The annual projection process begins with the development of assumptions for two key drivers—the world oil price and the macroeconomic growth baseline—that are determined exogenously to the National Energy Modeling System (NEMS) used to prepare the AEO. While the integrated nature of NEMS results in some dynamic feedback that slightly modifies the initial assumptions about the world oil price and the macroeconomic growth baseline, these feedbacks tend to be relatively small, so the initial assumptions for these drivers largely determine the overall projection environment. To the extent that the initial assumptions used to drive the NEMS projections deviate from the realized environment, the NEMS projection results for energy markets will also generally deviate from realized energy outcomes.

Table 1 provides a summary of 25 comparisons based on 21 primary concepts (four concepts are energy price projections presented in both real-dollar and nominal-dollar valuations). The detailed results that underlie Table 1 are presented in Tables 2 through 22. These tables show the Reference case projections, historical values and percent differences between projected and actual for all years available. They also provide the average absolute difference across all AEOs for each year of history. The tabulated results can change from one year’s evaluation to the next due to data revisions for prior years that can occur in the Monthly Energy Review (MER), and, occasionally, in gross domestic product (GDP).

The first column in Table 1 is the simple average of the absolute value of percent differences for each concept across all AEOs since the 1994 edition and across all available historical years. In general, energy consumption quantities tend to be more stable and thus projected with greater accuracy than the relatively volatile concepts of net imports and energy prices. This is because energy consumption trends reflect underlying sources of inertia in the energy-consuming capital stock, substantial lead times for capital purchase decisions, locked-in contract periods, and consumer perceptions of and responses to price changes. The average absolute percent differences for energy consumption, production, and carbon dioxide emissions range roughly between 5 and 11 percent. The absolute percent differences for energy prices range between 10 and 37 percent. Net imports tend to exhibit even larger average absolute differences, ranging between 18 and 41 percent.

The second column of Table 1 provides a summary of the percentage of cases in which a particular concept is overestimated relative to its actual value out of the total number of annual projections made for which actual values are available across all AEOs since the 1994 edition. A zero would represent a concept that was always underestimated and 100 percent would represent a concept always overestimated. In an "unbiased" projection, one would expect the percentage of overestimates to be close to 50 percent when the sample size is large enough.

Table 1. AEO Reference Case Projection Results

Variable	Average Absolute Percent Differences	Percent of Projections Over- Estimated
Gross Domestic Product
Real Gross Domestic Product (Average Cumulative Growth)* (Table 2)	0.9	46.1
Petroleum
Imported Refiner Acquisition Cost of Crude Oil (Constant $) (Table 3a)	36.6	18.0
Imported Refiner Acquisition Cost of Crude Oil (Nominal $) (Table 3b)	35.9	18.9
Total Petroleum Consumption (Table 4)	7.1	68.9
Crude Oil Production (Table 5)	6.7	55.3
Petroleum Net Imports (Table 6)	18.0	71.4
Natural Gas
Natural Gas Wellhead Prices (Constant $) (Table 7a)	33.6	32.5
Natural Gas Wellhead Prices (Nominal $) (Table 7b)	33.0	33.5
Total Natural Gas Consumption (Table 8)	7.9	68.4
Natural Gas Production (Table 9)	7.3	63.6
Natural Gas Net Imports (Table 10)	41.0	72.3
Coal
Coal Prices to Electric Generating Plants (Constant $)** (Table 11a)	19.4	37.9
Coal Prices to Electric Generating Plants (Nominal $)** (Table 11b)	21.1	41.3
Total Coal Consumption (Table 12)	9.4	63.6
Coal Production (Table 13)	6.6	66.5
Electricity
Average Electricity Prices (Constant $) (Table 14a)	10.0	35.9
Average Electricity Prices (Nominal $) (Table 14b)	12.4	40.8
Total Electricity Sales (Table 15)	4.5	50.5
Total Energy, Carbon and Intensity
Total Energy Consumption (Table 16)	6.3	80.6
Delivered Residential Energy Consumption (Table 17)	5.1	71.8
Delivered Commercial Energy Consumption (Table 18)	6.4	48.1
Delivered Industrial Energy Consumption (Table 19)	11.1	87.9
Delivered Transportation Energy Consumption (Table 20)	7.2	80.1
Total Energy Related Carbon Dioxide Emissions (Table 21)	7.5	65.0
Energy Intensity (Energy Consumption / Real $ GDP) (Table 22)	10.1	94.2
AEO - Annual Energy Outlook Source: These statistics summarize the calculations in Tables 2 through 22. The data in Tables 2 through 22 are based on the 1994 through 2013 AEO Reference case projections. Historical data are taken from the Monthly Energy Review August 27, 2013, DOE/EIA-0035(2013/08) (Washington, DC, August 2013), except for GDP-related concepts which are from the Bureau of Economic Analysis, U.S. Dept. of Commerce, July 2013, http://www.bea.gov/national/xls/gdplev.xls. * The basis for GDP comparison is the projection differences in the cumulative average growth rate of real GDP from the first year shown for each AEO. The summary information for projection differences given for GDP growth rates is absolute percentage point differences; for all other AEO concepts the comparison basis is absolute percent differences. ** Beginning in AEO2003, EIA electric generating projections incorporated combined heat and power (CHP) electricity generation in electricity generating plants. Prior to AEO2003, coal price projections reflected data collected, estimated, and reported to electric utilities and excluded CHP power generation.

The underlying reasons for deviations between the AEO Reference case projections and actual outcomes are numerous and varied. Changes in economic, market, and technological trends and in policies affecting energy choices are often important drivers of these deviations. EIA monitors these developments and uses the best available data to continuously update NEMS and the AEO projections. EIA makes extensive use of scenarios in the AEO to explore the implications of major breaks from historical trends. Even though the precise timing and magnitude of a change in trend cannot be predicted, the likelihood that there will be some changes in trend is virtually certain, and the impact of such changes is estimated using the AEO scenarios. Thus, the full range of potential outcomes cannot be evaluated with the AEO Reference case alone, but only in conjunction with the relevant AEO scenarios and other analysis as applicable to a specific situation.

Some notable factors contributing to deviations between actual outcomes and AEO projections are:

• Macroeconomic factors play a very important role in determining energy usage. The financial crisis and resulting economic weakness starting in 2007 was not anticipated in AEO modeling as NEMS does not project future business cycles. AEO2001 through AEO2009 over-projected macroeconomic growth (Table 2). Earlier AEOs systematically underestimated cumulative growth, even through 2007-2009, due in part to the robust 3.9% average growth in real GDP from 1993 through 2000. Post-2000 real GDP growth has decreased to an average of 1.6%, and AEOs produced during this period have tended to over-project, even before the impacts of the latest recession. In general, the AEO tends to lag behind macroeconomic trends as a result of an analytic framework which returns economic factors to long-run trends, rather than attempting to predict business cycles or other economic phenomena.

• The imported refiner acquisition cost of crude oil in 2012 remained over $100 per barrel, down slightly from its peak in 2011 (Tables 3a, 3b). Many of the AEOs prior to AEO2006 projected nominal prices less than $30 per barrel. Underestimation of crude oil prices since 2000, and higher vehicle efficiency standards that were not included in the Reference Case assumptions, contributed to overestimation of petroleum consumption. This coupled with the effects of the 2007-2009 recession has generally resulted in overestimates of petroleum consumption for the years 2008 through 2012 (Table 4).

• Petroleum net imports peaked in 2005, falling successively in each year since. Before AEO2006, with few exceptions, AEOs had projected rising net imports for this period. From AEO2006 on, projections for net imports have included years of decline (Table 6). This effect was caused primarily by the economic situation, with a lesser contribution from increasing vehicle fuel efficiency. More recently, rising domestic oil production has also played an important roll. Going forward, the phased increase in light-duty vehicle fuel economy standards through 2025 will continue to reduce projected petroleum consumption.

• Natural gas production has increased each year after 2005 after a long period of relatively flat production, and achieved a record high level in 2012 (Table 9). Natural gas net imports peaked in 2007, falling successively in each year since, with 2012 at the lowest level shown (Table 10). AEOs since AEO2006 have tended to underestimate production growth and consequently overestimate net imports.

• Prior to 2009, the fuel with the largest absolute percent difference between the projections and historical consumption has generally been natural gas. This can be partially traced to the performance of natural gas price projections. In earlier projections, the natural gas price was influenced by the changes in the world oil price, but this relationship has decoupled since 2000. Through AEO2005, the Reference case has generally overestimated natural gas consumption (Table 8), in part due to the tendency for significant underestimates of the natural gas wellhead price (Tables 7a, 7b). In addition, industrial natural gas usage is highly sensitive to business cycles, leading to greater volatility in overall natural gas consumption.

• Coal prices to electric generating plants have been increasing since 2000 and set a new high in 2012 (Tables 11a, 11b). This coupled with the continued increases in natural gas production and low and declining natural gas prices, has resulted in consumption for 2012 falling to its lowest level shown (Table 12).

• Electricity prices were almost always underestimated in the Reference case projections from AEO2000 on (Tables 14a, 14b). In deregulated markets, natural gas tends to determine marginal electricity prices, and with part of the United States remaining regulated, this presents a complex pricing environment for a national projection; but overall, the underestimation of natural gas and coal prices noted above also results in underestimated electricity prices.

• Total energy consumption fell from 2011 and is lower in 2012 than it was in 1999. Nearly all AEOs projected rising consumption over the historical period, with AEO2008, AEO2009 and AEO2011 being exceptions. This is the result of earlier AEOs not anticipating the strong effects of the recent recession as well as underestimation of reductions in the energy intensity of the economy, as discussed below. Projection differences tend to be relatively small; however, some pattern of overestimation is evident (Tables 16 to 20).

• Projected energy intensity has been the concept most often overestimated (Table 22). Energy intensity for the retrospective comparisons is defined as the ratio of two other projected concepts: total energy consumption and real GDP. As a ratio, the projection differences are dependent upon relative differences in energy consumption and real GDP. The overestimates of energy intensity are the result of two "reinforcing" features of past projections. During the period through AEO2000, GDP tended to be underestimated, which imparts a tendency to overestimate energy intensity. Post-AEO2000, energy consumption has tended to be overestimated, once again imparting a tendency to overestimate intensity. The combination of these two occurrences has resulted in energy intensity being overestimated for nearly all years.

 

---

Notes

¹Each Annual Energy Outlook includes a variety of other cases with different assumptions – for AEO2013, 26 alternative cases exploring different assumptions and different policies were presented in addition to the Reference case.

²The National Energy Modeling System (NEMS) has been used to prepare the Annual Energy Outlook since AEO1994, and this publication covers only these AEOs. The AEO2009 results are from "An Updated Annual Energy Outlook 2009 Reference Case Reflecting Provisions of the American Recovery and Reinvestment Act and Recent Changes in the Economic Outlook" (passed after the official publication of the AEO2009). This version is only available on the EIA website at: http://www.eia.gov/analysis/requests/archive/1999/stimulus/. Finally, projections from AEOs for 1994 and 1995 ended in 2010, so beyond 2010, entries for these two publications are now blank in Tables 2 through 22.

³GDP and the GDP price deflators historical data are from the U.S. Department of Commerce, Bureau of Economic Analysis.