Release Date: February 2005
Next Release Date: February 2006

Printer-friendly version

Annual Energy Outlook Forecast Evaluation*

Table 1.Comparison of Absolute Percent Errors for Present and Current AEO Forecast Evaluations Printer Friendly Version Average Absolute Percent Error Variable AEO82 to AEO99 AEO82 to AEO2000 AEO82 to AEO2001 AEO82 to AEO2002 AEO82 to AEO2003 AEO82 to AEO2004 Consumption Total Energy Consumption 1.9 2.0 2.1 2.1 2.1 2.1 Total Petroleum Consumption 2.9 3.0 3.1 3.1 3.0 2.9 Total Natural Gas Consumption 7.3 7.1 7.1 6.7 6.4 6.5 Total Coal Consumption 3.1 3.3 3.5 3.6 3.7 3.8 Total Electricity Sales 1.9 2.0 2.3 2.3 2.3 2.4 Production Crude Oil Production 4.5 4.5 4.5 4.5 4.6 4.7 Natural Gas Production 4.7 4.6 4.6 4.4 4.5 4.6 Coal Production 3.6 3.5 3.8 3.7 3.8 4.1 Imports and Exports Petroleum Net Imports 8.8 8.3 7.7 7.4 7.3 7.2 Natural Gas Net Imports 16.0 15.9 16.1 15.8 15.4 15.2 Prices, Economic, and Environmental Variables World Oil Prices 68.5 68.4 65.5 62.4 59.3 56.4 Natural Gas Wellhead Prices 84.9 82.6 78.8 74.5 70.6 67.7 Coal Prices to Electric Generating Plants* 45.1 46.5 49.8 49.1 48.4 47.8 Average Electricity Prices 17.7 18.6 20.1 19.8 19.6 19.6 Gross Domestic Product 5.6 5.7 5.9 5.8 5.6 5.5 Total Carbon Dioxide Emissions N/A 2.3 2.6 2.5 2.6 2.6 Energy Intensity 5.7 5.8 6.1 6.0 5.8 5.7

The Office of Integrated Analysis and Forecasting of the Energy Information Administration (EIA) has produced annual evaluations of the accuracy of the Annual Energy Outlook (AEO) since 1996. Each year, the forecast evaluation expands on the prior year by adding the projections from the most recent AEO and replacing the historical year of data with the most recent. The forecast evaluation examines the accuracy of AEO forecasts dating back to AEO82 by calculating the average absolute percent errors for several of the major variables for AEO82 through AEO2004. (There is no report titled Annual Energy Outlook 1988 due to a change in the naming convention of the AEOs.) The average absolute percent error is the simple mean of the absolute values of the percentage difference between the Reference Case projection and the actual value (also referred to as percent error). The historical data are typically taken from the Annual Energy Review (AER).¹ The last column of Table 1 provides a summary of the most recent average absolute percent errors. The calculation of forecast errors is shown in more detail in Tables 2 through 18. These tables also provide the average absolute error, which is the simple mean of the absolute value of the difference between the Reference Case projection and the actual value.

This AEO forecast evaluation changes the basis of measure for comparing dollar denominated values. Gross domestic product (GDP) and energy prices are now evaluated on a nominal dollar basis using the contemporaneous deflator series from each AEO rather than a current deflator series. Each AEO presents a “real” GDP measure and a matched price index. In past forecast evaluations the calculation of nominal GDP was done by applying the latest official historical series to the “real” GDP forecast to generate a nominal series for the evaluation. This process has been revised to now use the values of the price index from each AEO document to inflate the “real” numbers to a nominal series. These revisions also change the energy intensity table, which is now presented as energy consumption divided by nominal dollar GDP. These changes improve the consistency of comparing across AEOs in ways discussed in detail below.

The growth in GDP is a good measure of the growth of the aggregate economy over time. However, GDP is but one measure. Other concepts like disposable income growth, industrial output, vehicle sales, expansion of commercial floorspace, to name a few, become the actual drivers to the energy forecast. GDP is perhaps the single best metric to show how the economy grows; and the assessment of how forecasts of GDP compare to history is a good proxy for how economic activity in general influences energy markets.

Table 1 shows the forecast errors from previous and current forecast evaluations. As indicated in Table 1, the forecasts of consumption, production, and carbon dioxide emissions have been relatively accurate, the forecasts of net imports have been moderately accurate, and the forecasts of prices have been the least accurate (although both Table 1 and the individual tables show marked improvement in forecast errors for prices and net imports). The percent errors change from one year’s evaluation to the next as an additional year of data and projections is added. The percent errors may also change because of data revisions in the AER and the Monthly Energy Review (MER).

The underlying reasons for deviations between the projections and realized history tend to be the same from one evaluation to the next. The most significant are:

• Over the last two decades, there have been changes in laws, policies, and regulations that were not assumed in the projections prior to their implementation because of EIA’s statutory requirement to be policy neutral. Many of these actions have had significant impacts on energy supply, demand, and prices. For example, the Powerplant and Industrial Fuel Act (FUA) of 1978 restricted the use of natural gas in power plants and industrial boilers. After FUA was repealed in 1987, use of natural gas for electric generation and industrial processing increased sharply. Consequently, those AEOs completed prior to or immediately after repeal of the FUA, e.g., AEO86, AEO87, and AEO89, had considerably larger underestimation of natural gas consumption in 2000 than AEO90 and all subsequent AEOs.
• AEO85 through AEO90 forecasts of GDP overestimated GDP growth in the 1990s, whereas AEO91 and later AEOs underestimated GDP growth over that period. The economic slowdown that began in 2001 has led to overestimates of GDP. Because GDP is a good indicator of the economic activity which drives the energy consumption forecasts, those forecasts often have similar error trends as the GDP forecasts (Table 16).
• The level of future electricity sales was underestimated in all years in each of the AEOs published between 1994 and 1999 (Table 6). Since about 90 percent of the demand for coal results from electricity generation, the underestimation of electricity sales led to underestimation of coal consumption in those years (Table 5). The forecast errors of electricity sales were particularly large in AEO94, AEO95, and AEO96.
• Overestimation of world oil prices, particularly in earlier AEO publications, resulted in overestimation of domestic petroleum production and underestimation of imports. In addition, high world oil price forecasts tended to cause underestimation of actual oil consumption until AEO2001 when the opposite results are observed (Table 3).
• Natural gas generally has been the fuel with the least accurate forecasts. As regulatory reforms that increased the role of competitive markets were implemented beginning in the mid-1980s, the behavior of natural gas in competitive markets was especially difficult to predict. In the earlier forecasts, EIA’s technology improvement expectations proved conservative, as technological advances made petroleum and natural gas less costly to produce. After natural gas curtailments were eased in the mid-1980s, environmental pressures made natural gas an increasingly attractive fuel source, particularly for electricity generation. Historically, natural gas price instability was strongly influenced by natural gas resource estimates, which steadily rose, and by the world oil price, which was subject to its own error. More recently, natural gas consumption has been overestimated (Table 4) due to natural gas price projections that proved to be significantly lower than what occurred (Table 13).
• Technological improvements in both the production and use of energy have had a significant impact on the price, supply, and consumption of energy. For the most part, earlier AEOs assumed much slower technology development than actually occurred, accounting for some of the deviation between the forecasts and history. This trend was recognized, in part, by this type of forecast evaluation exercise. Beginning with the Annual Energy Outlook 1994, the projections in the AEO were produced using the National Energy Modeling System (NEMS). Because NEMS was designed with methodologies to represent technology in a more detailed fashion, there has been an improvement in the capability to represent technological change throughout energy markets. Additional studies on technological improvement have led to more optimistic assumptions in the more recent projections, along with modeling innovations, such as learning-by-doing, in which experience gained with new generation technologies and advanced end-use technologies leads to cost reductions in the model. These enhancements have significantly improved the projection capability within NEMS.
• External factors such as severe weather, economic cycles, and strikes have also had an impact on energy markets; however, these events cannot be anticipated in the mid- to long-term period and are not captured in the models underlying the AEO projections.

Revisions to Gross Domestic Product and Implications for the Forecast Comparisons

The concept of GDP is a commonly used measure of economic activity. It can be expressed in nominal dollars or, with the use of a matched price index to remove inflation, in “real” terms. Movements in nominal GDP show how the value of goods and services produced by the United States changes over time, while real GDP is a measure of how the physical production of the economy has grown. While simple in concept, the forecasting of nominal and real GDP and the interpretation of these forecasted measures relative to “history” is not simple or straightforward. The Bureau of Economic Analysis (BEA) within the U.S. Department of Commerce continually adjusts the National Income and Product Accounts data, with comprehensive revisions completed every 4 or 5 years. The last four major revisions (1985, 1991, 1995, and 1999) incorporated definitional and statistical changes, as well as emphasizing new ways of presenting the data.

The changes are not restricted to the nominal series. Concept changes in the derivation of the real GDP also change the perception of economic activity. Prior to 1995, BEA measured real GDP on a fixed-base-year-method. Price deflators for each component of GDP were used to deflate the value of nominal GDP to a base year. While this method allowed total real GDP to be computed by adding the component values (consumption, investment, government, net exports), this reliance on base-year-weighting caused problems. The perception of growth in real GDP using the fixed-weighted approach is dependent on the selection of the base year. In switching the base year from 1982 to 1987, all the growth rates for the respective “real” components changed. A solution is to convert to a chain-weighted measure that eliminates the problem of historical growth rates changing when the base year is updated.² If this approach is followed, however, the component pieces of GDP do not add to the total GDP.

Following the BEA, the AEO forecasts prior to AEO1997 used a fixed-weight base year GDP with several different base years. Beginning with the AEO1997 forecast, the AEO forecast has been completed using chain-weighted dollars. Any forecast evaluation therefore must acknowledge that the forecasts were based on historical data, as it appeared at the time the forecasts were prepared. The problem is, there is only one set of historical values available now – the BEA series on nominal GDP and real GDP expressed in 2000 chain-weighted dollars. What should be the basis of comparison of forecast error – nominal or real?

Each AEO shows the price deflator (fixed-weighted or chained) and the real GDP series (also fixed-weighted or chained). A simple multiplication yields the nominal GDP series. While the transformation is simple, there is the difficulty associated with the gap in the data where BEA has added more nominal GDP to the history. Since the GDP forecast cannot anticipate such changes, it will be incorrect from the start [this is the option reflected in this evaluation.] On the real side, each AEO reports real GDP, but some are in fixed-weighted dollars with varying base years, and since 1997, some use a chain-weighting method. Again, there exists only one true history – the BEA chained-weighted 2000 dollar concept. Lack of available data from old AEOs precludes moving the old fixed-weighted real GDP into a chain-weighted concept. The alternative of viewing history from a fixed-weighted perspective is not viable because BEA does not maintain these series in that form. And even if the conversions issues did not exist, the real GDP measure would still contain the gap in the nominal series due to data revisions.

Tables 2 through 18

*Contacts: GDP issues, Ronald.Earley@eia.doe.gov; other issues, Crawford.Honeycutt@eia.doe.gov

1 GDP and the GDP price deflators come from the Bureau of Economic Analysis, while coal prices to electric generating plants are from the Monthly Energy Review (MER).

2 “Chained-Dollar Indexes: issues, Tips on Their Use, and Upcoming Changes,.” J. Steven Landefeld, Brent R. Moulton, and Cindy M. Vojtech, Survey of Current Business, Bureau of Economic Analysis, November 2003. http://www.bea.gov/bea/ARTICLES/2003/11November/1103%20Chain-dollar.pdf