Annual Energy Outlook Forecast Evaluation
The Office of Integrated Analysis and Forecasting has been providing an evaluation of the forecasts in the Annual Energy Outlook (AEO) annually since 1996. Each year, the forecast evaluation expands on that of the prior year by adding the most recent AEO and the most recent historical year of data. However, the underlying reasons for deviations between the projections and realized history tend to be the same from one evaluation to the next. The most significant conclusions are:
- Over the last two decades, there have been many significant changes in laws, policies, and regulations that could not have been anticipated and were not assumed in the projections prior to their implementation. Many of these actions have had significant impacts on energy supply, demand, and prices; however, the impacts were not incorporated in the AEO projections until their enactment or effective dates in accordance with EIA's requirement to remain policy neutral and include only current laws and regulations in the AEO reference case projections.
- Energy prices (including petroleum, natural gas, coal, and electricity) have been far more difficult to predict than consumption, production, and net imports. Typically the rate of increase in energy prices has been overestimated rather than underestimated. More rapid technological improvements, the erosion of the market power of the Organization of Petroleum Exporting Countries starting in the mid-1980s, excess productive capacity, and market competitiveness are all factors that led to lower energy prices than projected. In the 1980s and 1990s, productivity and technology improvements and the effects of gradual deregulation and changes in industry structure more than offset the factors that tended to result in higher energy prices, such as resource depletion and growing energy demand. However, the tendency to overestimate energy prices may be reversing. In more recent years, better adherence by the oil-producing countries to production agreements (crude oil), more rapid depletion (crude oil and natural gas), or unexpected consequences of deregulation plans (electricity) have led to generally higher prices than expected only a few years ago.
- The level of future electricity sales have been consistently underestimated in all years in each of the AEOs published between 1991 and 2002. The underestimation averages less than 3 percent, which is a relatively small error in a long-term forecast. It implies that the fuels consumed to generate electricity (e.g., coal, oil, natural gas) are also being underestimated. There have been many changes in the structure and growing sources of uncertainty in the electric industry and markets over the last decade that may help to explain this underestimation. This includes deregulation of electric generation, the ubiquity of computer equipment and consumer electronics, and growth in internet use and the expansion of the information technology sector during the economic boom of the 1990s. Another important factor potentially contributing to the underestimation of electricity consumption is the series of unpredictable weather patterns that has affected much of the country over the past decade.
- Natural gas generally has been the fuel with the least accurate forecasts in consumption, production, and prices. Natural gas was the most recent fossil fuel to be affected by regulatory reforms following the strong regulation of energy markets in the 1970s and early 1980s. As regulatory reforms were implemented, the behavior of natural gas in competitive markets was especially difficult to predict. Natural gas market trends have been unstable in the past 25 years, which increased the difficulty of determining future levels for market prices and volumes. Furthermore, in past projections the natural gas market outlook was strongly influenced by the assumed world oil price, which was subject to its own error; by natural gas resource estimates, which steadily rose; and by technology improvement expectations, which proved conservative.
- External factors such as severe weather, economic cycles, and strikes have also had an impact on energy markets; however, these events cannot be anticipated in the mid- to long-term period and are not captured in the models underlying the AEO projections.
- Technological improvements in both the production and use of energy have had a significant impact on the price, supply, and consumption of energy. For the most part, earlier AEOs assumed much slower technology development than actually occurred, accounting for some of the deviation between the forecasts and history. This influence was identified, in part, by the forecast evaluation exercise. Beginning with the Annual Energy Outlook 1994, the projections in the AEO were produced using the National Energy Modeling System (NEMS). Because NEMS was designed with methodologies to represent technology in a more detailed fashion, there has been an improvement in the capability to represent technological change throughout energy markets. Additional studies on technological improvement have led to more realistic assumptions in the more recent projections, along with modeling innovations, such as the incorporation of learning-by-doing, in which experience gained with new generation technologies and advanced end-use technologies is assumed to lead to cost reductions in the model. These enhancements have significantly improved the projection capability within NEMS.
The last column of Table 1 provides a summary of the average absolute forecast errors for each of the major variables for AEO 1982 through AEO 2002. The calculation of the forecast error is shown in more detail in Tables 2 through 18. The average absolute forecast error is computed as the simple mean, or average, of all the absolute values of the percent errors, expressed as the percentage difference between the Reference Case projection and actual historic value, shown for each AEO, for each year in the forecast, for a given variable. The historical data is typically taken from the AER. However, because the AER was not available for use in the comparison for this year, the 2001 data were taken from the Monthly Energy Review, July 2002.
Table 1 also shows the forecast error from the previous evaluations. As indicated in Table 1, the forecasts of consumption, carbon dioxide emissions, production, and gross domestic product have generally been the most accurate, and the forecasts of prices the least accurate. The percent errors change from one year's evaluation to the next as an additional year of data and projections is added. The percent errors may also change due to data revisions in the Annual Energy Review (AER) and the Monthly Energy Review (MER), which is the source of the historical values used for this comparison.
Relative to the evaluation last year, net coal exports and gross domestic product have higher errors, and world oil prices, natural gas wellhead prices, coal prices to electric utilities, and average electricity prices have lower errors. For the most part, the percent errors remain similar or improves over time; however, the average error for net coal exports has increased significantly. The average absolute percent error increases because of the very high forecast errors in the earlier AEOs that did not foresee the precipitous decline in coal exports experienced since the mid 1990s due to an expectation about the competitiveness of U.S. coal supplies on world markets. An additional large percent error is added to the average calculation each year as another forecast year is picked up from the earlier AEOs. The earlier AEO forecasts have a disproportionate weight in the calculation (more observations) that acts to push up the average absolute error. In this year's evaluation, the forecasts from the earlier AEOs for coal exports in 2001 were added to the calculation. In some years, the coal export forecasts are up to four times the actual level achieved in 2001. The newer AEOs reflect the drop in coal exports and project export totals that are closer to the actual data, but this is not sufficient to offset the impact of the large error in the early AEOs. In the more recent AEOs, the forecasts of coal exports, however, are still optimistic relative to actual experience.