by 
Susan H. Holte

OIAF has been providing an evaluation of the forecasts in the Annual Energy Outlook (AEO) annually since 1996. Each year, the forecast evaluation expands on that of the prior year by adding the most recent AEO and the most recent historical year of data. However, the underlying reasons for deviations between the projections and realized history tend to be the same from one evaluation to the next. The most significant conclusions are:

• Natural gas has generally been the fuel with the least accurate forecasts of consumption, production, and prices. Natural gas was the last fossil fuel to be deregulated following the strong regulation of energy markets in the 1970s and early 1980s. Even after deregulation, the behavior of natural gas in competitive markets has been difficult to predict.

• Energy prices have been far more difficult to predict than consumption, production, and net imports, and prices have been more typically overestimated than underestimated. More rapid technological improvements, the erosion of the market power of the Organization of Petroleum Exporting Countries in the mid-1980s, excess productive capacity, and market competitiveness are all factors that led to lower energy prices than projected. In the 1980s and 1990s, productivity and technology improvements and the effects of gradual deregulation and changes in industry structure have more than offset the factors that have tended to raise energy prices, such as resource depletion and increasing energy demand.

• Over the last two decades, there have been many significant changes in laws, policies, and regulations that could not have been anticipated or assumed in the projections. These include the National Appliance and Energy Conservation Act of 1987, the Natural Gas Wellhead Decontrol Act of 1989, the Clean Air Act Amendments of 1990, the ozone transport rule, the Energy Policy Act of 1992, the repeal of the Power Plant and Industrial Fuel Use Act of 1978, the North American Free Trade Agreement, the Omnibus Budget Reconciliation Act of 1993, the Outer Continental Shelf Deep Water Royalty Relief Act of 1995, the Tax Payer Relief Act of 1997, the Climate Change Action Plan developed by the Clinton Administration in 1993 to achieve stabilization of greenhouse gas emissions, various orders issued by the Federal Energy Regulatory Commission on natural gas restructuring and open access to interstate electricity transmission lines, and various state initiatives for the restructuring of electricity markets. These actions have had significant impacts on energy supply, demand, and prices; however, the impacts were not incorporated in the AEO projections until their enactment or effective dates in accordance with the mandate that EIA remain policy neutral and the practice that AEO projections include only current laws and regulations.

• External factors such as severe weather, economic cycles, and strikes have also had an impact on energy markets; however, these events cannot be anticipated in the mid to long-term period and are not captured in the models underlying the AEO projections.

• Technological improvements in both the production and use of energy have had a significant impact on the price, supply, and consumption of energy. For the most part, earlier AEOs assumed much slower technology development than actually occurred, accounting for some of the deviation between the forecasts and history. This trend was recognized, in part, by this type of forecast evaluation exercise. Beginning with the Annual Energy Outlook 1994, the projections in the AEO were produced using the National Energy Modeling System (NEMS). Because NEMS was designed with methodologies to represent technology in a more detailed fashion, there has been an improvement in the capability to represent technological change throughout energy markets. Additional studies on technological improvement have led to more optimistic assumptions in the more recent projections, along with modeling innovations, such as learning-by-doing, in which experience gained with new generation technologies leads to cost reductions in the model. These enhancements have significantly improved the projection capability within NEMS.

Because the underlying reasons for differences between the projections and realized history do not generally change from one year to the next, the text of last year's analysis was not updated this year. The 1999 forecast evaluation with the complete narrative is available at www.eia.doe.gov/oiaf/issues/forecast.html.

The last column of Table 1 provides a summary of the average absolute forecast errors for each of the major variables included in this analysis, which are shown in more detail in Tables 2 through 18. The average absolute forecast error is computed as the mean, or average, of all the absolute values of the percentage errors, expressed as percentage differences of the reference case projection from actual, shown for each AEO, for each year in the forecast, for a given variable. Table 1 also shows the same summary of forecast errors from the four previous evaluations. As indicated in Table 1, the forecasts of consumption, carbon dioxide emissions, production, and gross domestic product have generally been the most accurate, and the forecasts of prices the least accurate. The percent errors change from one year's evaluation to the next as an additional year of data and projections is added.

Table 1. Average Absolute Percent Errors from AEO Forecast Evaluations: 1996 to 2000