Evaluation of AEO2018 and Previous Reference Case Projections
The U.S. Energy Information Administration (EIA) produces projections of U.S. energy production, consumption, and prices each year in the Annual Energy Outlook (AEO). The AEO is produced on an alternating cycle: a full AEO (including complete documentation and a large number of alternative cases) followed by a shorter AEO (more limited documentation and only a few alternative cases). Each year that a full AEO is released, EIA also produces the AEO Retrospective Review (AEO Retrospective). The AEO Retrospective compares recent history with the Reference case projections from previous editions of the AEO. The AEO Retrospective shows the relationship between past AEO projections and actual energy indicators, and it informs discussions about the underlying models.
The projections presented in the AEO are not statements of what will happen but of what may happen given the assumptions in the underlying National Energy Modeling System (NEMS). These assumptions include projections of oil prices and gross domestic product (GDP) and are publically available. [1] The AEO Reference case projection generally assumes that other trends are consistent with historical and current market behavior, technological and demographic changes, and current laws and regulations.
Although the AEO Retrospective focuses on the Reference case projections, readers are encouraged to review the full range of cases in the AEO, which illustrate the uncertainty inherent in long-term projections. Beyond the Reference case, the discussion and analysis in each year’s AEO typically include Low and High Economic Growth cases and Low and High Oil Price cases. Recent AEOs have also included alternative cases examining the impact of proposed policy changes, variation in the rate of technology improvement, and different assumptions about the size of energy resources.
The AEO Retrospective summarizes the relationship between the Reference case projections and realized energy usage by comparing absolute differences and the average absolute percent differences for many of the major measures published in AEOs from 1994 to 2018. [2] The average absolute percent difference is the simple mean of the absolute values of the percentage difference between the Reference case projections and the actual values. Most historical data are from the Monthly Energy Review (MER), which can be accessed from the EIA open data API. [3]
Table 1 provides a summary of 25 comparisons based on 21 primary measures projected in the AEO. Table 2 contains the standard deviations of the percentage forecast errors (between the Reference case projections and the actual values) for every horizon H=0 to H=15 computed with AEOs from 1994 to 2018. [4],[5] For price quantities, the standard deviations of the log errors were computed. Tables 3 through 23 show the detailed results that include the Reference case projections, historical values, and percent differences between projected and actual values for all covered years. The detailed tables also provide the average absolute difference across all AEOs for each covered year. Previous-year statistics can change from one year’s evaluation to the next because of updates to historical data in the MER and changes in the measurement of GDP. The statistics in Table 1 do not account for the continual improvements made to NEMS in an attempt to more accurately capture developing energy market trends based on more recent experience.
As reflected in the relatively lower average absolute percent differences in the second column of Table 1, measures of energy consumption are projected with greater accuracy than measures of net imports and of energy prices. Energy consumption tends to change at a slower pace than other indicators; many energy-using devices, such as appliances, automobiles, and industrial equipment, are expensive to purchase and have long service lives. The substantial lead time for these large purchases, along with the effects of energy contracts and other mechanisms, tend to slow changes in consumer energy demand.
For the measures with wider deviations, many factors contribute to differences between the AEO Reference case projections and realized outcomes, but two primary contributors are the initial projections of future oil prices and the overall economic activity that are used in NEMS. [6] These projections can greatly influence the other projections the model makes, which is why each recent AEO includes alternative cases exploring differences in economic growth (Low and High Economic Growth cases) and in oil prices (Low and High Oil Price cases).
In an unbiased projection, with a sufficiently large number of samples, over- or under-estimates over time would occur in equal measure. The third column of Table 1 provides a summary of the percentage of occurrences in which a particular measure was overestimated relative to its actual value out of the total number of annual projections for which an actual value can be compared. A zero represents a measure that was always underestimated, and 100% represents a measure that was always overestimated. In comparing the AEO Reference case projections with realized outcomes from 1994–2017, out of the 25 variables listed in Table 1, projections were overestimated for 15 variables.
Changes in industry-specific market conditions, significant technological breakthroughs, and new laws or regulations subsequent to the publication of an AEO can also lead to differences between projections and realized outcomes. For example, AEO Reference case projections before AEO2018 projected the United States to continue to be a natural gas net importer in 2017.
| Variable | Average absolute percent differences | Percent of projections overestimated |
|---|---|---|
| Gross domestic product | ||
| Real gross domestic product (average cumulative growth) [1] (Table 2) | 0.9 | 55.7 |
| Petroleum | ||
| Imported refiner acquisition cost of crude Oil (constant $) (Table 3a) | 41.5 | 25.1 |
| Imported refiner acquisition cost of crude Oil (nominal $) (Table 3b) | 40.8 | 26.4 |
| Total petroleum consumption (Table 4) | 9.5 | 72.6 |
| Crude oil production (Table 5) | 14.3 | 38.8 |
| Petroleum net imports (Table 6) | 59.5 | 79.2 |
| Natural gas | ||
| Natural gas price, electric power sector (constant $) [2] (Table 7a) | 36.0 | 47.6 |
| Natural gas price, electric power sector (nominal $) [2] (Table 7b) | 36.9 | 49.8 |
| Total natural gas consumption (Table 8) | 8.4 | 62.5 |
| Natural gas production (Table 9) | 8.6 | 48.9 |
| Natural gas net imports (Table 10) [3] | 292.4 | 73.6 |
| Coal | ||
| Coal prices to electric generating plants (constant $) [4] (Table 11a) | 20.0 | 35.8 |
| Coal prices to electric generating plants (nominal $) [4] (Table 11b) | 19.9 | 39.1 |
| Total coal consumption (Table 12) | 19.6 | 75.6 |
| Coal production (Table 13) | 16.3 | 76.5 |
| Electricity | ||
| Average electricity prices (constant $) (Table 14a) | 10.4 | 28.3 |
| Average electricity prices (nominal $) (Table 14b) | 11.2 | 34.2 |
| Total electricity sales (Table 15) | 6.2 | 64.2 |
| Total energy, carbon and intensity | ||
| Total energy consumption (Table 16) | 8.2 | 82.1 |
| Delivered residential energy consumption (Table 17) | 6.6 | 73.6 |
| Delivered commercial energy consumption (Table 18) | 6.8 | 55.7 |
| Delivered industrial energy consumption (Table 19) | 11.6 | 83.4 |
| Delivered transportation energy consumption (Table 20) | 9.7 | 79.8 |
| Total energy related carbon dioxide emissions (Table 21) | 11.8 | 76.2 |
| Energy intensity (energy consumption / real $ GDP) (Table 22) | 9.1 | 84.4 |
| AEO – Annual Energy Outlook. Source: These statistics summarize the calculations in Tables 2 through 22. The data in Tables 2 through 22 are based on the 1994 through 2018 AEO Reference case projections. Historical Data are from the U.S. Energy Information Administration open data API (http://www.eia.gov/opendata/) (Washington, DC, July 2018), with the series listed under each appendix table, except for GDP data which are from the Bureau of Economic Analysis, U.S. Dept. of Commerce, September 2014, http://www.bea.gov/national/xls/gdplev.xls. 1. The basis for GDP comparison is the projection differences in the cumulative average growth rate of real GDP from the first year shown for each AEO. The summary information for projection differences given for GDP growth rates is absolute percentage point differences; for all other AEO concepts, the comparison basis is absolute percent differences. 2. As of 2013, the wellhead price of natural gas was no longer reported by EIA. With the 2015 edition of the Retrospective, the natural gas price to the electric power sector replaced the wellhead price. 3. As natural gas net imports have approached zero, the average absolute percent difference has increased significantly. 4. Beginning in AEO2003, EIA electric generating projections incorporated combined heat and power (CHP) electricity generation in electricity generating plants. Before AEO2003, coal price projections reflected data collected, estimated, and reported to electric utilities and excluded CHP power generation. |
||
Notes
1. NEMS documentation is available on the EIA website at: https://www.eia.gov/outlooks/aeo/nems/documentation/
2. The National Energy Modeling System has been used to prepare the AEO since 1994. This publication considers only the projections made after 1994. In addition, the Annual Energy Outlook 2009 results are from “An Updated Annual Energy Outlook 2009 Reference Case Reflecting Provisions of the American Recovery and Reinvestment Act and Recent Changes in the Economic Outlook,” which is available on the EIA website at: https://www.eia.gov/analysis/requests/archive/1999/stimulus/. Further, projections in the 1994 and 1995 AEOs ended in 2010, so entries for these two publications are blank for the later years in Tables 2 through 22.
3. The exceptions include historical data related to Gross Domestic Product from the U.S. Department of Commerce, Bureau of Economic Analysis (BEA).
4. The methodology used is derived from Kaack, et al. “Empirical prediction intervals improve energy forecasting,”, Proceedings of the National Academies of Sciences, 114(33) published August 15, 2017.
5. H is defined as the number of years since the first projection year, which varies by AEO (e.g., H=0 corresponds to the year 2017 for AEO2018 and to the year 2016 for AEO2017).
6. Although dynamic feedback in the model can modify these initial forecasts, the resulting changes tend to be small.