Summary of Advice from the
American Statistical Association (ASA)
Committee on Energy Statistics
Meeting with the
Energy Information Administration (EIA)
April 28 and 29, 2005
Regionalizing the Short-Term Energy Outlook (STEO) Forecast: Margot Anderson, Director, EMEU Plenary session discussed the key features of the Regional STEO model. Topics included: (1) the rationale for regionalizing the model, (2) how EIA approached developing the regional model, (3) key components of the model; and (4) how the regional forecast will compare with the current STEO forecast.
Summary of Advice from the
The
Summary of EIA’s Intended Response to the
EIA intends to follow the Committee advice in the documentation.
STEO Performance Indicators: Diagnostics and Forecast Errors, Margot Anderson, Director, EMEU Plenary session presented and discussed: (1) diagnostic tools for gauging forecast accuracy, (2) which variables make sense and which tools are practical, (3) how important variables and tools are in gauging model and forecast accuracy, and (4) how EIA might share the results with customers.
Summary of Advice from the
The
Summary of EIA’s Intended Response to the
EIA intends to follow the Committee advice in the documentation.
STEO Electricity Modeling: Data Analysis and Model Calibration, Phillip Tseng, SMG, Lead, Dave Costello, EMEU, EIAEIA completed the development of a thirteen-region electricity demand and supply model in early 2005. Subsequently, EIA modelers used the generic least-cost dispatching algorithm to simulate regional generation patterns and fuel choices in meeting a given set of regional electricity demand. Comparison of model results with 2002 and 2003 monthly generation data showed that cost captures only part of dispatching decisions facing electric utilities. And it became clear that modifications to the input assumptions are needed to improve the performance of the model.
This paper documents how EIA data are used to identify and estimate key model parameters that are important to the performance of the model. In addition, this paper will also demonstrate that the calibrated model will be as robust and can provide insightful information on winter heating fuel market, natural gas market, and summer electricity market.
The data used for model calibration include:
Key model parameters EIA modelers have to estimate are:
Summary of Advice from the
The members of ASA Energy Committee commented on the modeling methodology and the trade offs between a simpler and more complicated approach.
One member preferred a simpler approach such as using historical trends and market shares to determine fossil fuel generation. Other members voiced support for a more informed, structured approach in modeling dispatching decisions.
The approach selected depends on how EIA plans to use the model. For fuel switching, a structured model can provide more insight and the results are more tractable. The levels of effort required in model maintenance and data requirements were also mentioned as factors in the methodology selection. Several members indicated that the data seemed massive but the concept of the model is very simple and is very easy to understand.Summary of EIA’s Intended Response to the
Bureau of Census Frames Comparisons, Rick Hough and Vicki Haitot, Bureau of Census
The purpose of the frames evaluation conducted by Census was to determine the coverage of the following five EIA frames: (1) EIA-3, Quarterly Coal Consumption and Quality Report, Manufacturing Plants, (2) EIA-5, Quarterly Coal Consumption and Quality Report, Coke Plants; (3) EIA-860, "Annual Electric Generator Report" (for combined heat and power plants in NAICS codes 31-33 only; (4) EIA-63a, "Annual Solar Thermal Collector Manufacturers Survey"; and (5) EIA-63b, "Annual Photovoltaic Module and Cell Manufacturers Survey.
Two representatives from the Bureau of Census presented the following topics: (1) Methodology for matching and analysis of coverage (both by count and by volume) for all five EIA frames; (2) Results of matching and analysis for EIA-63a (solar thermal collectors), EIA-63b (photovoltaic cells and modules), and EIA-5 (coke plants); (3) Identifying differences between EIA and Census frames which impact coverage analysis; (4) Identifying characteristics of establishments missing on EIA frames; and (5) Identifying next steps for assessing coverage of EIA-3(Manufacturers that consume coal) and EIA-860 (Combined Heat and Power Plants).
Summary of Advice from the
The Committee expressed interest in the results from the final two assessments being conducted. Committee members further suggested that the frame assessments may not be necessary every five years for surveys with small sample sizes. The likelihood of a big dramatic change is probably small for these surveys. Finally, committee members suggested using the first stage of the matching process to assess weather the more detailed second stage is necessary.
Summary of EIA’s Intended Response to the
Results from the two final assessments are expected in August, 2005, and will be made available to EIA. Census will evaluate the need for the frame assessments for surveys every five years for surveys with small sample sizes when the 2007 Economic Census data are available. Census did conduct a feasibility assessment where they looked at the number of cases that could be identified in their database. In the future, this may not be necessary, and Census and EIA may discuss investigating using the first stage as the ASA committee suggested.
Regional STE Model, Propane and Heating Oil Modules, Tancred Lidderdale, EMEU, presented the new regional short-term forecasting modules for propane and residential heating oil. The propane and heating oil modules are structurally similar with reduced-form equations for the
The objective of the regional propane and heating oil modules is to generate short-term monthly forecasts of residential retail prices in the four Census regions. The modules are structurally identical reduced-form models. Regression equations for wholesale prices, regional demands, and regional inventory change are included to support to estimation of residential retail prices.
Regional demands are used to weight regional prices in the calculation of average monthly
The first problem in forecasting regional demands is that EIA publishes only annual residential consumption volumes and does not publish monthly residential consumption data. Regional demand numbers for the model are calculated from unpublished EIA-782C survey volumes. However, these survey volumes are only used for creating State and regional average prices for the Petroleum Marketing Monthly and do not represent reliable measures of actual consumption volumes. Moreover, rebasing of the survey every few years produces shifts in the data series.
The second problem in forecasting regional demands is that the numbers of houses in each region that use each fuel as the primary space heating fuel reported in the EIA Residential Energy Consumption Survey (RECS) have large margins of sample error. For example, the 95 percent confidence interval for the number of homes that use propane for space heating in the northeast is four hundred thousand plus or minus 50 percent. Moreover, the RECS survey is conducted only every four years.
Summary of Advice from the
Discussion at the ASA session focused our attention that the objective of the model was to forecast prices, not demands. Dr. Hengartner suggested that normalizing the numbers such as with a ratio estimate may eliminate some of the artifacts.
Summary of EIA’s Intended Response to the
Following the meeting the regional models were revised by changing the demand series to demand shares. The model documentation now discusses how the calculated regional demand shares are used to weight prices and that while they may produce unreliable regional demand numbers (when multiplied by annual consumption data) they nevertheless yield more reliable price averages. For example, consider two States, one with a price of $1.10 and the other with a price of $1.00. Assume the actual demands in the two States are identical but are each measured with 20 percent error in opposite directions: State A’s measured demand share is 0.6 and State B’s is 0.4, rather than 0.5 each. While the error in the demands is large, the error in the volume-weighted price average is small: $1.04 when demands are measured with error and $1.05 when demands are correct.
Hands-On Usability Testing of EIA's New Website Design: Can You Find What You're Looking For? Colleen Blessing,
Working since last fall, and based on input from many sources, the Web Redesign Group created a new prototype design we wanted
EIA conducted a similar usability test with ASA members before our redesign in 2000 and received useful feedback. There was no preparation necessary for this session, except if Committee members wanted to record some reports or data tables that they would like to be able to find on the new site. Looking at the new design for the first time in this session allowed EIA to capture Committee members’ first impressions.
Summary of Advice from the ASA Committee
The ASA committee members were asked to evaluate the effectiveness and understandability of a proposed new web page design by EIA's Cognitive Evaluation and Usability team. Eleven
Some specific areas that tested well were the new link names on the home page, hints under the major bin names, information in the new right side bar and the second-level pages.
Some specific areas that did not test well were finding information inside specific publications, finding the glossary, the need for integrated presentation of information (having both history and forecasts links available from the same second-level page) and the link name Energy Overview.
EIA collected much useful information about the usability and functionality of the new design. The overall conclusion from the ASA committee members is that they liked the new design and thought we were on the right track.
Summary of EIA's Intended Response to the ASA Committee Advice
Following the ASA tests, EIA conducted usability tests with 25 additional users, both internal EIA staff and external users from various customer groups. Feedback from all these tests is being used to modify and improve the proposed new design. Watching users actually work on the site gives direct messages about what they see, what they understand, and what they use. Items that tested well will be kept; items that don't test well will be dropped or changed.
Follow-up on Frames Team Activities, Howard Bradsher-Fredrick, SMG, EIA This session presented a summary of the activities and conclusions of the various incarnations of the frames team from March 2004 through April 2005. The frames team was an inter-office EIA team composed of staff members from the EIA program offices, SMG and OIT. In general, all interested and affected offices had representation on each incarnation of the team
The impetus for the effort on improving frames emanated from the EIA Strategic Plan and specifically from the Strategic Information Technology Subgroup. These frames teams were charged with identifying efficiencies EIA can pursue regarding frames and the computer systems that support frames.
In particular, the first team was chartered with providing recommendations concerning activities in the frames area that will lead EIA toward saving resources, facilitating more commonality in data and software, improving processes, and resulting in frames and frame information that can easily be shared within EIA. The first frames team met from March 2004 to early May 2004 and developed a set of six recommendations.
The second incarnation of the team met from August 2004 to January 2005. The primary focus of its activities was to assess each unique EIA survey frame for “sufficiency.” In order for the team to ascertain sufficiency or insufficiency for individual frames, a number of tasks had to be accomplished, some of which involved extensive survey frame information gathering.
The third incarnation of the team began meeting in February 2005. It is charged with following up on the recommendations put forward by the first incarnation of the team.
Summary of Advice from theThe
The committee also questioned whether or not it was a worthwhile exercise to evaluate frames as often as once every three years. It may be a more useful utilization of EIA’s resources to evaluate frames on a less frequent basis. The committee expressed no opinion on the question posed by the presenter, that of whether each survey frame should be evaluated as opposed to each master frame.
Summary of EIA’s Intended Response to the
EIA appears to be caught between the advice given by the committee and the performance measurement evaluations required by the Department. The committee does not believe that it is useful for EIA to evaluate all frames for sufficiency nor does it believe that sufficiency evaluations should be taken on as often as every three years. This seems to be in direct conflict with the Department’s request to evaluate all frames for sufficiency on a regular periodic basis. EIA will consider the advice given by the committee prior to comprehensively evaluating frames in the future. This consideration will include both the issue of frequency of evaluation and whether all frames should be evaluated or only those that are most likely to be problematical (e.g., those in the greatest state of flux, those that are relatively new).
EIA’s Proposed Strategy for Addressing Declining Response Rates in the Residential Energy Consumption Survey (RECS), Dwight French, EMEU, EIA Over the 25-year history of EIA’s Residential Energy Consumption Survey (RECS), response at various stages of this survey system has slowly but steadily decreased. This trend, which has been true for most government household surveys, is especially troublesome for RECS, which is actually a sequence of interconnected data collections: a survey of the household to get its energy-related characteristics, a request during the household survey to get the household respondent to sign a waiver form allowing the household’s energy suppliers to submit billing information, and then a survey of the suppliers to collect the energy consumption and expenditures information. All of these collections are necessary to get a complete set of survey data; therefore, non-response at each stage has a cumulative negative effect on data quality.
This presentation described an aggressive, multi-faceted strategy that EIA planned to address the comprehensive non-response issue, including additional and updated preliminary contacts, an incentive experiment, enhanced non-monetary incentives, the use of scanners to collect utility bill information at the household, a Spanish translation of the questionnaire, publicity about data confidentiality based on EIA’s new CIPSEA legislation, and a potential non-response sub-sampling process. The committee was asked to critique the approaches and suggest any others they might think would be useful.
Summary of Advice from and Intended Response to the
The
1. Advice: A question was asked about the justification for our proposed incentive levels. What did we do to determine that these were appropriate levels, in that they would be sufficient to change householders' minds about responding? Could we ask a question of non-respondents to the survey as to what level of incentive WOULD be sufficient to make them respond?
Response: EIA did not do any official research study to determine appropriate incentive levels. We were constrained by our financial resources, the fact that OMB wanted an experiment to test different incentive levels, and the concern that incentives that were too large might turn people off as an example of inappropriate Federal largesse. EIA did review the literature on other incentive experiments and payments. Our proposed incentives fit into the broad range of what has been done by others, but we have nothing that indicates that our incentives can be expected to be optimum in some sense. And even if one of the incentive levels were to turn out to be "optimum", that would mean that the other levels aren't. EIA will, of course, study the effects of the incentives after the RECS interviewing is complete. But EIA will not ask the question about how much it would take for a non-respondent to be persuaded to respond. The government does not reimburse people to provide information - we will be very careful to tell respondents that the incentive is a token of thanks, not a payment for information (which is why the incentive is unconditional). Asking a non-respondent how much it would take for them to respond is tantamount to suggesting that the incentive is a reimbursement.
2. Advice: The committee suggested that if EIA were doing an incentive experiment, the selection of categories should NOT be associated with household income, because one doesn't want to have an income effect confounded with the incentive category effect.
Response: EIA's response is that the incentive amounts will be randomized across segments, but each household in a segment will get the same incentive (avoiding the jealous neighbor effect). We do not plan to have household income as any part of the design for assigning incentive categories to segments.
3. Advice: A comment was brought up about using priority mail or some other attention-catching mailing for advance information, to emphasize the importance of the survey. Reaction to this was mixed - some thought that priority mail has been over utilized, and used for junk-type mailings in some cases, so that its effect has been diluted. But there has been some evidence that attention-getting mail can improve response at least slightly.
Response: EIA's response is that only advance documents will be dealt with by mail, so it isn't as if we're doing a mail survey. However, we will use the best mailing that budgets allow.
4. Advice: A suggestion was made that if we try to get energy data from a household, we get it for only a month rather than the entire reference year.
Response: EIA's response is that we intend to ask the households, in the advance letter, to have their most recent energy bills available for the interviewer. We will scan those. However, we may be able to get a year's worth of consumption data from some of these bills, because certain utilities are now posting the previous 12 months' consumption on the bill to give the household a more complete picture of its energy needs. If we can get a year's worth of consumption data from many of these bills, it will be an enormous help in cases where the supplier cannot be reached, refuses to cooperate, or cannot provide the information.
5. Advice: In response to an EIA question about how to compute and quote the survey response rate if non-response sub-sampling were employed, there seemed to be strong feeling from the committee members that it was misleading and statistically improper to weight up the non-response cases to compute the survey response rate rather than simply taking the un-weighted final number of respondents/the number of in-scope sample cases to be the response rate. It is okay to use the non-response sub-sampling weights to compute parameter and variance estimates. In fact, you'd really need to do so to avoid bias, unless the initial non-respondents prior to sub-sampling are found to be no different from the original respondents when the eventual responses are compared to the responses from the original respondents. But don't use it to compute a weighted response rate.
Response: EIA's response is that we might compute both response rates -- weighted and un-weighted, but that we would not avoid citing the un-weighted response rate. It should be noted that non-response sub-sampling is NOT a given for the 2005 RECS. The process will be used ONLY if the initial response is sufficiently low that sub-sampling is necessary to improve estimation and keep control of non-response conversion costs. EIA's and its fieldwork contractor (NORC's) plan is that the survey process will be successful enough (achieving a minimum of 80% response within the budget for fieldwork) that non-response sub-sampling will not be needed.
Assessments: Presentations and a Panel Discussion (Two Parts),
Part I. External Review of Survey Programs: A Progress Report, Brenda Cox, Battelle, under contract to EIA, and
Summary of Advice from the
The Committee was concerned that EIA’s program of activities should involve more than programs of surveys. There are also extensive modeling and data analyses activities occurring that also need to be considered. The Committee was interested in the total number of programs that EIA might have. Finally, the Committee was concerned about the exclusive use of publicly available documentation for the survey and program evaluations, with no staff interactions.
Summary of EIA’s Intended Response to the
For this trial evaluation, we have meet with Petroleum Marketing Program staff and gotten their reactions to the draft evaluations. In the Methods Report documenting the results of this investigation, we will recommend staff interaction in eliciting input for the evaluations and for reacting to the draft evaluations.
The overarching question that the external study team will address is:
Given continuing tight EIA budgets and other external drivers, is EIA doing the “right things” now, and what are the “right things” it should do in the next five years?
The question intentionally focuses on “is EIA doing the right things” (i.e., what products and services are provided to customers) and not on “is EIA doing things right” (i.e., how it conducts its internal business processes). At the ASA session, EIA reviewed its goals for this external study, outlined project’s tentative timeline (scheduled to be completed in November 2005), and solicited ideas and suggestions from ASA Energy Committee members who may wish to have shared their experience with evaluative processes that resulted in positive change.
Summary of Advice from theIndividual members of the Committee suggested that: (1) EIA perform a “self study,” answering the study questions before posing them to the external tests, (2) that the current outline was a “core dump” and that it would not help the external team understand everything about EIA, (3) that it would be difficult to completely isolate the question, “Are we doing the right things?” from the issue of “how” we do things, (4) that the relationship between EIA staff and contractors was often very cozy, that the contractors were not always executing their work up to current industry standards in econometric and statistical methods and suggested the review include whether we are using the best methods for the work that we do , (5) that past recommendations by similar studies were not accepted or acted upon; (6) that it is not possible for a high-level review to work without delving into the details; (7) that the balance of the Study Team (mix of industry, energy sector, and data/forecasting/analysis interests) was not equitable, and (8) that the issue of tradeoffs be more explicit.
Summary of EIA’s Intended Response to the
The EIA External Study Support Team (ESST) received, analyzed and responded to all recommendations received by the ASA committee by either: 1) incorporating the suggestion directly into its planning process, or 2) referring the matter to EIA Senior Staff.
EIA’s response will include (but not be limited to): (1) an EIA support team discussion of options for conducting an internal EIA self-study; (2) EIA support team consideration of revamping the Overview document; (3) EIA support team recognition of the difficulty in totally separating the right thing from doing things right; (4) EIA support team will recommend balance of study team (mix of industry, energy sector, interests, geography, etc); (5) EIA support team review of statements from the OMB PART report that EIA did not do well on, in particular, a) “Regular independent evaluations used to support program improvement, assess effectiveness and relevance” and b) “Independent evaluations indicate program effective and achieving results”; (6) EIA support team consideration of recommendations to support program improvement, and to assess relevance; (7) EIA support team inclusion of words about “effectiveness, and achieving results” in the Overview document; (8) EIA support team will better enunciate that along with “Outputs” the Study team will be asked to think about “Outcomes”. Outputs alone are too narrow; (9) EIA support team consideration of the relationship between EIA staff and contractors; (10) EIA support team recommendation to Senior staff that a focus on HOW we do business be a focus of a future review; (11) EIA support team will ask the external study team to establish priorities for their recommendations; and, (12) EIA support team recognizes that comments on the inadequacies of the EIA website may be reflected in this study.
Discussion: Where Next?, Howard Gruenspecht, EIA Deputy Administrator and Moderator
Progress on EIA’s 914: Response rate and kinds of challenges, John Wood, Reserves and Production Division (RPD), EIA At the fall 2004 meeting with the ASA Energy Committee, EIA introduced the new and approved Natural Gas Production Survey, Form EIA-914. The spring 2005 meeting will provide the first opportunity to discuss EIA’s preliminary assessment of the survey data for January, 2005, the first reporting month for this new survey. EIA will discuss the challenges faced during the early months of survey operation, as well as our plans for review, analysis, and comparisons of final natural gas production estimates based on the early sample of final production provided by the EIA-914 survey data with our previous methods of estimating final natural gas production. This review, comparison, and analysis is a critical step in the process of deciding when EIA should start publicly releasing the EIA based production estimates as our official EIA natural gas production estimates.
Summary of Advice from the
The first major point made in the ASA discussant summary was that the EIA-914 survey was a better way to provide natural gas production estimates; and the response rates, through a lot of effort and contact and follow-ups with the producers themselves, seemed to be quite impressive and the survey results look very good at this time. In response, EIA is proceeding to move the EIA- 914 estimates through the clearance process, to make the estimates public, and then make them the official EIA natural gas production numbers.
The ASA thought that the high response rates would inspire public confidence in the EIA-914-based natural gas production estimates and that EIA should describe for the public the follow-up we performed with operators to achieve the high response rates. In response, EIA will strive to maintain high response rates in the future, publicize the high response rates when the data are publicly released and prominently emphasize to the public the quality assurance and follow up with operators that seemed reassuring to the
Capitalizing on the operator level data now available, a formal imputation system that estimates monthly production for non-responding operators based on previously submitted data has been put in place. We expect it will get more reliable over time as a longer reported data series is collected.
Estimate EIA’s 826 Since the Last Time: Differences, Estimation Groups, Outliers and Test Results, Joseph Sedransk, Jim Knaub, Lindolfo Pedraza and
EIA Form-826 collects information, monthly, from regulated and unregulated companies that sell or deliver electric power to end users. It collects State-level sales volumes, sales revenues, and number of customers by end-use sector (residential, commercial, industrial, and total). The existing sample and methodology to estimate population totals were described at the fall 2004 meeting of the Energy Committee. This session will describe current efforts to form homogeneous subpopulations for estimation, and evaluations based on predictions using annual data (where truth is assumed known). We assess the importance of outlier detection, under alternative scenarios. We will be asking for advice on next steps for analysis, and on how to frame convincing recommendations for implementation.
Summary of Advice from the
The ASA committee recommended that two of the summary statistics that were presented were the more useful ones; i.e., (1) the coefficient of variation, and (2) one of the per cent error quantities (per cent error in estimating the non-sampled companies in the stratum or per cent error in estimating the stratum total).
A second recommendation was to evaluate the alternative choices (e.g., outlier detection or not, use of IOUs in estimation or not) using monthly data rather than the annual data used in constructing the tables shown at the meeting.
There were no recommendations concerning alternative methods of forming the post-strata for estimation.
Summary of EIA’s Intended Response to the
EIA will emphasize the two summary statistics noted above. An evaluation using monthly data has been planned, but has not yet been implemented.