Second Team Charter
for
NWS Hydrologic Forecast Verification
October 1, 2009

Vision: River forecast verification tools will be readily available to various users including forecasters, service hydrologists, managers, and the general public to produce verification information that is meaningful to each user group. Standard strategies and products for hydrologic forecast verification will help answer the following questions: 1) How good are the forecasts? 2) What are the strengths and weaknesses of the forecasts? 3) What are the sources of uncertainty and error in the forecasts? 4) How are new science and technology improving the forecasts? 5) What should be done to improve the forecasts? RFC forecasters and modelers will systematically generate and communicate river forecast verification results and identify, based on routine verification, how the forecasting system may be improved to meet the evolving service needs. Ultimately, forecast verification is considered successful when its results are used by forecasters and modelers to guide improvement of the forecasting system and the forecast process, and by users to maximize the utility of forecast information in their decisions.

Statement of the Problem: Currently, information on NWS river forecast performance is limited in scope and generally not communicated to most user groups. In recent years, nationally-supported verification software has been developed, which will be integrated in the near future into the CHPS Verification Service (CHPS-VS). Also verification case studies have been carried out at all RFCs and recommendations on standard verification metrics and products, as well as verification analyses, have been developed by this team as described in the September 09 team report (available at http://www.nws.noaa.gov/oh/rfcdev/docs/NWS-Hydrologic-Forecast-Verification-Team_Final-report_Sep09.pdf). For verification to become a routine operation at the RFC, these verification standards need to be field-tested and further evaluated for improvement by the RFCs with verification case studies. Also, in order for CHPS-VS to meet all user needs, more detailed analysis of user requirements is needed.

Mission: Carry out the following to communicate meaningful river forecast verification information to user groups including forecast users, forecasters, service hydrologists, and managers using existing software (IVP and EVS) and to support the development of CHPS-VS:

1. Generate and evaluate the standard verification metrics and products recommended in the September 09 team report with new RFC verification case studies;
   
2. Perform detailed user analysis of the verification products in collaboration with the RFC Service Coordination Hydrologists and OCWWS/HSD and develop requirements for disseminating verification information for RFC river forecasts to users;
   
3. Support the design and development of CHPS-VS by providing input and reviewing user requirements and software design documents and by testing prototype functionalities (e.g., EVS).

Success Criteria: The team will develop a verification report by September 30, 2011 that describes improved standard verification metrics and products, as well as RFC case studies using these standards. The RFC case studies will include the analyses of the impacts of the QPF horizon and the run-time modifications made on the fly on the quality of river stage forecasts. To accomplish this, each RFC will develop a report on their verification case study to show the quality of the forecasts under different conditions and/or for different forecasting scenarios, and the value of verification for them and for their forecast users. The team leader will describe verification analyses of experimental ensemble forecasts for various RFC test basins to evaluate new science. All the verification case studies will include a discussion on how verification helps guide improvement of the forecasting system and the forecast process. The team will also coordinate with the RFC Service Coordination Hydrologists (SCHs) and OCWWS/HSD to define which standard verification products should be disseminated by the NWS Performance Branch and by the RFCs. The team will deliver the prototype capabilities developed at OHD and at the RFCs to produce the standard verification products using IVP, EVS, and the CHPS display capability (provided by the Graphics Generator, the FEWS Spatial Display, and the FEWS Time Series Display) and to disseminate the verification information along with forecast products to users.

Scope of Authority / Limitations: The team will normally meet every two/three months via teleconference. Team members will share their progress on the verification case studies and on the prototype capabilities to produce standard verification products using existing software. The team will also evaluate the recommended standard metrics and products and improve them if necessary. The team will collaborate with the RFC SCHs and OCWWS/HSD to perform a more detailed user analysis of the verification products needed for all user groups. The user analysis may require sub-groups of verification experts (including experts from the NWS Performance Branch and outside the NWS), OCWWS experts, and RFC SCHs to focus on specific forecast users. The team will develop requirements for disseminating verification information for RFC river forecasts to users. The team will also review user requirements and software design documents for the CHPS Verification Service and test prototype functionality to report required enhancements. A verification workshop may be organized in FY 2011 to share progress on verification science, software and case studies made in the NWS, other agencies, and academia. The team should realize the success criteria defined here no later than September 30, 2011.

No travel beyond a RFC verification workshop is authorized for these goals.

Proposed Team Membership: Team will be comprised of verification focal points and co-focal points from the 13 RFCs.
     Julie Demargne (OHD/HSMB) - lead
     Ernie Wells (OCWWS/HSD)
     Larry Lowe (ABRFC verification focal point)
     James Coe (APRFC verification focal point)
     Ashley Nielson (CBRFC verification focal point)
     Kevin Werner (CBRFC verification co-focal point and SCH coordinator)
     Alan Takamoto (CNRFC verification focal point)
     Kai Roth (LMRFC verification focal point)
     Bill Marosi (MARFC verification focal point and NWSEO representative)
     Andrew Philpott (MARFC verification co-focal point)
     Julie Meyer (MBRFC verification focal point)
     Holly Reckel (NCRFC verification focal point)
     Tom Econopouly (NERFC verification focal point)
     Steve King (NWRFC verification focal point)
     Tom Adams (OHRFC verification focal point)
     Christine McGehee (SERFC verification focal point)
     Greg Waller (WGRFC verification focal point)

Verification software experts in OHD will serve as technical advisors to the team:
     James Brown (OHD/HSMB)
     Yuqiong Liu (OHD/HSMB)
     Hank Herr (OHD/HSEB)

Proposed Schedule:
The schedule reflects the need for the RFCs to work on the CHPS implementation first. Some of the recommended verification products need to be produced using the CHPS display capabilities, whereas IVP and EVS can be used outside CHPS to produce a subset of the verification products. Therefore the schedule would be different for the CAT RFCs and the CAT-II RFCs. First the CAT RFCs would start developing the verification standards using the CHPS display capabilities, and share their progress with the CATII RFCs. The CATII RFCs would develop the standard verification products using CHPS when their CHPS implementation is being finalized (~late FY 2010).
October 2009: Finalize the second team charter.
October 2009 – June 2010: Perform user analysis to identify user requirements for CHPS-VS. June 2010: report due on forecasters’ and users’ feedback on standard verification products to be disseminated along with forecast products.
April – September 2010: Define requirements for the NWS Performance Branch and the RFCs to disseminate RFC verification information.
October 2009 – April 2011: Perform RFC verification case studies using IVP or EVS software as well as CHPS capabilities to test and improve the verification standards; continue to review/test new prototype functionality for CHPS-VS (e.g., EVS Version 2.0).