About Forecast Verification
Water supply forecast verification is an increasingly demanded product. The verification section a way to examine forecasting skill in water supply basins. Several agencies produce water supply forecasts, and these forecasts can be displayed with this product. The data source option allows the user to select the agengies to compare, including: National Weather Service, Natural Resource Conservation Service, California Department of Water Resources, Statistical Water Supply, and Ensemble Streamflow Prediction in a variety of distributions. There is also a coordinated forecast value that is the official water supply forecast value that is agreed upon by the National Weather Service and Natural Resource Conservation Service or California Department of Water Resources.
Forecasts are produced in water supply areas before runoff begins. The forecast issuance date is the month and year the forecasts were created while the forecast period is the span the forecast is valid. The observations are then summed over this span for comparison. A threshold value is also established for comparison with the historical mean, the average streamflow from the historically defined period (1971-2000), as a default. This value can be changed to compare any relevent numbers.
The right-hand menu allows users to select the verification statistic, data sources, time scale, threshold, and graph options. A scroll bar will apear below graphs that require more width to adequately view the data. The plots display the statistics in a combination of year and month formats. Statistics by year average the issuance month while the statistics by month are averaged by the years. The statistics by month are also referred to as lead time or how far in advance the forecast was issued before the valid forecast period.
![sample plot of historical time series](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/historical.png)
The historical time series plot displays the forecast progression of water supply forecasts. For the selected agencies, forecasts are displayed, in 1000s of acre feet (KAF), by issuance month and year for the selected forecast period. The observed streamflow period of record maximum (green) and minimum (red) are displayed as well as the threshold value (black) and observed streamflows (blue). The forecast range is displayed with the official, 50% exceedance value (circle), 90% exceedance value (triangle), and 10% exceedance value (inverted triangle).
![sample plot of streamflow histogram](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/streamflow.png)
The streamflow histogram displays a bar graph of historical observed streamflow for the selected forecast period. This plot can be manipulated by using the graph options. By default, the observed streamflow for the forecast period is displayed in year order of the selected years. The graph options allow you to sort the observed streamflows from lowest to highest, display the full historical set of observed streamflows, and view individual months in the forecast period or all individual months with the summed period streamflow. The observed streamflow period of record maximum (green) and minimum (red) are displayed along with the threshold value (black).
![sample plot of scatterplot](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/scatter.png)
The streamflow scatterplot displays the observed versus forecast streamflow for the selected years and forecast period. Each forecast issuance month is plotted against the observed streamflow. The ideal scatterplot would fall on the diagonal line, when the forecasted streamflow falls very close to the observed. Under graph options, the plot can display individual forecast issuance months. When individual months are displayed, the year will be displayed in the color of the data source instead of the symbol.
![sample plot of Mean Absolute Error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/maem.png)
The mean absolute error by month (lead time) is displayed for the selected agencies as well as the threshold value for each forecast issuance month of the forecast period. The mean absolute error is the average of the absolute differences between the forecast and observed streamflow for each month. The MAE of the threshold value assumes forecasting the same streamflow each month of every year. The ideal MAE is 0. By using the graph options, the forecast issuance months can be broken down and the MAE can be displayed by year within each month.
![sample plot of mean](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/maem_eq.png)
![sample plot of mean absolute error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/maey.png)
The mean absolute error by year is displayed for the selected agencies as well as the threshold value for each year for the forecast period. The mean absolute error is the average of the absolute differences between the forecast and observed streamflow for each year. The threshold value assumes forecasting the same streamflow each month of every year. The ideal MAE is 0. By using the graph options, the years can be broken down and the MAE can be displayed by forecast issuance month within each year.
![sample plot of mean absolute error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/maey_eq.png)
![sample plot of mean root mean square error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rmsessm.png)
The RMSE skill score by month (lead time) is displayed for the selected agencies as well as the threshold value for each forecast month of the forecast period. The forecast root mean squared error is the square root of the average of the squared difference between the forecast and observed streamflow for each month. The threshold root mean squared error is calculated in the same manner, except using the threshold value which assumes forecasting the same streamflow each month of every year. The skill score is then calculated by 1 minus forecast RMSE divided by the threshold RMSE. The ideal skill score is 100%. By using the graph options, the forecast issuance months can be broken down and the RMSE skill score can be displayed by year within each month.
![sample plot of mean root mean square error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rmsessm_eq.png)
![sample plot of mean root mean square error skill score](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rmsessy.png)
The RMSE skill score by year is displayed for the selected agencies as well as the threshold value for each year for the forecast period. The forecast root mean squared error is the square root of the average of the squared difference between the forecast and observed streamflow for each year. The threshold root mean squared error is calculated in the same manner, except using the threshold value which assumes forecasting the same streamflow each month of every year. The skill score is then calculated by 1 minus forecast RMSE divided by the threshold RMSE. The ideal skill score is 100%. By using the graph options, the years can be broken down and the RMSE skill score can be displayed by forecast issuance month within each year.
![sample plot of mean root mean square error skill score](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rmsessy_eq.png)
![sample plot of mean root mean square error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rmsem.png)
The root mean squared error by month (lead time) is displayed for the selected agencies as well as the threshold value for each forecast month of the forecast period. The root mean squared error is the square root of the average of the squared difference between the forecast and observed streamflow for each month. The threshold value assumes forecasting the same streamflow each month of every year. The ideal RMSE is 0. By using the graph options, the forecast issuance months can be broken down and the RMSE can be displayed by year within each month.
![sample plot of mean root mean square error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rmsem_eq.png)
![sample plot of mean root mean square error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rmsey.png)
The root mean squared error by year is displayed for the selected agencies as well as the threshold value for each year for the forecast period. The root mean squared error is the square root of the average of the squared difference between the forecast and observed streamflow for each month. The threshold value assumes forecasting the same streamflow each month of every year. The ideal RMSE is 0. By using the graph options, the years can be broken down and the RMSE can be displayed by forecast issuance month within each year.
![sample plot of mean root mean square error](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rmsey_eq.png)
![sample plot of percent difference](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/percent.png)
The percent different plot is displayed for each selected agency as well as the threshold value for each forecast month of each selected year. The percent difference is the difference between the forecast and observed streamflow divided by the observed streamflow. The threshold value assumes forecasting the same flow each month of every year. The ideal percent difference is 0%.
![sample plot of percent difference](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/percent_eq.png)
![sample plot of Probability of Detection](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/poda.png)
The probability of detection above threshold is displayed for the selected agencies for each forecast issuance month of the forecast period. The POD is calculated by the number of incidents where the forecast and observed streamflow occurred above the threshold value during each month divided by the total number of years the observed fell above the threshold for each month. The ideal POD is 100%. The POD above threshold illustrates when a forecast is above the threshold, how often the observed actually fell above the threshold.
![sample plot of Probability of Detection](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/poda_eq.png)
![sample plot of Probability of Detection](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/podb.png)
The probability of detection below threshold is displayed for the selected agencies for each forecast issuance month of the forecast period. The POD is calculated by the number of incidents where the forecast and observed streamflows occurred below the threshold value during each month divided by the total number of years the observed fell below the threshold for each month. The ideal POD is 100%. The POD below threshold illustrates when a forecast is below the threshold, how often the observed actually fell below the threshold.
![sample plot of Probability of Detection](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/podb_eq.png)
![sample plot of False Alarm Rate](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/fara.png)
The false alarm rate above threshold is displayed for the selected agencies for each forecast issuance month of the forecast period. The FAR is calculated by the number of incidents where the observed streamflow was below the threshold value, but the forecast streamflow was above threshold during the year divided by the total number of years the forecast was above threshold. The ideal FAR is 0%. A large FAR above threshold indicates the forecasts are often over-forecast when the observed streamflow falls below the threshold.
![sample plot of False Alarm Rate](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/fara_eq.png)
![sample plot of False Alarm Rate](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/farb.png)
The false alarm rate below threshold is displayed for the selected agencies for each forecast issuance month of the forecast period. The FAR is calculated by the number of incidents where the observed streamflow is above the threshold value, but the forecast streamflow was below threshold during the year divided by the total number of years the forecast was below threshold. The ideal FAR is 0%. A large FAR below threshold indicates the forecasts are often under-forecast when observed streamflow is above the threshold.
![sample plot of False Alarm Rate](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/farb_eq.png)
![sample plot of Rank Histogram](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rank.png)
The rank histogram is displayed for the agency coordinated number only, for the forecast period. The rank histogram uses the 10%, 50%, and 90% forecast exceedance values to create four bins. The observed streamflow is then categorized into the appropriate bin. All months and years are combined. The ideal histogram represents the ideal number of observed streamflow years that should fall in each category. The higher exceedance probabilities indicate lower observed streamflows and an over-forecast while the lower exceedance probability indicates higher observed streamflows and an under-forecast. The threshold value is not used with this plot.
![sample plot of Rank Histogram](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rank_eq.png)
![sample plot of Rank Histogram](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/rank_lead.png)
The rank histogram by month is displayed for the agency coordinated number only, for the forecast period and displays the histograms by forecast issuance month. The rank histogram by month uses the 10%, 50%, and 90% forecast exceedance values to create four bins for each forecast issuance month. The observed streamflow is then categorized into the appropriate bins for each forecast month of the selected years. The ideal histogram represents the ideal number of observed streamflow years that should fall in each category. The higher exceedance probabilities indicate lower observed streamflows and an over-forecast while the lower exceedance probability indicates higher observed streamflows and an under-forecast. The threshold value is not used with this plot.
![sample plot of Climate Variability](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/climate.png)
The climate variability plot is displayed for each year there is a streamflow record for the forecast period. The plot is created by specifying the categories. The threshold value divides the above and below categories while a percentage (25% by default) above and below the threshold defines the much above and much below categories respectively. The observed streamflow is then categorized for each year (along the x-axis) and the following year represents the change and gets tallied into a category and represents the frequency. Under graph options, the percentage defining the categoies can be adjusted.
![sample plot of Contingency Table](https://webarchive.library.unt.edu/web/20130214095653im_/http://wateroutlook.nwrfc.noaa.gov/img/about/table.png)
The contingency table displays the forecast versus the observed streamflow for each year for the forecast period. The table displays one forecast issuance month (default is January) and can be changed in the time scale options. The table is created by specifying the categories. The threshold value divides the above and below categories while a percentage (25% by default) above and below the threshold defines the much above and much below categories respectively. The years are then placed in the corresponding sections of the table based on which category the observed and forecast streamflows fall. Under graph options, the percentage defining the categoies can be adjusted. Forecasts falling along the diagonal is ideal, from top left to bottom right.