|
biastext
SUBJECTIVE
LIST OF MODEL PERFORMANCE CHARACTERISTICS
1.
Introduction
The NCEP model suite is upgraded numerous
times annually. Since each "model" is actually a system of integrated
components, even slight “tweaks” to any of the components can drastically
effect the model's performance characteristics. For example, the
Global Forecast System (GFS) consists of the initialization scheme (Global
Data Assimilation Scheme - GDAS), the Global Model itself, and the post
processed grids that are made available for use in grid and grib format.
Even a slight modification to any one of these components can drastically
effect the perceived performance of the model. As a result it
is not only difficult to isolate consistent model performance characteristics
(loosely referred to as “bias”) across the model upgrades, but also the
source of the bias.
Examples of modifications to a model
system that can effect model performance characteristics include:
-
Modification of initial data ingested by model
(volume, type, density, accuracy)
-
Modification of model structure (horizontal/vertical
resolution, time step increments, domain, grid point vs spectral wave model,
vertical coordinate, hydrostatic vs non hydrostatic models)
-
Physics packages used by the model (radiation
schemes, diffusion, land/surface representations)
-
Parameterization schemes used by the model
(convective parameterization, frictional components)
-
Post Processing of model data (precip type
algorithms, resolution of the grid the model data is displayed upon)
2.
Model Performance Characteristics
Model performance is a function of model
error - which can be split into two components (systematic error
and random error).
The random component of error is that
which we are unable to easily attribute a cause (and therefore not
easily correct).
Systematic error can be automatically
removed from model output to correct or minimize the amount of error in
the model solution.
A subjective list of model performance
characteristics (biases) have been obtained via forecasters and is available
in the next section. This information is useful for both forecasters
and by the modelers.
The majority of the biases listed below
are from HPC. However, in order for EMC to gain a more complete picture
of model performance, all users of NCEP model output should provide their
subjective observations of model performance by submitting
their subjective observations of bias.
This information will be conveyed to HPC
and EMC and potentially added to this web page.
As a reminder, results of OBJECTIVE
methods to quantify the systematic error in model output are found at these
sites
EMC
home site http://www.emc.ncep.noaa.gov/modelperf/
HPC
interactive model bias site http://www.hpc.ncep.noaa.gov/mdlbias/index.html
3.
Table of Subjective Model Performance Characteristics
Click on a model name to see the latest
subjective bias list.
4.
Submit a subjective bias
Click these words
to submit a bias.
5.
Other model information
Want to know the latest upcoming plans
for model development and system resources as it relates to modeling at
NCEP ???
Click this link http://www.hpc.ncep.noaa.gov/html/model2.shtml#synergy
Want a quick summary of model attributes
? Click this link http://meted.ucar.edu/nwp/pcu2/index.htm
OPERATIONAL
MODELS
AFWA
MM5
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
|
Over predicts QPF at times in regions
of moderate synoptic scale forcing
|
Southeast and east coast
|
Anytime
|
NCEP HPC
|
2000
|
.
|
Initial Analysis Scheme ?
|
CMC
GEM
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
|
.
|
.
|
.
|
.
|
.
|
.
|
CMC
Global
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
|
.
|
.
|
.
|
.
|
.
|
.
|
ECMWF
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
|
.
|
.
|
.
|
.
|
.
|
.
|
NAM
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
|
GFS
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
GFS too ambitious with strength and speed of systems crossing Sierras after fhr 36
|
SW US
|
Cool season
|
USGS
|
Dec 2005
|
Too progressive and strong with systems crossing Sierras
|
Model resolution of topography
|
Convective Feedback
|
Primarily east of front range and
west of Appalachians
|
Warm season, any time of model day
|
NCEP HPC
|
Spring of 1998
|
When specific thresholds in the mass
fields are met, convective scheme is triggered and then dumps a large amount
of QPF over a grid point - releasing so much latent heat over the grid
point that the model is forced to adjust the mass fields by producing a
local vertical motion max in the mid troposphere (~ 500mb), a corresponding
upper level jet max over the vertical motion max - an intense and small
scale vort max in the mid levels (MCV).
The model scales up the mesoscale circulation
at mid levels and holds onto it as a real feature for as long as 3 days.
The model can produce precipitation in
association with the feature as it tracks along in the flow.
|
GFS Convective Parameterization Scheme
|
Dry bias north of areas where over
2" of QPF has been produced in a 6hr period
|
Primarily east of front range and
west of Appalachians
|
Warm season, any time of model day
|
NCEP HPC
|
Spring of 1998
|
QPF produced from convective feedback
blocks northward advection of moisture
|
Result of GFS Convective Parameterization
Scheme
|
QPF verification historically better
than Eta
|
CONUS
|
Cool Season only
|
NCEP HPC
|
1999
|
Rely more heavily on QPF from GFS
- especially beyond 36 hours
|
GDAS better than EDAS ?
|
Aerial coverage of QPF and mass fields
over done (QPF at low thresholds .01" and .10")
|
CONUS
|
Anytime
|
NCEP HPC
|
Since mid 1990's
|
Over forecast of aerial coverage of
precip can lead to high bias in PoPs
|
Model resolution (the lower the resolution
the more geography a QPF pattern can get spread over)
|
Slightly ambitious with magnitude
of high amplitude patterns
|
North America
|
Cool season so far
|
NCEP HPC
|
Since fall 2002
|
Prediction of southward progression
of cold air over done
Model a bit too extreme in temp patterns
beyond 84 hours
Precip Type Algorithm off of GFS
too eager to depict snow
|
?
|
Ambitious to phase northern and southern
stream systems in fast and spit flow patterns beyond fhr 84
|
North America
|
Cool season
|
NCEP HPC
|
Cool season 2001 (not noticed yet
in 2002)
|
Over forecast of cyclogenesis east
of Rockies
|
Suspect related to model resolution
and lack of dense obs data where associated systems originate in forecast
cycle
|
Major difference in QPF forecast than
ETA
|
CONUS
|
Warm season
|
NCEP HPC
|
Since mid 1990's
|
Lack of run to run continuity in QPF
|
Different convective parameterizations
between models result in different QPF forecasts (primarily in areas where
synoptic scale forcing is weak)
|
NOGAPS
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
Prior to recent upgrade was typically the
model that exhibited lowest AC scores.
Seemed very angular in its depiction of
500mb heights (extreme gradients beyond 48 hours).
|
.
|
.
|
.
|
.
|
.
|
.
|
RUC
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
|
.
|
.
|
.
|
.
|
.
|
.
|
UKMET
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Seems better than GFS with forecast
of phasing of systems in northern and southern branch of jet
|
North American middle latitudes
|
Anytime
|
NCEP HPC
|
Since fall of 2001
|
When GFS is showing phasing of systems
beyond 84 hours, check UKMET to see if solution is consistent
|
GDAS ?
|
ENSEMBLE
PREDICTION SYSTEMS
CMC
EPS
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
|
.
|
.
|
.
|
.
|
.
|
.
|
ECMWF
EPS
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
|
.
|
.
|
.
|
.
|
.
|
.
|
NCEP
GFS EPS
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Ensemble mean maxima/minima much more
subdued compared to actual verification
|
Anywhere
|
Anytime
|
NCEP HPC
|
Fall 2000
|
Position forecast decent, but actual
strength of a system watered down
|
This is normal for ensemble output
with significant spread as it takes the average of numerous solutions thereby
flattening the amplitude of "ridges or valleys in mass fields
|
The ensemble mean does well, in situations
where operational GFS is a bit too aggressive, the GFS ENS MEAN seems to
verify better at 500mb beyond 96 hours.
|
North America
|
Anytime
|
NCEP HPC
|
Fall 2001
|
More run to run continuity in EPS output, than operational runs.
|
Because EPS output is average of numerous solutions, it is less
sensitive to extreme solutions.
|
NCEP
SREF EPS
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Ensemble mean maxima/minima much more
subdued compared to actual verification
|
Anywhere
|
Anytime
|
NCEP HPC
|
Fall 2000
|
Position forecast decent, but actual
strength of a system watered down
|
This is typical for ensemble output
with significant spread as it takes the average of numerous solutions thereby
flattening the amplitude of "ridges or valleys in mass fields
|
NOGAPS
EPS
Subjectively Observed
Bias
|
Geographical location
of bias
|
Annual/Diurnal
attribute
|
Submitted by
|
Date Submitted
|
Operational Implication
|
Suspected Cause
|
Non submitted so far
|
.
|
.
|
.
|
.
|
.
|
.
|
Submit
a bias
Fill out the form and click SUBMIT and it will be reviewed by HPC personnel.
We will then need to contact you for more specifics, so please leave
an email address.
|