Dennis Keyser - NOAA/NWS/NCEP/EMC
(Last Revised 6/23/2008)
Please take a moment to read the Disclaimer
for this non-operational web page.
Most of the observational data at NCEP are stored in WMO BUFR format. This format is an international standard and provides an efficient means for transferring data. In addition it allows for great flexibility for adding new observation elements.
There are a series of tables associated with BUFR. “Table A” defines the data category associated with a particular BUFR message containing report data. “Table B” classifies and defines data elements, or descriptors, according to scale, reference value, number of bits and units. “Table C” defines data description operators. “Table D” defines the list of common sequences. In addition, there are BUFR code and flag tables as well as code tables common to BUFR and other binary and alphanumeric codes. The need for external tables can make the process for BUFR data encoding and decoding quite cumbersome for a typical user.
As such, a special application has been designed at NCEP which provides user-friendly access to the BUFR files through a series of FORTRAN and C subroutines in a machine independent BUFR library (called BUFRLIB). These routines allow one to encode or decode data into BUFR using mnemonics to represent the data. The mnemonics are associated with BUFR descriptors in a special version of the Tables A, B, C and D. When a BUFR file is created, the mnemonic table is read in from an external location and is itself encoded into BUFR messages at the top of the output file. These messages have Table A data category (message type) 11 (BUFR tables). This allows each BUFR file to be “self defined”. No external tables are needed to decode data out of the file.
NCEP Central Operations has written a BUFRLIB software user guide which provides a detailed explanation of the NCEP BUFRLIB subroutines along with other useful information on BUFR as it is used at NCEP.
Next is a brief outline on the current method for processing
observations
that arrive at NCEP. Its main function is to provide links to web pages
which discuss each item in detail.
1. NCEP receives the majority of its data from the Global Telecommunications System (GTS) and the National Environmental Satellite, Data, and Information Service (NESDIS).
a. The GTS and aviation circuit bulletins are transferred from the NWS Telecommunication Operations Center (TOC/NWSTG) to NCEP's Central Operations (NCO) and networked to one of two interactive nodes on the NCEP IBM Central Computer System (IBM-CCS) (one production, one development) using LDM (Local Data Manager) and DBNet (Distributive Brokered Networking) software packages. These data are then decoded from their native format and encoded into WMO BUFR format using decoder software designed by NCO's System's Integration Branch.
b. Most of the satellite data are processed in batch mode as they become available from the various NESDIS servers. Regularly scheduled jobs on the IBM-CCS transfer "new" files from these servers and encode the data into WMO BUFR format. See http://www.emc.ncep.noaa.gov/data_processing/satellite_ingest.doc/document.htm for more information.
c. NCEP receives "Level 3" radial wind data from 158 NEXRAD radar stations via the radar multicast (NIDS format). At eight minutes past each hour, a script runs to gather up the most recent radar data into eight pairs of files containing the raw data and a station list. At thirty-five minutes past each hour these raw data are superobed and encoded into BUFR format.
d. NCEP receives on-site superobed Level 2 (which we call "Level 2.5") radial wind data from 158 NEXRAD radar stations via the NOAA/Radar Operations Center (ROC) Open Systems Radar Product Generator (ORPG) stream from the NWS Telecommunication Operations Center (TOC/NWSTG) . Raw data files arriving at NCEP are placed in a holding area. Twice each hour at five and thirty-five minutes past the hour, all newly arrived files are first uncompressed and then encoded into WMO BUFR format.
e. NCEP receives the full complement of Level 2 radial wind and reflectivity data from 158 NEXRAD radar stations via the NOAA/Radar Operations Center (ROC) Radar Data Acquisition stream from the NWS Telecommunication Operations Center (TOC/NWSTG) . Raw data files arriving at NCEP are placed in a holding area. Four times each hour at ten, twenty-five, forty and fifty-five minutes past the hour, all newly arrived files are first uncompressed, then quality controlled (both radial wind and reflectivity) and then encoded into WMO BUFR format.
f. Mesonet surface data are gathered by NOAA/ERSL/GSD and then pushed to an NCO server serval times each hour. NCO converts the data from netCDF to WMO BUFR format.
All of the encoded BUFR data are then appended to the
appropriate
files
in the data base. The files are organized by the WMO BUFR
type
and
local subtype and contain information in 24 hour blocks (based on
report
time). Observational files remain on-line for up to 10 days
before
migration to offline cartridges. This allows late arriving
observations
to be accumulated. While online, there is open access to them
not
only for operations, but also for research and study.
2. The various NCEP networks access the observational data
base at a
set time each day (i.e., the data cutoff time) and perform a
time-windowed
dump of requested observations. Observations of a similar
type
[e.g.,
satellite-derived winds ("satwnd"), surface land reports ("adpsfc")]
are
dumped into individual BUFR files which maintain the original structure
of reports, although some interactive quality control is
applied,
duplicate reports are removed, and upper-air report "parts" are
merged.
See http://www.emc.ncep.noaa.gov/mmb/data_processing/data_dumping.doc/document.htm
for more information.
3. The final step in preparing most of the observational data
for
assimilation
by the analysis invloves the execution of a series of programs which
read
in the observations from the various dump files, add forecast ("first
guess")
background and observation error information, perform automated quality
control, and finally output the observations in a monolithic BUFR file
known as "PREPBUFR". See http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/document.htm
for more information.
4. The PREPBUFR file is read by the Global
Statistical Interpolation (GSI) analysis which runs in the Global
Forecast
System (GFS) and Global Data Assimilation System (GDAS), the
GSI analysis
which runs in the regional North American Model (NAM) and North
American
Data Assimilation System (NDAS) which currently use the WRF-NMM model,
and
the 3DVAR which runs
in the Rapid
Update
Cycle (RUC) system.
Here are some other links to my web sites devoted to data processing:
PREPBUFR Report Types used by Global GFS and GDAS GSI analyses
PREPBUFR Report Types used by Global CDAS/reanalysis analyses
PREPBUFR Report Types used by NAM and NDAS GSI analyses (WRF-NMM model)
PREPBUFR Report Types used by upper-air Rapid Update Cycle 3DVAR analysis
Summary of Virtual Temperature Processing in PREPBUFR
RAOB/PIBAL Balloon Drift Processing in PREPBUFR
Redesign of the NOAA/NCEP Wind Profiler Quality Control Module
Sample program to decode reports from PREPBUFR file
Summary of Changes to the PREPDATA Program
User Guide to Interpreting Data Dump Counts in Data Dump Status Files
BUFR
Table B Descriptors and
Mnemonics
in NCEP Observational Database