Observational Data Processing at NCEP
Dennis Keyser -
NOAA/NWS/NCEP/EMC
(Last Revised 7/23/2011)
Please take a moment to read the Disclaimer for this non-operational web page. Most of the observational data at NCEP are stored in WMO BUFR format. This format is an international standard and provides an efficient means for transferring data. In addition it allows for great flexibility for adding new observation elements. There are a series of tables associated with BUFR. “ Table A” defines the data category associated with a particular BUFR message containing report data. “ Table B” classifies and defines data elements, or descriptors, according to scale, reference value, number of bits and units. “ Table C” defines data description operators. “ Table D” defines the list of common sequences. In addition, there are BUFR code and flag tables as well as code tables common to BUFR and other binary and alphanumeric codes. The need for external tables can make the process for BUFR data encoding and decoding quite cumbersome for a typical user. As such, a special application has been designed at NCEP which provides user-friendly access to the BUFR files through a series of FORTRAN and C subroutines in a machine independent BUFR library (called BUFRLIB). These routines allow one to encode or decode data into BUFR using mnemonics to represent the data. The mnemonics are associated with BUFR descriptors in a special version of the Tables A, B, C and D. When a BUFR file is created, the mnemonic table is read in from an external location and is itself encoded into BUFR messages at the top of the output file. These messages have Table A data category (message type) 11 (BUFR tables). This allows each BUFR file to be “self defined”. No external tables are needed to decode data out of the file. NCEP Central Operations has written a BUFRLIB software user guide which provides a detailed explanation of the NCEP BUFRLIB subroutines along with other useful information on BUFR as it is used at NCEP. Click here to view a sample Fortran code which uses BUFRLIB routines to encode reports into a BUFR file. Next is a brief outline
on the current method for processing
observations
that arrive at NCEP. Its main function is to provide links to web pages
which discuss each item in detail.
Click
here
to see a flow
chart depicting the data procesing system
at NCEP. 1. NCEP receives the majority of its data from the Global Telecommunications System (GTS) and the National Environmental Satellite, Data, and Information Service (NESDIS). a. The GTS and aviation circuit bulletins are transferred from the NWS Telecommunication Operations Center (TOC/NWSTG) to NCEP's Central Operations (NCO) and networked to one of two interactive nodes on the NCEP IBM Central Computer System (IBM-CCS) (one production, one development) using LDM (Local Data Manager) and DBNet (Distributive Brokered Networking) software packages. These data are then decoded from their native format and encoded into WMO BUFR format using decoder software designed by NCO's System's Integration Branch. b. Most of the satellite data are processed in batch mode as they become available from the various NESDIS servers. Regularly scheduled jobs on the IBM-CCS transfer "new" files from these servers and encode the data into WMO BUFR format. See http://www.emc.ncep.noaa.gov/data_processing/satellite_ingest.doc/document.htm for more information. c. NCEP receives "Level 3" radial wind data from 158 NEXRAD radar stations via the radar multicast (NIDS format). At eight minutes past each hour, a script runs to gather up the most recent radar data into eight pairs of files containing the raw data and a station list. At thirty-five minutes past each hour these raw data are superobed and encoded into BUFR format. d. NCEP receives on-site superobed Level 2 (which we call "Level 2.5") radial wind data from 158 NEXRAD radar stations via the NOAA/Radar Operations Center (ROC) Open Systems Radar Product Generator (ORPG) stream from the NWS Telecommunication Operations Center (TOC/NWSTG) . Raw data files arriving at NCEP are placed in a holding area. Twice each hour at five and thirty-five minutes past the hour, all newly arrived files are first uncompressed and then encoded into WMO BUFR format. e. NCEP receives the full complement of Level 2 radial wind and reflectivity data from 158 NEXRAD radar stations via the NOAA/Radar Operations Center (ROC) Radar Data Acquisition stream from the NWS Telecommunication Operations Center (TOC/NWSTG) . Raw data files arriving at NCEP are placed in a holding area. Four times each hour at ten, twenty-five, forty and fifty-five minutes past the hour, all newly arrived files are first uncompressed, then quality controlled (both radial wind and reflectivity) and then encoded into WMO BUFR format. f. Mesonet surface data are gathered by NOAA/ESRL/GSD and then pushed to an NCO server several times each hour. NCO converts the data from netCDF to WMO BUFR format. All of the encoded BUFR
data are then appended to the
appropriate
files
in the data base. The files are organized by the WMO BUFR
type
and
local subtype and contain information in 24 hour blocks (based on
report
time). Observational files remain on-line for up to 10 days
before
migration to off line cartridges. This allows late arriving
observations
to be accumulated. While on line, there is open access to
them
not
only for operations, but also for research and study. 2. The various NCEP
networks access the observational data
base at a
set time each day (i.e., the data cutoff time) and perform a
time-windowed
dump of requested observations. Observations of a similar
type
[e.g.,
satellite-derived winds ("satwnd"), surface land reports ("adpsfc")]
are
dumped into individual BUFR files which maintain the original structure
of reports, although some interactive quality control is
applied,
duplicate reports are removed, and upper-air report "parts" are
merged.
See
http://www.emc.ncep.noaa.gov/mmb/data_processing/data_dumping.doc/document.htm
for more information. 3. The final step in
preparing most of the "conventional" observational data
for
assimilation
by the analysis involves the execution of a series of programs which
read
in the observations from the various dump files, add forecast ("first
guess")
background and observation error information, perform automated quality
control, and finally output the observations in a monolithic BUFR file
known as "PREPBUFR". See
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/document.htm
for more information. 4. The PREPBUFR file is
read by the
Global
Statistical Interpolation
(GSI) analysis which
runs in the
Global
Forecast
System (GFS)
and the Global Data Assimilation System (GDAS)
(running the Global Spectral Model),
the regional
North
American Model (NAM)
and North
American
Data Assimilation System (NDAS) [running the NOAA Environmental
Modeling System (NEMS) version of the Non-Hydrostatic Multi-scale Model
in B-grid (NMMB)], the
Rapid
Refresh
(RAP)
[running
the RAP-configuration of the WRF model with Advanced Rearch
WRF
(ARW) core],
and
the.
Real-Time
Mesoscale Analysis (RTMA). Here are some other links to my web sites devoted to data processing: PREPBUFR Report Types used by Global GFS and GDAS GSI analyses PREPBUFR Report Types used by Global CDAS/reanalysis analyses PREPBUFR Report Types used by NAM and NDAS GSI analyses PREPBUFR Report Types used by Rapid Refresh GSI analysis PREPBUFR Report Types used by Real-Time Mesoscale Analysis Summary of Virtual Temperature Processing in PREPBUFR RAOB/PIBAL Balloon Drift Processing in PREPBUFR Redesign of the NOAA/NCEP Wind Profiler Quality Control Module Sample program to decode reports from PREPBUFR file Summary of Changes to the PREPDATA Program User Guide to Interpreting Data Dump Counts in Data Dump Status Files BUFR Table B Descriptors and Mnemonics in NCEP Observational Database BUFR Table D Descriptors and Mnemonics in NCEP Observational Database Satellite Historical Documentation Non-Satellite Data Historical Documentation Format of Tropical Cyclone Vital Statistics Records ("tcvitals") NCEP BUFR File Structure [with emphasis on embedded BUFR table (A, B, D) messages] |
NOAA
/ National
Weather Service
National Centers for Environmental Prediction Environmental Modeling Center 5200 Auth Road Camp Springs, Maryland 20746-4304 Page Author: EMC Webmaster |
---|