Several of the mesonets available from MADIS are national in scope, and contain thousands of stations. To cut down on the data volume, you can filter out the stations that are outside your geographic area of interest, by specifying the corners of a latitude/longitude rectangle. Only stations inside the rectangle will be sent into LDAD. If the mesonet is MesoWest (also called UTMESNET), there's an additional option to include only those subproviders that are specified by the user. The filtering scheme will also work with any mesonet obtained from a source other than MADIS, as long as the mesonet meets this requirement: the first column of the data files must contain the same site ID's as found in the first column of the station table. Note that some of the mesonets from MADIS do not meet this criteria, and cannot be filtered with this scheme, but these are all regional and shouldn't need filtering. Here is the list of which mesonets can and cannot be filtered:
Mesonet | Can be Filtered | Has Header | |
---|---|---|---|
AFA | Yes | No | |
AIRNow | Yes | No | |
AKDOT | No | No | |
AK-Meso | Yes | No | |
APG | No | No | |
APRSWXNET | Yes | No | |
ARLFRD | Yes | No | |
AWS | Yes | No | |
AWX | Yes | No | |
CA-Hydro | Yes | No | |
CAIC | Yes | No | |
CODOT | Yes | No | |
CO_E-470 | Yes | No | |
DCNet | Yes | Yes | |
DDMET | Yes | No | |
DEOS | Yes | No | |
FLDOT | Yes | No | |
FL-Meso | Yes | No | |
GADOT | Yes | No | |
GLDNWS | Yes | No | |
GLOBE | Yes | No | |
GoMOOS | Yes | No | |
GPSMET | Yes | Yes | |
HADS | Yes | No | |
HF-METAR | Yes | No | |
IADOT | No | Yes | |
IEM | Yes | No | |
INDOT | Yes | No | |
INTERNET | Yes | No | |
ITD | Yes | No | |
KSDOT | Yes | Yes | |
KYTC-RWIS | Yes | No | |
LCRA | Yes | No | |
LSU-JSU | No | Yes | |
MAP | Yes | No | |
MDDOT | No | No | |
MEDOT | No | No | |
MesoWest | Yes | No | |
MISC | Yes | No | |
MNDOT | No | No | |
MOComAgNet | Yes | No | |
MQT-Meso | Yes | No | |
NC-ECONet | Yes | No | |
NDDOT | No | No | |
NEDOR | Yes | Yes | |
NERRS | Yes | No | |
NHDOT | Yes | No | |
NJWxNet | Yes | No | |
NonFedAWOS | Yes | No | |
NOS-NWLON | Yes | No | |
NOS-PORTS | Yes | No | |
NWS-COOP | Yes | No | |
OHDOT | Yes | No | |
OK-Meso | No | Yes | |
PADEP | Yes | No | |
RAWS | Yes | No | |
RDMTR | Yes | No | |
SFWMD | Yes | No | |
UDFCD | Yes | Yes | |
UrbaNet | Yes | No | |
USouthAL | Yes | No | |
VADOT | Yes | No | |
VTDOT | Yes | No | |
WIDOT | Yes | No | |
WT-Meso | Yes | No | |
WxFlow | Yes | No | |
WXforYou | Yes | No | |
WYDOT | No | No |
The way this works is that you choose which mesonets you want to filter, create the list of stations to be included by running a script, then process incoming data files with a second script that will filter the unwanted stations and send files containing only the included stations into LDAD.
Here are the details:
The include list that's created is called "included_stations.txt". The script will also save the previous version of that file as "included_stations.txt.old", so you can go back if something goes wrong.
* Whenever you update your LDAD station tables on ds1, you will also need to update them here on ls1, rerun build_station_idx.pl, and then copy over the filtered station tables back to /data/fxa/LDAD/data.
The /ldad/madisfilter/new subdirectory will contain versions of the *Station.txt files that only contain stations that are in the include list. These should be copied to /data/fxa/LDAD/data. This is important because if you go over 15,000 stations progressive disclosure won't work.
As none of the info in the header line of the raw data files is actually needed, this will cause no harm.
The input file (MESOWHATEVER.dat) can be in any directory you like *except* /data/Incoming. You can keep these around if you want to see what was in there before the filtering, or you can just toss them once you're satisfied that the filter is working as expected.
The output from the script will be: /data/Incoming/MESOWHATEVER.dat, which will then be processed in the normal manner by LDAD.
FSL5 ^LDAD\.aws\.(.*) FILE -overwrite -close /data/Incoming/\1 FSL5 ^LDAD\.raw\.(.*) FILE -overwrite -close /data/Incoming/\1 FSL4 ^GPSIPW_CSV_(gpsmet\..*) FILE -overwrite -close /data/Incoming/\1 - Change that to: FSL5 ^LDAD\.aws\.(.*) PIPE -close /ldad/madisfilter/MADISfilter.pl \1 FSL5 ^LDAD\.raw\.(.*) PIPE -close /ldad/madisfilter/MADISfilter.pl \1 FSL4 ^GPSIPW_CSV_(gpsmet\..*) PIPE -close /ldad/madisfilter/MADISfilter.pl \1
* The MADIS team would like to thank Clark Safford of FFC for contributing the original software on which this filtering scheme is based. (Please contact us, however, and not Clark, with any questions.)