QCRad1Long: Data Quality Assessment for ARM Radiation Data
The QCRad VAP has been developed to assess the data quality and to enhance data continuity for the ARM radiation data collected at the SGP extended facilities and other ARM ARCS (Atmospheric Radiation and Cloud Station) facilities.
Information Last Updated: April 2007
General Description
The QCRad methodology in essence uses climatological analyses of the surface radiation measurements to define reasonable limits for testing the data for unusual data values. The main assumption here is that the majority of the climatological data are "good" data, which for field sites operated with care such as those of the Atmospheric Radiation Measurement (ARM) Program is a reasonable assumption. Then data which fall outside the normal range of occurrences are labeled either "questionable" or "bad", depending on how far outside the normal range the particular data reside. The methodology not only sets fairly standard maximum and minimum value limits, but includes many cross-comparisons based on what we have learned about how these instruments behave in the field in developing other Value Added Products (VAPs) such as the Diffuse IR Loss Correction VAP (Younkin and Long, 2004) and the Best Estimate Flux VAP (Shi and Long, 2002).
The Algorithm and Methodology
QC Testing Criteria: For the QCRad methodology, three levels of testing are in many cases applied. These we define in order of their severity as far as data falling outside the limits. A data Quality Control (QC) flag value of "0" indicates that the data have not failed any of the tests applied, and as far as we can tell represent "good" data. Then the greater the QC flag value, the more the data fall outside the normal range of values typically seen for the quantity represented. The QCRad code applies testing using the largest limits first, and then if the data passes that testing, applies the next smaller limits testing. Thus the flag value represents at what level of severity the tested data failed. The larger the QC flag integer value, the more severe the testing failure. In general, an odd QC flag value indicates a measurement that falls below the corresponding minimum limit, while an even value indicates a measurement that falls above the corresponding maximum test limit. The one exception is for a specific test not being possible, usually because some value needed to perform the test is not available. In this case, a QC flag value of "-1" is assigned, and simply means that the data could not be tested in that way. For example, if the ambient air temperature measurements are missing or deemed "bad," then the tests based on using the air temperature cannot be performed, and the corresponding flags are set to "-1." A flag of "-1" does NOT mean the data are "bad," merely that the particular test could not be applied, thus we do not know if it would have passed or failed had it been tested that way.
For the QCRad VAP, we set the smallest testing limits to reflect the range within which the majority of data typically fall. We then set the next largest limits to include data that have been recorded at the site and have been shown to be possible, yet that occur rarely and so might also be "bad" data. In this latter case, the QC flags are set to a value of either 1 or 2 as appropriate, but the data value itself is left in the output files. The intent is to flag these values as "possible, but rarely occurring," and it is left to the user to determine whether they wish to use these data or not. Any data that fall outside of the second level of testing are deemed to be "bad," the QC flag value is set to either 3 or 4, and the corresponding data value is set to a "bad" value of "-9999.0," i.e., the data value is not given to the user.
For the shortwave (SW) variables, limits are typically set using the cosine of the solar zenith angle as the independent variable. For the longwave (LW), some tests apply simple maximum and minimum limits on acceptable values. Other tests use the high degree of correlation known to exist between the surface 2-meter ambient air temperature and typical ranges of measured LW for the particular climate.
Flow Chart
Not available.
Input Data Sources
The input data sources for QCRad VAP are files containing measurements of downwelling SW direct and diffuse irradiances, downwelling LW irradiance, and upwelling SW and LW irradiance measurements. The input data also include surface meteorological fields such as relative humidity, air temperature, pressure, etc.
The input files involved in this VAP are as follows:
XX stands for the facility name (C1, C2, E1, E2, etc); YY stands for the data level (a0, a1, a2, or b1).
SGP Site
sgpsirs1duttXX.c1
or sgpsiros1duttXX.c1
or sgpsirsXX.YY
or sgpbrs1duttXX.c1
or sgpbrsXX.YY
or sgpbeflux1longC1.c1
sgp1smosXX.YY
or sgp5ebbrXX.YY
sgpmfrsrXX.YY
TWP site
twpskyrad1duttXX.c1
or twpskyrad60sXX.b1
twpgndrad1duttXX.c1
or twpgndrad60sXX.b1
twpsmet60sXX.b1
twpmfrsrXX.a1
NSA Site
nsaskyrad1duttXX.c1
or nsaskyrad60sXX.b1
nsagndrad1duttXX.c1
or nsagndrad60sXX.b1
nsamettwr2hC2.b1
or nsamettwr4hC1.b1
nsamfrsrXX.a1
All the above listed files can be ordered online from the ARM Data Archive. Refer to ARM Technical Report TR-074 for a complete list of input files and fields for this VAP.
Output Products
The QCRad VAP produces two daily netcdf files containing radiation measurement fields and their QC values. The output data are in 1-minute intervals. The first output file contains all the detailed bit-packed QC information. The second file summarizes and simplifies the QC information as follows:
QC flag = -1; Data missing
QC flag = 0; Data good
QC flag = 1; Data indeterminate
QC flag = 2; Data bad
The VAP also produces daily quicklooks and monthly and yearly summary files useful for statistical analysis of data distribution.
The daily QCRad VAP output files follow the ARM naming convention:[site]qcrad1long[facility].LL.YYYYMMDD.hhmmss.cdf
where
[site] is the ARM site name (SGP, NSA, TWP); [facility] is the facility name (C1, C2, E1, E2, etc); LL is the level of the data stream, LL = c1 for the netcdf file with the bit-packed QC information and LL = s1 for the summary file; YYYY is the year; MM is the month of the year; DD is the day of the month; hh is the hour of the day of data start; mm is the minutes of the hour; and ss is the seconds of the minute. The only exception to this naming convention is the SGP BRS datastream, which is named sgpqcradbrs1longC1.LL.YYYYMMDD.hhmmss.cdf. The output fields and their units and descriptions are listed in Appendix B in ARM Technical Report TR-074.
Data Quality Assessment Included
See QC Testing Criteria under The Algorithm and Methodology.
Known Algorithm Caveats
N/A
VAP History
Current version of the code is 1.0.
N/A
Compatibility of Results from Different Version
N/A
Plans for Future Modifications
No modifications are currently being planned.
N/A
Output Products
Descriptions of Products
Two netcdf files are produced daily for the QCRad VAP. The first file is a c1 level file which contains radiation measurement fields and the bit-backed QC information associated with each field. The summary netcdf file (.s1) contains the same radiation measurement fields with simplified QC information as mentioned in the previous section. The .s1 files are the recommended ARM surface radiation files for the general user. The fields included in the c1 level output files are:
- Date/Time – given in UTC, a time zone based at Greenwich, England.
- Irradiances – each daily file contains following irradiance fields: Total downwelling SW, diffuse and direct normal SW, upwelling SW, downwelling and upwelling LW, and best estimate of total downwelling SW
- Instrument environment – case and dome temperatures
- Surface meteorological measurements – relative humidity, air temperature, air pressure, wind speed, wind direction, and precipitation.
- qc_fieldname – bit-packed fields with complete data quality check of the corresponding fields
- aqc_fieldname – ancillary qc fields for scientists who are interested in the details of how the qc fields are set.
Below is an example of the downwelling diffuse SW field and its associated qc and aqc fields. Refer to Appendix B of the QCRad technical report (ARM Technical Report TR-074) for more detailed qc definition for each field.
float down_short_diffuse_hemisp(time) ;
down_short_diffuse_hemisp:long_name = "Downwelling Shortwave Diffuse Hemispheric Irradiance" ;
down_short_diffuse_hemisp:units = "W/m2" ;
down_short_diffuse_hemisp:missing_value = -9999.f ;
int qc_down_short_diffuse_hemisp(time) ;
qc_down_short_diffuse_hemisp:long_name = "Data Quality Check for down_short_diffuse_hemisp" ;
qc_down_short_diffuse_hemisp:units = "Unitless" ;
qc_down_short_diffuse_hemisp:description = "This field contains bit packed values which should be interpreted as listed; no bits set (zero) represents good data" ;
qc_down_short_diffuse_hemisp:bit_1_description = "Valid data value not available in input file, data value in output file set to -9999" ;
qc_down_short_diffuse_hemisp:bit_1_assessment = "Bad" ;
qc_down_short_diffuse_hemisp:bit_2_description = "data too low (UC1)" ;
qc_down_short_diffuse_hemisp:bit_2_assessment = "Indeterminate" ;
qc_down_short_diffuse_hemisp:bit_3_description = "data too high (UC1)" ;
qc_down_short_diffuse_hemisp:bit_3_assessment = "Indeterminate" ;
qc_down_short_diffuse_hemisp:bit_4_description = "data too low (UC2)" ;
qc_down_short_diffuse_hemisp:bit_4_assessment = "Bad" ;
qc_down_short_diffuse_hemisp:bit_5_description = "data too high (UC2)" ;
qc_down_short_diffuse_hemisp:bit_5_assessment = "Bad" ;
qc_down_short_diffuse_hemisp:bit_6_description = "data too low (PP)" ;
qc_down_short_diffuse_hemisp:bit_6_assessment = "Bad" ;
qc_down_short_diffuse_hemisp:bit_7_description = "data too high (PP)" ;
qc_down_short_diffuse_hemisp:bit_7_assessment = "Bad" ;
qc_down_short_diffuse_hemisp:bit_8_description = "GSW2SumSW (SZA <= 75 and SumSW > 50 W/m2; GSW/SumSW < 0.92 or GSW/SumSW > 1.08)" ;
qc_down_short_diffuse_hemisp:bit_8_assessment = "Indeterminate" ;
qc_down_short_diffuse_hemisp:bit_9_description = "GSW2SumSW (93 > SZA > 75 and SumSW > 50; GSW/SumSW < 0.85 or GSW/SumSW > 1.15)" ;
qc_down_short_diffuse_hemisp:bit_9_assessment = "Indeterminate" ;
qc_down_short_diffuse_hemisp:bit_10_description = "DifSW2GSW (SZA < 75; DifSW/GSW 1.05, GSW > 50 W/m2)" ;
qc_down_short_diffuse_hemisp:bit_10_assessment = "Indeterminate" ;
qc_down_short_diffuse_hemisp:bit_11_description = "DifSW2GSW (93 > SZA > 75; DifSW/GSW > 1.10, GSW > 50 W/m2)" ;
qc_down_short_diffuse_hemisp:bit_11_assessment = "Indeterminate" ;
qc_down_short_diffuse_hemisp:bit_12_description = "data failed Rayleigh limit test" ;
qc_down_short_diffuse_hemisp:bit_12_assessment = "Bad" ;
qc_down_short_diffuse_hemisp:bit_13_description = "data failed tracker off test" ;
qc_down_short_diffuse_hemisp:bit_13_assessment = "Bad" ;
int aqc_down_short_diffuse_hemisp(time) ;
aqc_down_short_diffuse_hemisp:long_name = "Data Quality Check for down_short_diffuse_hemisp" ;
aqc_down_short_diffuse_hemisp:units = "Unitless" ;
aqc_down_short_diffuse_hemisp:description = "This field contains integer values which should be interpreted as follows:" ;
aqc_down_short_diffuse_hemisp:-1 = "missing data" ;
aqc_down_short_diffuse_hemisp:0 = "data ok" ;
aqc_down_short_diffuse_hemisp:1 = "data too low (UC1)" ;
aqc_down_short_diffuse_hemisp:2 = "data too high (UC1)" ;
aqc_down_short_diffuse_hemisp:3 = "data too low (UC2)" ;
aqc_down_short_diffuse_hemisp:4 = "data too high (UC2)" ;
aqc_down_short_diffuse_hemisp:5 = "data too low (PP)" ;
aqc_down_short_diffuse_hemisp:6 = "data too high (PP)" ;
aqc_down_short_diffuse_hemisp:8 = "data failed Rayleigh limit test" ;
aqc_down_short_diffuse_hemisp:9 = "data failed tracker off test" ;
For s1 level files, see examples below.
Aside from the daily netcdf files, we also generate monthly and yearly summary files and plots to help us further identify problematic data. See next section for examples of these files and plots.
Examples of Output Data and Output Filename Structure
- An example of the c1 level netcdf file: Click here to see the netcdf header of an example output netcdf file from QCRad VAP, which describes each output field, their units, QC definitions, etc.
- An example of the summary (.s1) netcdf file.
- Examples of daily quicklooks
- An example of the monthly summary file
- Examples of yearly statistical and summary plots
Description of Data Quality Fields
N/A
Status and Location of Results
Files are available through the ARM Archive. Place orders using the example file names given above.
Notes for Data Users
N/A
Related Products Data and Links
Ordering Data
Data from the QCRad VAP may be ordered through the ARM website or at the ARM Archive.
To order data from the ARM website, return to the QCRad overview page and select the Add to Cart button on the right side of the page under Order Data.
To order data from ARM Archive, please refer to the instructions at http://www.archive.arm.gov/docs/finding/. Quicklooks are also available at the Archive. Instructions for viewing quicklooks at the Archive can be found at http://www.archive.arm.gov/docs/dataplots/index.shtml.
Links to Notification File
N/A
Link to Data Release History
N/A
Other
Frequently Asked Questions
There are none at this time.
Contacts
- Translator
-
Chuck Long
Phone: 1 (509) 372 - 4917
Fax: 1 (509) 372 - 6168
Email: Chuck.Long@arm.gov - Developer
-
Yan Shi
Phone: 1 (509) 375 - 6858
Email: yan.shi@pnl.gov
VAP Specific Glossary and Acronyms
For ARM-wide acronyms, see the ARM Acronyms/Glossary.
- Diffuse SW
- The downwelling SW from the hemispheric view EXCLUDING the direct SW, i.e. sky radiation.
- Direct normal SW
- That downwelling SW coming from the angular area centered on the sun in the sky, typically measured with a normal incidence perheliometer with a 5O field-of-view.
References
Long, C. N., and Y. Shi, (2008): An Automated Quality Assessment and Control Algorithm for Surface Radiation Measurements, TOASJ, 2, 23-37, doi: 10.2174/1874282300802010023.
Long, C. N. and Y. Shi, (2006): The QCRad Value Added Product: Surface Radiation Measurement Quality Control Testing, Including Climatologically Configurable Limits, Atmospheric Radiation Measurement Program Technical Report, ARM TR-074, 69 pp. Available via http://www.arm.gov/publications/tech_reports/arm-tr-074.pdf.
Shi, Y. and C. N. Long, (2006): Surface Radiation Measurement Data Quality Assessment at the ARM TWP and NSA Sites, 16th ARM Science Team Meeting Proceedings, Albuquerque, New Mexico, March 27 - 31, 2006. Available online at http://www.arm.gov/publications/proceedings/conf16/extended_abs/shi_y.pdf.
Shi, Y. and C. N. Long, (2005): Examples of Detecting Measurement Errors with the QCRad VAP, 15th ARM Science Team Meeting Proceedings, Daytona Beach, Florida, March 14-18, 2005. Available online at http://www.arm.gov/publications/proceedings/conf15/extended_abs/shi_y.pdf.
Shi, Y. and C. N. Long, (2004): Techniques and Methods used to determine the Best Estimate of Total Downwelling Shortwave Radiation, 14th ARM Science Team Meeting Proceedings, Albuquerque, New Mexico, March 22-26, 2004. Available online at http://www.arm.gov/publications/proceedings/conf14/extended_abs/shi-y.pdf.
Younkin, K. and C. N. Long, (2004): Improved Correction of IR Loss in Diffuse Shortwave Measurements: An ARM Value Added Product, Atmospheric Radiation Measurement Program Technical Report, ARM TR-009, Available online at http://www.arm.gov/publications/tech_reports/arm-tr-009.pdf.
Shi, Y., and C. N. Long, (2003): Preliminary Analysis of Surface Radiation Measurement Data Quality at the SGP Extended Facilities, 13th ARM Science Team Meeting Proceedings, Broomfield, Colorado, March 31-April 4, 2003. Available online at http://www.arm.gov/publications/proceedings/conf13/extended_abs/shi-y.pdf.
Shi, Y. and C. N. Long, (2002): Best Estimate Radiation Flux Value Added Product: Algorithm Operational Details and Explanations, Atmospheric Radiation Measurement Program Technical Report, ARM TR-008, Available at http://www.arm.gov/publications/tech_reports/arm-tr-008.pdf.
