4 km AVHRR Pathfinder Current Status

• 14 April 2009 (4:50 pm, EDT): In accordance with the RSS entry from 07 April 2009, the following redundant directories containing final Pathfinder data from 2002-2006 and interim Pathfinder data from 2002-2007 have been deleted from the HTTP and FTP servers: ../pathfinder/Version5.0_NOAA16/, ../Version5.0_NOAA17/, ../Version5.0_NOAA18/, ../Version5.0_interim_NOAA16/, and ../Version5.0_interim_NOAA17/. For access to all Pathfinder data, please use the newly consolidated ../pathfinder/Version5.0/ and ../Version5.0_interim directories.
• 13 April 2009 (11:38 am, EDT): For the first time in the nearly 20-year history of the AVHRR Pathfinder program, data for September of 1981 through January of 1985 from the NOAA-7 Polar Operational Environmental Satellite are now available. These data were produced using Version 5.1 of the Pathfinder algorithm, which uses a higher resolution reference data set than Version 5.0. This reference set, the Reynolds Optimum Interpolation 1/4-degree Daily SST Analysis Version 2, is used as a first guess SST and in the quality control procedures. The primary effect of this change is to retain more data in the coastal and high gradient regions and in regions where meandering or feature advection is present. All of the Pathfinder data are available via ftp, http, and the NODC Ocean Archive system.
• 07 April 2009 (4:19 pm, EDT): Formal Pathfinder Version 5 data for 1985 through 2006 have now been consolidated under a single directory on the HTTP and FTP servers. This directory is, for HTTP: http://data.nodc.noaa.gov/pathfinder/Version5.0/ and for FTP: ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0/. Interim Pathfinder Version 5 data for 2002 through 2007 have also been consolidated under a single directory. This directory is, for HTTP: http://data.nodc.noaa.gov/pathfinder/Version5.0_interim/ and for FTP: ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0_interim/. The following redundant directories and all subdirectories will be deleted from the HTTP and FTP servers on April 13, 2009: ../Version5.0_NOAA16/, ../Version5.0_NOAA17/, ../Version5.0_NOAA18/, ../Version5.0_interim_NOAA16/, and ../Version5.0_interim_NOAA17/. Please note this is simply a change in the location of Pathfinder Version 5 data files; the content of all files will remain unchanged.
• 06 February 2009 (3:34 pm, EDT): Due to an error with the AVHRR instrument on board NOAA-18, data for some daytime pixels may be missing. There was a problem with the daytime quality flags for NOAA-18 which resulted in removing some valid data within the sunglint region. This is a minor error and does not affect the quality of the data that are present. All data from NOAA-18 are being reprocessed to correct this error and will be made available as soon as possible. Affected data include final data from day 156 of 2005 through 2006, and interim data for 2007.
• 18 August 2008 (1:33 pm, EDT): Formal data for 2002 and 2004-2006 have been added to the Pathfinder web site, along with interim data for 2007. The formal data are available under ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0 and http://data.nodc.noaa.gov/pathfinder/Version5.0, along with the earlier 1985-2001 data. The older directory paths will be going away shortly, so please update your links. OPeNDAP remains offline but we hope it will be available again soon.
• 12 October 2007 (06:40 am, EDT): An RSS news feed was established for the Pathfinder Project. Users subscribing to this feed will receive occaisional updates regarding newly-available data, problems identified with the data, and other items of interest to the Pathfinder user community.
• 03 July 2007 (07:20 am, EDT): The volume tables in the Quality Assurance section have been fixed to properly follow symbolic links, and so now give accurate representation of the data volumes, even for the files that are in the archive.
• 30 May 2007 (03:15 pm, EDT): Interim Version 5.0 data for all of 2006 are now available! Our OPeNDAP server is currently offline due to IT security issues, but access through HTTP ( http://data.nodc.noaa.gov/pathfinder/Version5.0_interim_NOAA17 and FTP ( ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0_interim_NOAA17) is still possible. Data are available for all averaging periods. The Data Available page has been updated to reflect the newly available 2006 data.
• 23 April 2007 (08:18 am, EDT): The User Guide and Data Available pages have been updated to include Live Access Server and GoogleEarth data access mechanisms through our colleagues at NOAA's Southwest Fisheries Science Center.
• 18 December 2006 (08:40 am, EST): Thanks to Jason Roberts of Duke University for pointing out some extraneous files in the Daily, 1999 directory. Please ignore the following files in that directory:

  1999001.m04d1pfv50-bsst.hdf
  1999001.m04d1pfv50-sdev.hdf
  1999001.m04d1pfv50-sst.hdf
  1999001.m04d3pfv50-bsst.hdf
  1999001.m04d3pfv50-sdev.hdf
  1999001.m04d3pfv50-sst.hdf
  

  Each of the above files has a corresponding file with an "s04" iinstead of "m04" and "16b" in the file name which should be used instead.
• 24 August 2006 (11:00 am, EST): The problem noted below on 18 August has been corrected. All of the affected Pathfinder data files have been recovered and restored to their proper condition. The problem was limited to the 1985 trough 2001 yearly directories.
• 18 August 2006 (1:00 pm, EST): PLEASE NOTE - We are experiencing a problem with many of the Pathfinder HDF files which have been corrupted. You will be able to tell a file has been corrupted if it is only a few kilobytes in size... normal Pathfinder HDF file sizes are on the order of megabytes. We are working to resolve this problem. Thank-you for your patience.
• 11 July 2006 (02:30 pm, EDT): Thanks to Jason Roberts of Duke University for reporting that one of the 1994 8-day SST files was not able to be read. While the file in question is partly accessible, trying to read the entire contents of the data array appears to hang various applications. The corrupted file is (http and OPeNDAP access are also impacted): ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0/8day/1994/1994329-1994336.s0483pfv50-sst-16b.hdf We are working to fix the problem. In the meantime, you can access the working verison of this file from the archive at: http://www.nodc.noaa.gov/archive/arc0003/0001806/01-version/data/0-data/8day/1994/1994329-1994336.s0483pfv50-sst-16b.hdf
• 28 June 2006 (11:45 am, EDT): Thanks to Scott Heron of NOAA's Coral Reef Watch project for finding a bad quality flag file: the file http://data.nodc.noaa.gov/pathfinder/Version5.0/Daily/2000/2000294.m04d1pfv50-qual.hdf has a patch of erroneous quality flags extending from Florida, USA through parts of the Gulf of Mexico and down to northernm South America. The quality flags appear uniformly too high and the corresponding SST data in this region should not be used. You can see the problem in the preview graphic: http://data.nodc.noaa.gov/pathfinder/Version5.0/browse_images/Daily/2000/QualityFlag/2000294.m04d1pfv50-qual.hdf.jpg.
• 28 June 2006 (7:30 am, EDT): To avoid possible confusion regarding older NOAA16 interim data for 2003 and 2004 that still reside on the server (both of which were updated back in March of 2005 to use NOAA17 data instead), several directories under Version5.0_interim_NOAA16 have been renamed. All the NOAA16 interim 2003 data directories called "2003" have been renamed "2003_USE_Version5.0_NOAA17_INSTEAD", and all the NOAA16 interim 2004 data directories have been renamed "2004_USE_Version5.0_interim_NOAA17_INSTEAD".
• 28 June 2006 (6:00 am, EDT): Good news: We have found funding for the Pathfinder project for another year. Production of interim 2006 data has commenced and should be available for the first 5 months of 2006 in early July.
• 13 March 2006 (2:00 pm, EST): All 5-day, 7-day, 8-day, monthly, yearly, and daily files for the remainder of 2005 have been added to the collection, quality-controlled, and placed on-line. At this point, there is no funding to continue the processing of AVHRR Pathfinder data so unless this situation changes there will be no data for 2006 and beyond, or for years prior to 1985. See the recent PO.DAAC Notice for more information
• 07 March 2006 (2:30 pm, EST): Thanks to Grayson Wood of NOAA's National Marine Fisheries Service for identifying an inconsistency in the way we've named the variables inside the V5 Pathfinder data. The names all make sense except for the standard deviation files (with sdev in the filename), which were inadvertantly named "sst" instead of "sdev". Here is the list for completeness:
  • SDSname = mask1 for files with msk1 in filename
  • SDSname = mask2 for files with msk2 in filename
  • SDSname = num for files with num in filename
  • SDSname = qual for files with qual in filename
  • SDSname = bsst for files with bsst in filename
  • SDSname = sst for files with sst in filename
  • SDSname = sst for files with sdev in filename
• 19 January 2006 (1:00 pm, EST): Interim data for 2005 are now available through day 309, week 44, and month 10. Data for the 5-day and 8-day averages are also available through that time. All portions of the web site, user guide, and data access tables have been updated to reflect these newly available data.
• 29 December 2005 (2:15 pm, EST): An entry was made in the Known Problems section of the User Guide describing a warning to users of the data that they may need to apply a "reality check" to the data to make sure they aren't using pixels with a physically unreasonably cold SST. Surface SST values do not fall below -1.8 or -1.9 degrees C. A pixel value of 16 corresponds to an SST of -1.8 deg C. Values that are unreasonably cold can sometimes survive the quality flagging procedures at high latitudes. The text of the entry follows: Users of the data, especially in high latitudes, are cautioned to put a "reality check" on the data even after applying the quality flags. In some cases, the quality checks that would normally exclude unreasonable SST values may fail (the two most relevant checks are the ones on the brightness temperatures and the comparison against the OISSTv2 analysis). In high latitudes where the SSTs can drop to as much as -1.8 or -1.9 degrees C these checks can sometimes fail, resulting in SSTs with values as cold as -2.925 degrees C (pixel values between 1 and 15) that still have high quality levels. Since surface ocean temperatures can not become this cold, these data should be masked out. This problem occurs in the time series data as well as in the resulting climatologies.
• 28 September 2005 (9:45 am, EDT): A corrupted file, Daily/1993/1993249.s04d1pfv50-sst-16b.hdf, was discovered and replaced with the correct copy from the archives. If you have downloaded that file in the past we recommend you verify that it can be opened properly. You can also verify its md5 checksum, which should be: 596349b7b2df5af9d2b6f04c47e60f73
• 17 August 2005 (9:30 am, EDT): Example Code for reading the Pathfinder data in Matlab, IDL, and Fortran has been added to the web site.
• 01 June 2005 (1:30 pm, EDT): All of the browse imagery for NOAA-17 2003 and interim NOAA-17 for 2004 and 2005 has been updated. A new category of browse imagery was also added: declouded SSTs using only pixels with an overall quality flag of 7. All of the new browse imagery also contains a color scale and daytime or nighttime designations.
• 25 May 2005 (10:30 am, EDT): In order to comply with a broader naming convention that includes NASA MODIS data on Terra and Aqua, the NOAA-17 files for formal 2003 and interim 2004 and 2005 have been given day-night flags of "2" for daytime and "4" for nighttime, instead of the "3" for daytime and "1" for nighttime used in all of the previous Pathfinder data files from NOAA-9, 11, 14, and 16. This change was necessary because NOAA-17 is the first "morning" satellite (meaning its daytime observations are collected around 10 AM in the morning) used in the Pathfinder processing, while all previous platforms used in Pathfinder were "afternoon" satellites (meaning their daytime observations are taken around 2 PM in the afternoon). The User Guide was updated to reflect this change. Note, however, that the browse imagery is still being regenerated.
• 24 May 2005 (3:30 pm, EDT): The interim 2004 and 2005 data from NOAA-17 found in the directories called "Version5.0_interim_NOAA17" and the formal NOAA-17 data for 2003 found in "Version5.0_NOAA17" have been renamed to fix the day-night naming problem described below. The browse images have also been renamed but they must be completely regenerated since the name of the file appears on the graphic itself. A note will be made on this page when the interim NOAA-17 2004 and 2005 browse images and the formal 2003 NOAA-17 browse images have been recreated.
• 24 May 2005: A change was made to the User Guide to better describe the file naming convention.
• 24 May 2005: Important note: All of the NOAA-17 data (both formal data for 2003 and interim data for 2004-2005) are incorrectly named. If you have downloaded any of these data (prior to 3:30 pm EDT, 24 May 2005 -see correction note on that date/time above), you must reverse the day and night designations (simply reversing the file names will fix the problem). The problem is being corrected now and an announcement will be made on this web page when the properly named files will be available for download. This problem occured because NOAA-17 is in a different orbit (see the User Guide for more information on overpass times) than all of the previous satellites (NOAA-9, 11, 14, and 16) used in the Pathfinder program. In the meantime, all of the NOAA-17 files on the web site currently are incorrectly named.
• 05 May 2005: A new batch of interim NOAA17 data for 2005 is now available via ftp, http, and OPeNDAP. The Available Data table has been updated and all browse graphics have been generated. We now have daily data through 2005106, 5day data through 2005095, 7day data through 200515, and 8day data through 2005096. No monthly or annual data are yet available for 2005
• 12 April 2005: An error in the description found in the User Guide concerning the use of the sea ice masking in V5.0 was corrected.
• 02 March 2005: Formal data for 2003 have been added to the Pathfinder dataset. In addition, interim data for 2002, and 2004-near present have also been added. All of the site web pages and user guide have been added to reflect this new data availability.
• 25 February 2005: Climatologies corresponding to each averaging period are now available as part of the Pathfinder Version 5 collection. See the README.txt file for full information. The climatologies, like the rest of the Pathfinder Version 5 data, are available via http ftp, and OPeNDAP
• 17 November 2004: Several changes were made to the Pathfinder Version 5 User Guide including the additional of information regarding satellite overpass times and drift rates.
• 05 October 2004: Additional tables documenting the number of files and the volumes contained in the browse image and data directories are now being generated hourly and are being made available through the Quality Assurance page.
• 27 September 2004: A problem has appeared with the daily number of observations file: Daily/1993/1993153.m04d1pfv50-num.hdf. Although it previously passed quality control procedures it now appears that this file can no longer be opened and read. This problem is being investigated.
• 07 September 2004: Yet another error in the land mask was discovered, this one in the Antarctic. A graphic describing the problem is available in the Known Problems section of the Draft User Guide. These problems will be corrected in the Version 5.1 reprocessing which we hope will commence this fall.
• 01 September 2004: Another error in the land mask was discovered on the Grand Bank off Newfoundland. A graphic describing the problem is available in the Known Problems section of the Draft User Guide. These problems will be corrected in the Version 5.1 reprocessing which we hope will commence this fall.
• 28 June 2004: FTP access was not functioning since June 24, but has been restored. We apologize for the inconvenience.
• 22 June 2004: Several updates were made to the web site today, including new information on accessing the data via the OPeNDAP Data Connector * and more complete and accurate georeferencing information. These changes have been made in the Available Data page and in the Draft User Guide page.
• 11 June 2004: We are pleased to announce that some of the Version 5.0 data are now available through the JPL PO.DAAC! * Access to daily SST and quality flag files is available via direct ftp * as well as through the online subsetting and display system POET. * Various pieces of the NODC web site are being updated over the next week to reflect these new data access methods.
• 06 April 2004: Information regarding which NOAA satellites were used for the Pathfinder V5 dataset was added to the Draft User Guide, as was information regarding an error discovered in the landmask off the east coast of South America.
• 25 March 2004: Interim data are being generated for 2002, 2003, and for 2004 within about a week of real-time. The processes to put these interim products online is being established now. Within a few weeks these data should be available online. The interim data use the latest available coefficients (currently, those are from 2001) and will be phased out when true coefficients are developed for these more recent data. It is expected, however, that these interim data will still be very useful for many purposes.
• 25 March 2004: A Calendar Date Conversion Chart was added to the Draft User Guide to facilitate converting between YYYYDDD and calendar dates.
• 03 March 2004: The six missing monthly files identified below on 29 January have been generated and placed on the server.
• 29 January 2004: Data and corresponding browse images have been added for the missing data in 1996, including the 5-day, 7-day, 8-day, monthly, and yearly averaged fields. However, six monthly data files are still missing. The cause is being investigated now and the files will be replaced as soon as possible. The six missing monthly files are:
  • 199606.m04m3pfv50-msk1.hdf
  • 199606.m04m3pfv50-msk2.hdf
  • 199606.m04m1pfv50-num.hdf
  • 199606.m04m3pfv50-num.hdf
  • 199606.m04m1pfv50-qual.hdf
  • 199606.m04m3pfv50-qual.hdf
• 15 January 2004: Data and corresponding browse images have been added for the year 1996. However, a total of 55 days of data are missing including a large gap spanning days 126 through 167. These gaps are a result of repeated DLT tape failures. Level 1B AVHRR files are being retrieved from NOAA CLASS, and will be processed to fill these gaps. The yearly data and other affected temporally averaged fields are also absent and will be replaced when the daily data are recovered.
• 10 December 2003: Data and corresponding browse images have been added for the year 2001.
• 08 December 2003: A DRAFT User Guide has been made available on line. While incomplete and still very rough, the user guide provides some details not available on the rest of the web site.
• 08 December 2003: The problem with the 8-day SST file for year 2000, 2000089-2000096.s0483pfv50-sst-16b.hdf, has been corrected. The new file and its corresponding browse image have been updated on the server.
• 03 December 2003: A problem has been identified in an 8-day average SST file for year 2000, days 89-96: 2000089-2000096.s0483pfv50-sst-16b.hdf. An obviously artificial band of missing data in the southern hemisphere is present. The problem is currently being investigated.
• 03 December 2003: Data and corresponding browse images have been added for the year 2000. Both 1996 and 2001 are nearly complete save a few short gaps. Those two years will be added to the server once the gaps have been filled.
• 03 November 2003: Data and corresponsing browse images have been recovered and generated for two sets of missing files from the 1993 8 day averaging period. These files, corresponding to the 1993009-1993016 and 1993217-1993224 periods, have been placed on the server.
• 30 October 2003: Several issues related to small gaps are currently being addressed:
  1. From 1985, days 001, 002, and 003 are absent and will not be produced because those three days were observed by an earlier NOAA satellite than the rest of 1985. In the coming year, our ability to process that satellite, which covers 1981-1984 as well, is being addressed. At this point it appears likely we will be able to produce those SSTs, but no timeline has yet been set.
  2. Also from 1985, days 040, 041, and 042 are also absent. Attempts to recover these data are being made now.
  3. Data for the 8-day files for 1993009-1993016 are missing and are being regenerated now. The corresponding browse images for that 8-day period are also missing, as are the browse images for the 1993217-1993224 period.
• 20 October 2003: Data and browse imagery for all years and periods from 1985 through 1999, except 1996, are now online and available. A small gap in the 1996 processing is still being filled. When it is complete the entire year will be put online.
• 15 October 2003: Data and browse imagery for the 8-day, Monthly, and Yearly means for 1995 have been added to the server. All years and averaging periods for 1985-1995 are now on-line. Data for 1997-1999 have been processed and are currently being transfered from RSMAS. These new files will be fully integrated on the NODC server within about one week. Processing for 2000 and 2001 has begun.
• 02 October 2003: Large amounts of new data have been added to the server, including all averaging periods for 1985-1989, as well as daily, 5-day, and 7-day files for 1995. Corresponding browse images have also been updated (except for the daily browse images, which are still being generated). It is estimated that by 10 October the entire 1985-1999 time series will be complete. Official coefficients for 2000 and 2001 have also been recently released, so these years will be processed in the next couple of weeks.
• 11 September 2003: The problems with the OPeNDAP server have been resolved, and now core OPeNDAP functionality appears to be working normally, including basic browsing and clicking on OPeNDAP file names to bring up the HTML form.
• 04 September 2003: Quality assurance procedures identified a set of missing files in the 8day/1993 directory. These files correspond to the periods 1993009-1993016 and 1993217-1993224 (YYYYDDD). These files are being regenerated and will be placed on the server shortly.
• 04 September 2003: Yearly files for 1990 through 1994, along with their corresponding browse images, were added to the server.
• 04 September 2003: Monthly files for 1992, 1993 and 1994, along with their corresponding browse images, were added to the server.
• 04 September 2003: 8-day files for 1992 and 1993, along with their corresponding browse images, were added to the server.
• 04 September 2003: 7-day files for 1986 and 1988, along with their corresponding browse images, were added to the server.
• 04 September 2003: 5-day files for 1992 and 1993, along with their corresponding browse images, were added to the server.
• 02 September 2003: Weekly files for 1992, 1993, 1997, 1998, and 1999 were added to the server. Corresponding browse images were also added.
• 01 September 2003: Direct FTP access to all of the files has been enabled at ftp://data.nodc.noaa.gov/pub/data.nodc/pathfinder/Version5.0/.
• 24 July 2003: A Quality Assurance page was added to the web site to clairfy steps taken to insure the intergrity and availability of the data files.
• 24 July 2003: A complete set of browse images for the data available to date have been created and placed on line. They may be accessed at http://data.nodc.noaa.gov/pathfinder/Version5.0/browse_images. These images are very small and very coarse looks at the available data files (only one of every thirty-two pixels in latitude and longitude is displayed). They are part of the quality assurance system, enabling rapid visual inspection for any gross file errors and testing file access and format integrity. The web site has been updated in several places to reflect the availability of the browse images.
• 24 July 2003: Of the over 40,000 files currently on the system, eight were identified as missing during a systematic review performed yesterday and today. The missing files were all from the 1990, 8-day average period for days 017-024. The missing files have been recovered and placed on the server. The missing files were:
  1. 1990017-1990024.m0481pfv50-msk1.hdf
  2. 1990017-1990024.m0481pfv50-msk2.hdf
  3. 1990017-1990024.m0481pfv50-num.hdf
  4. 1990017-1990024.m0481pfv50-qual.hdf
  5. 1990017-1990024.m0483pfv50-msk1.hdf
  6. 1990017-1990024.m0483pfv50-msk2.hdf
  7. 1990017-1990024.m0483pfv50-num.hdf
  8. 1990017-1990024.m0483pfv50-qual.hdf
• 11 July 2003: This change log was added to the data server at http://data.nodc.noaa.gov/pathfinder/Version5.0.
• 10 July 2003: The land mask used in the Pathfinder v5.0 processing was added to the data server at http://data.nodc.noaa.gov/pathfinder/Version5.0/pfv50_land.m04.hdf (Accessible via OPeNDAP at http://data.nodc.noaa.gov/cgi-bin/nph-dods/pathfinder/Version5.0/pfv50_land.m04.hdf)
• 08 July 2003: Several new batches of files have been added to the server: Monthly means for 1990 and 1991; 8-day means for 1990, 1991, and 1994; 7-day means for 1990 and 1991; and 5-day means for 1990, 1991, and 1994.
• 23 May 2003: The file Version5.0/Daily/1991/1991308.s04d3pfv50-sdev-16b.hdf was discovered to have been corrupted during the transfer process to NODC. The corrected file has been placed on the server and the incorrect version deleted.
• 23 May 2003: The file Version5.0/Daily/1990/1990006.m04d3pfv50-num.hdf was discovered to have been corrupted during the transfer process to NODC. The corrected file has been placed on the server and the incorrect version deleted.
• 22 May 2003: Daily and (correctly computed) 7-day average files for 1994 have been placed on the server.
• 22 May 2003: The incorrect weekly files (1990-1992 and 1999) described in the note below have been removed from the server.
• 15 May 2003: A problem has been discovered with the 7-day average files (found under "7day"). A processing error resulted in lower quality observations going into the mean SSTs than was intended. The mistake has been corrected and new files will be reissued soon. An announcement will be made here when the new files have been posted. For the moment, the incorrect files will remain on the server for developmental purposes.
• 25 April 2003: Daily and 7-day average files for 1993 have been placed on the server.
• 15 April 2003: 4 Years of Daily and Weekly data (1990, 1991, 1992, and 1999) are now online and accesible via http and OPeNDAP. Please see the Available Data page for more details and access to the data.
• 1 August 2002: Web site established on NODC intranet developmental server
• 21-23 August 2002: 4km Pathfinder team meeting held in Miami, FL