GTSPP Hobart, Mar, 2002
Meeting Report
The GTSPP Ad Hoc meeting began at 0900 at the CSIRO in Hobart, Tasmania on March 18, 2002. Participants were Charles Sun (NODC), Bob Keeley (MEDS), Loic Petit de la Villeon (IFREMER), Gary Meyers (CSIRO), Peter Hacker (IPRC), Lisa Cowen (JAFOOS), Krystyna Jankowska (AODC), Ann Gronell (CSIRO), Rick Bailey (JAFOOS). The agenda for the meeting is included in Annex A. Action items extracted from the text are contained in Annex B.
Some of the participants of the meeting were unfamiliar with recent developments of the GTSPP, so Bob provided a quick review. He discussed the objectives, the data flow, and some of the reasons behind how the program was structured.
Because the agenda was quite full, not all of the topics were discussed in detail during the one day session. Where this occurred, it is noted in the meeting report and actions will be followed up by email.
1. Review status of WOCE V3CD production
Since the production of the final CDs for WOCE would be discussed in the following days, it was of highest priority to review the detailed preparations needed for the UOT CD. Charles reported that the 1998 data had all been returned from the Science Centre QC and were being loaded in the archives at NODC. The data from 1990-1998 which were received at NODC after those years had been reviewed by Science Centres should be sent out during this meeting or the following week. There are about 7,000 stations from the Indian Ocean, 30,000 from the Atlantic Ocean and 48,000 from the Pacific Ocean. In addition to these there are about 30,000 delayed mode stations that have not yet been added to the NODC archives and would contribute more stations to each ocean. Considering the short time schedule to complete the QC on these data by the Science Centres (a final version of the CD must be provided to NODC by 1 May), it is unreasonable to expect that all of these data can be QC'ed in time. In line with the recommendations made by both DPC and at the previous GTSPP meeting, NODC will create a separate directory to hold unQC'ed data. This will include all data from 1999 to 2000, but whatever other of the delayed mode data that Science Centres cannot deal with in time to meet CD production schedules.
Charles reported that he was estimating this CD would contain 1.4 million stations and would certainly spill over to 2 CDs. He has taken the draft web pages provided by Bob and modified them as appropriate. He has also included JavaScript to allow for easier selection of data and graphics.
Concerning netCDF files, Charles noted that the use of dimension names that are identical to attribute names will continue to cause problems for the automatic generation of FORTRAN code to read the files. He will bring this to the attention of the DPC meeting.
Charles reported that Melanie Hamilton had made very good progress getting information from a significant number of data providers about what kind of XBT probes and what fall rate calculations were used. A series of emails prior to the meeting drafted out a strategy for marking profiles so that a user would know if a correction had been made and how. The strategy for doing this was confirmed. Of the two possible ways to correct the fall rate, the simplest is to simply multiply the existing depths by a factor of 1.0336. This will be the technique employed with the multiplication factor stored in the file structure as agreed by email. Finally it was agreed that Science centres can make corrections to depths even when they do not have all of the information about probe types, but when they are sufficiently sure that a correction is appropriate. If NODC or MEDS makes a correction, it will do so only when it has checked that no correction has been applied by a Science Centre and only if there is sufficient information to be sure that a correction is appropriate.
It was agreed that corrections to the depth would be applied only to the data that will be placed on the WOCE CDs, not to the existing archives of GTSPP at NODC. The correction to global archives must be carried out in cooperation with other data centres around the world, and this will need time to organize and care to be sure confusion is not introduced.
Charles reported that he will include on the CD all high density XBT lines from 1990 to 2000. The data from lines that had been QC'ed would be placed in one directory and those not QC'ed would be placed in a separate directory. He noted, however, that the designation of what lines were high density ones and which were not was done by a manual inspection of data rather than being based on a list or determined by software. He noted that the DIU had a list of HD lines, but this was not without mistakes. He was asked to consult with Bert Thomson on this matter to see if the DIU list could be sorted out quickly. He also intended to carry out the same manual inspection method employed last time since time was short in CD production.
Bob proposed and the meeting agreed, that GTSPP should issue an update to the CDs (or DVDs) in approximately 2 years time. This would continue to show the activity of GTSPP in delivering data and products to clients. The participants suggested that we should try to organize a small meeting devoted to drafting the contents of such a CD and setting up a production schedule. Others suggested that we should be looking to extend the Science Centre QC backwards in time from 1990 to improve the compete data set available. Work should be done by email before such a meeting and Bob volunteered to circulate some ideas he had.
A possible venue would be the upcoming Argo Data Management meeting in Ottawa in September of 2002. It is expected that most participants of GTSPP will be present. Unfortunately, though the Argo meeting is scheduled for 3 days, the preceding 2 days are expected to be devoted to The Surface Underway Data Project started last November in Brest. Bob will make suggestions of what might be done.
2. Review new Project Plan draft
Bob had sent a table of contents for the draft by email before the meeting. There were a number of comments that impinge on the plan. Charles noted that GTSPP needed to encourage members from other than developed countries. He noted that Kurt Schnebele thought that China might be a good participant. Their participation depends on what role they are able to play. The new plan intended to describe the functions required by GTSPP and allow participants to choose those functions they could provide and so define the role most appropriate for them.
Loic suggested that the experience of trying to keep IFREMER and NODC files in step (i.e. as mirror images) was both time consuming and never ending. He suggested that IFREMER and NODC should explore how to operate a truly distributed GTSPP archive where both the centres could hold data, but it was unnecessary for each to mirror the other. Although the GTSPP archives would be split over two locations, it was important that it appear as a seamless archive to a client. Bob welcomed the idea. Charles agreed that this was a worthwhile goal and that he and Loic would explore this technically and also gauge the acceptance of such a scheme with their managers.
Charles briefly noted that NODC was combining all of its ocean profile data into a single archive system rather than having a special system for GTSPP as was the case before. Bob remarked that this was a good step since GTSPP goals are to handle all T and S data (as well as carrying other profile types measured in conjunction with T and S). Some significant fraction of the T and S data at NODC has not been through the GTSPP QC procedures and this would be a challenge to get done. The meeting encouraged NODC to both combine the data archives and carry out the QC as this would bring their operations more into accord with the goals of the GTSPP.
Some discussion took place about the activity of individual Science Centres. Rick noted that he was unsure that all centres were continuing to carry out QC functions in the same way as was agreed to at the meeting in Scripps in 1995. He noted that he would like to see detailed documentation of what each centre does. Bob agreed that such documentation was important, but also remarked that he did not have what he thought was required in his possession at the moment.
In addressing Science Centre QC and the desire to extend QC to more data and to data collected before 1990, there is a possible partnership with the Asia-Pacific Data Research Center (APDRC) at the University of Hawaii represented by Peter. One of the tasks of the center is data management and quality control of delayed-mode (ocean) data for climate research. Cooperation with GTSPP activities appears desirable. Over the next month Peter will be writing a proposal which includes a cooperative task with CSIRO on QC of historical temperature profile data. He has offered to circulate the proposal to others at this meeting for information.
Both Bob and Rick remarked that the possible functions listed for Argo Regional Centres had many similarities to the functions that GTSPP Science Centres perform. It was suggested that the part of the project plan dealing with science centres should take these similarities into consideration. The thought was that GTSPP Science Centres might wish to consider offering to participate as Argo Regional Centres. In fact, both AOML and JAFOOS had tentatively done so. Also, the APDRC has offered to serve as a linked member of the Argo regional centers in the Indian and Pacific Oceans.
Bob asked participants to send comments about the project plan to himself, Rick and Kurt by the end of May. They would redraft the entire plan by September of this year.
3. Review new draft brochure
Both Loic and Bob had prepared very preliminary versions of a new brochure. There was little discussion, because of lack of time. Both were asked to distribute their suggestions in a PowerPoint document for comment.
Loic noted that the original brochure and the form discussed briefly at the meeting were a single 8.5 by 11 folded page, He asked if people would be interested in producing a multi-page type document.
4. Technical issues
a. Unique GTSPP tagging to assist duplicates identification
Loic proposed a scheme that would attach tags generated as database IDs with an archive identifier at the start. These would be assigned when the data actually arrived at either NODC or IFREMER. The problem with this scheme was that it did nothing to associate real-time and delayed mode profiles.
Bob presented a scheme designed by people at the Australian Bureau of Meteorology and described at the SOT meeting by Graeme Ball. This scheme was based on attaching a 32 bit number generated through a cyclic redundancy check (CRC). For this to be effective, the CRC identifier must be attached to the delayed mode profile when it is created and based on standard rules about the structure of BATHY or TESAC messages. This received some discussion, but it was clear that for this scheme to work, we would need the cooperation of data providers. Bob agreed to prepare a document that described the scheme in greater detail and set out what conditions would be required to be met in order for this to work effectively. Participants were to review this as quickly as possible with the goal of putting a plan into effect as soon as possible.
b. Data State Indicators for GTSPP
This was only briefly discussed. Bob had circulated a document before the meeting about how this would be applied to GTSPP. Participants agreed to provide comments by the end of May.
c. Minimum data content
This was not discussed. Participants agreed to send comments to Bob by the end of May.
d. GTSPP domain name
Loic reported that IFREMER had purchased www.gtspp.org for the next 2 years. Charles had circulated an email shortly before the meeting asking for comments on a new GTSPP web site he was building. When participants are happy, Charles should use the new domain name. He also suggested that he would set up an email distribution list so that GTSPP contact points in each site would receive emails posted from the new web pages.
Participants are to review Charles' pages and provide comments to him as soon as possible.
e. Preparation of test data
At the GTSPP meeting in Brest, it was agreed that Rick and Bob would be asked to look into preparing test data that could be used to evaluate quality control procedures. This was based on the Bob's reading of an email sent by Rick to the meeting. Rick was not sure this is what he meant and so not willingly to pursue this until it was better defined.
Rick did note that such a test data set may be useful considering the earlier discussion about the uncertainty of the uniformity of what QC was being undertaken at Science Centres. He suggested that each centre could take such a data set, apply their QC and then come to a workshop to discuss comparisons. Bob asked Peter if ADPRC would be willing to host such a workshop, and he replied that it was possible. Bob asked Peter if the APDRC would be willing to host such a workshop. Peter pointed out that Jim Cummings (with Ed Harrison and Breck Owens) as part of GODAE was organizing a virtual QC intercomparison activity prior to a one-day workshop in association with the upcoming “En route to GODAE” symposium in Biarritz, 13-15 June 2002. A follow-on workshop focusing on GTSPP activities may be needed, and could occur in conjunction with the next Argo DM meeting or be hosted later by the APDRC. This needs more consideration, and Rick and Bob will explore it.
f. GTSPP data in Argo netCDF
Participants were asked to review this idea and send comments to Bob.
2. Data Issues
a. Access through web sites - not discussed
b. Code lists – not discussed
c. Review of MEDS, Meteo France differences
Loic noted that he had a French version in hand, but that it had not been translated into English. He will distribute it when available. He noted that the general conclusions were that the data available from Meteo France and MEDS were almost identical, but that those from MEDS had quality flags and so were the more desirable.
d. More frequent data exchanges
This was not discussed directly, but in fact is influenced by discussions under item 2 above.
e. SEAS full resolution data
Bob informed the meeting that the US SEAS program is planning on sending 2m resolution XBT data ashore by mid year. The group immediately asked for greater detail. Bob agreed to ask Steve Cook why the 0.6 m resolution data were not being sent, if the 0.6 m data were still being recorded on floppy disks to come back from the ships later, what QC procedures were being applied to the 2m data before they left the ship, and what was the process for the data going to NCEP? Bob will report results by email.
Loic explained that Coriolis was now receiving 0.6 m data from French ships and that they were applying GTSPP QC before issuing the data to the GTS. Rick stated that Australia was intending to send the full resolution data ashore within the next 12 months.
Rick noted that he considered it a mistake to rely on satellite transmissions as a sole source of data. Ann stated that in order to carry out scientific QC is was very important to have the highest resolution possible. Rick noted that they had semi-automated QC procedures working now that required detailed and reliable means and standard deviations by regions. He said that with reliable statistics, it was possible to significantly reduce the number of profiles that had to be handled manually. With appropriate statistics from other oceans, the procedures should be applicable there. Ann volunteered to look into a test of data from the global ocean but noted that this was not something that could be carried out in time to meet the SEAS deadlines.
Loic noted that as a short-term measure, Coriolis would be willing to accept the SEAS data, carry out the QC and insert them onto the GTS. As the semi-automated procedures were developed, other strategies might be employed. Bob offered to contact Steve Cook and to put him in touch with Loic to work out the details. Rick asked that we be sure that in instituting a changed procedure, that we did not interrupt the data stream to NCEP.
6. Issues arising from JCOMM SOT meeting
a. Unique tag – see discussion in section 4a.
b. SEAS real-time data - see discussion in section 5e.
c. Revise doc on QC procedures to be carried out on board in light of full resolution being sent ashore - not discussed
d. review JCOMMOPS and MEDS products to remove overlaps - not discussed
e. Annual reporting
The SOT meeting agreed to an annual reporting of operations. Since GTSPP is the data system for SOOP, Bob suggested that it would make sense for GTSPP to institute an annual report as well and to meet the same time schedule as for the SOT. This was accepted as a good idea, with the first report to be issued approximately next March and covering the calendar year 2002. It was suggested that the report would include data system performance indicators or various kinds. Rick agreed to construct a template for the report, and Bob would assist.
6. Other business
a. Adjusted XBT fall rates and global archives - not discussed
b. Review actions from Brest - part of agenda items
c. Status of IPRC (http://apdrc.soest.hawaii.edu) discussions
Peter described the functions of the ADPRC (see the brief discussion earlier in section 2). It is a part of the International Pacific Research Center (IPRC). .It has a regional perspective and expertise, being mainly interested in the Pacific and Indian Ocean regions, but requires global data sets for climate research and modeling activities. The goal of the APDRC is to develop the computational, data management, and networking infrastructure to provide available data resources to scientists who are undertaking data intensive research. To further this they are working to improve the Live Access Server (LAS) and EPIC technology so that gridded and in-situ data can be provided with equal ease.
Their interests are in all kinds of profile data including T, S, nutrients, carbon, etc. They wish to prepare value added products and to serve these to interested scientists. In trying to increase the data resource, they recently acquired 20,000 bottle stations from Russia, and these will be forwarded to Syd Levitus for inclusion in the WDC-B archives. He noted that although they plan to help carry out regional QC of data, they had no plans to replace the activities of others at other locations, but rather wanted to accelerate the overall QC effort.
Charles noted that NODC was interested in cooperating with APDRC to help in the scientific verification of data. That is, if in the course of their work, they should find questionable data, a mechanism would be in place to notify NODC and appropriate measures taken to correct the original archives. Rick and others remarked that this service might complement what is presently carried out by Scripps. Charles was requested to discuss this issue with Scripps.
Rick noted that CSIRO is interested in cooperation in examining data from the Indian Ocean. Peter is willing to write a recommended strategy for this in the Indian Ocean and would circulate a plan by July.
The meeting closed at 1700 on 18 March. Bob thanked all for their help.
Annex A: Agenda
GTSPP Agenda – Hobart, Mar, 2002
1. Review status of WOCE V3CD production – NODC, MEDS
2. Review new Project Plan draft – MEDS, JAFOOS, NODC
3. Review new draft brochure - IFREMER
4. Technical issues
a. Unique GTSPP tagging to assist duplicates identification – IFREMER
b. Data State Indicators for GTSPP – MEDS
c. Minimum data content – MEDS
d. GTSPP domain name – IFREMER
e. Preparation of test data – MEDS and JAFOOS
f. GTSPP data in Argo netCDF – IFREMER, NODC
5. Data Issues
a. Access through web sites – NODC, IFREMER
b. Code lists – NODC, MEDS
c. Review of MEDS, Meteo France differences – IFREMER
d. More frequent data exchanges – NODC, IFREMER
e. SEAS full resolution data – NODC
6. Issues arising from JCOMM SOT meeting – JAFOOS
a. Unique tag – BOM see above
b. SEAS real-time data
c. Revise doc on QC procedures to be carried out on board in light of full resolution being sent ashore.
d. review JCOMMOPS and MEDS products to remove overlaps.
7. Other business
a. Adjusted XBT fall rates and global archives
b. Getting broader participation
c. Review actions from Brest
d. Status of IPRC discussions
Annex B: Action Items
Item 1: Review status of WOCE V3CD production
a. Charles was asked to consult with Bert Thomson to see if he can use the HDX lines listed by the DIU.
b. Bob will circulate ideas for a GTSPP CD to be produced in about 2 years time.
c. Bob will try to make use of another meeting at which a significant number of GTSPP participants are present to discuss the ideas of the CD for GTSPP.
Item 2: Review new Project Plan draft
a. Loic and Charles to explore the technical issues of IFREMER and NODC operating a distributed archive for GTSPP.
b. Peter will circulate the draft proposal of cooperation between APDRC and CSIRO for information.
c. Participants to send comments by the end of May to Bob, Rick, Kurt about the draft GTSPP project plan
Item 3: Review new draft brochure
a. Bob and Loic to distribute PowerPoint versions of draft brochures to participants for comment.
Item 4: Technical Issues
a. Bob will circulate a document that describes how the unique tag proposed by BOM could be used in GTSPP.
b. Participants to send comments by the end of May to Bob on the use of Data State Indicators in GTSPP.
c. Participants to send comments by the end of May to Bob on the minimum file content for delivering GTSPP data.
d. Charles will use the new GTSPP domain name in the GTSPP web site revision he is completing.
e. Participants should review the new GTSPP web site at NODC and provide comments to Charles as soon as possible.
f. Participants were asked to consider if GTSPP data should be made available in Argo formats. Comments should be sent to Bob.
Item 5: Data Issues
d. Loic to distribute as soon as possible a document that compares MEDS and Meteo France data streams for real-time data.
e. Bob will ask Steve Cook about the details of how SEAS data will be transmitted ashore.
f. Ann will look into testing data from many ocean areas to see how well the semi automated QC procedures developed by CSIRO could be used elsewhere.
Item 6: Issues arising from JCOMM SOT meeting
a. Participants agreed to produce an annual GTSPP report . Rick to lead and Bob assist in drafting a template for SOT. First report due, in March 2003 to cover calendar year 2002.
Item 7: Other business
a. Charles was asked to discuss with Scripps how activities at APDRC might complement the work carried out at Scripps in support of GTSPP.