The Rich Transcription 2007 Meeting Recognition evaluation was held during the Spring of 2007 in conjunction with the 2007 Classification of Events, Activities and Relationships (CLEAR) Evaluation and Evaluation Workshop.

The RT evaluation series is a collaborative effort of several projects: The Augmented Multi-Party Interaction (AMI) program, the Computers In the Human Interaction Loop (CHIL), the NIST Speech Group, and the Video Analysis and Content Exploitation (VACE) program.  The RT evaluation was the 5th open evaluation of technologies for the meeting domain.  The presentations and preliminary system description papers from the RT Workshop held May 10-11, 2007 can be found on the Agenda from the meeting.

We are currently beginning to plan the next RT evaluation.  If you'd like to join the RT community and/or participate in the evaluation, send your inquiries to rteval@nist.gov.

Evaluation Planning

• Minutes of the Tuesday, October 17, 2006 conference call
• Minutes of the Tuesday, November 14, 2006 conference call
• Minutes of the Tuesday, December 5, 2006 conference call
• Minutes of the Thusday, December 21, 2006 conference call
• Minutes of the Friday, January 19, 2007 conference call
  

Contributing Conference Room Meeting Recordings for the Evaluation Set

• The data must be delivered to NIST by December 13, 2006
• The meetings must be from the conference room meeting sub-domain which means: there must be 2-12 meeting participants and the meeting should be goal-oriented.
• Each participant must wear a close talking, cardiod or super-cardiod microphone.
• There must be at least 3 table top microphones located between the particiapants, one of which should be designated as the most-central microphone.
• There must be at least one digital video stream.  NIST will distributed mpeg-2, NTSC, 30 frames-per-second, XX I-frames per second.  If you are not collecting this kind of video, the video must be of sufficient quality to trans-code it to the distribution format.
  
• The microphone recordings and video recordings must be synchronized.  If the data collection infrastructure can not provide the synchronization, the meeting recordings must include directors "claps" before the meeting begins and after it ends using a modified directors clapboard that includes a simultaneous flash.  The paper "Synchronizing Multimodal Data Streams Acquired Using Commodity Hardware" (Michel, Stanford) contains a description of how the director's clapboard is used by NIST to synchronize its audio and video streams.  A modified clapboard is easy to make from the pictures of NIST's clapboard.
  

Evaluation Agreements and Data Licenses

Participants of the RT-07 evaluation must complete and return to NIST the RT-07 Evaluation Participation Agreement and the CHIL Data use license. Participants must also complete the AMI license which is available online at the URL corpus.amiproject.org. There are evaluation agreement and CHIL license documents that must be downloaded:


• RT-07 Evaluation Agreement : The evaluation participation agreement contains the fax number and address for mailing the forms if a fax is unavailable.
• The CHIL licensing agreement: The forms must be mailed to ELDA using the address within the document.
  

No training data or evaluation data will be distributed until NIST receives the above forms.

Data Resources

Participants of the evaluation are given access to RT data resources. The password protected page contains textual resources for previous year's evaluation corpora and links/pointers to other resources.  Participants may be eligble to recieve selected  LDC corpora for free.  Contact NIST for details.

Software

• SCTK v2.2.4 20070501-1541 The latest version of SCTK.  This package includes support for the SASTT task and now includes the rttmSmooth.pl, rttmSort.pl, and md-eval.pl scripts.
• SASTT-kit-v2 The starter kit for SASTT scoring. 
  
• spkr2sad-v1.pl Convert an RTTM reference file for the Speaker Diarization task (or system output) to a Speech Activity Detection RTTM file. The output must be smoothed with rttmSmooth.pl to remove the overlapping segments.

RT-07 System Description Papers and Workshop Presentations

RT Participants are expected to prepare a publication-quality system description paper for each task they participated in and presenters are expected to provided PDFs of their slides for the workshop.  The papers and presentations will be made available online during the RT-workshop and then later on the RT website. The submission deadline for papers is May 3, 2007  and the deadline for submitting PDFs of your presentations (two slides per page) is May 7, 2007.

A workshop proceedings will be published by Springer under the LNCS bookseries jointly with CLEAR. Therefore we ask you to submit revised final versions of your papers after the workshop. The tentative deadline will be June 1, 2007. A copy of the proceedings will be send to each workshop participant.

Please use the Springer LNCS style to format your paper. A style file for Latex2e can be found here. Further style files for Latex, Word and Framemaker can be found on Springer's website.

Paper Length: You may submit a paper of up to 12 pages for every evaluation task in which you participated.

Please send your papers and presentations to Jerome Ajot (ajot at nist dot gov).

RT-07 Schedule