RT Fall 2004 Evaluation

Information Technology Lab, Information Access Division

NIST: National Institute of Standards and Technology

Speech Group Home

Benchmark Tests

Rich Transcription Fall 2004 Evaluation

The Rich Transcription Fall 2004 Evaluation (RT-04F), implemented in October 2004, included STT tasks for broadcast news speech and conversational telephone speech in English, Mandarin, and Arabic and MDE tasks also for broadcast news speech and conversational telephone speech but only for English.

Documentation

RT Fall 2004 Evaluation Plan, v14, 8/30/04 [pdf, doc]
Simple Metadata Annotation Specification, v6.2, 2/3/04 [pdf]
Arabic STT Scoring Appendix, v5, 8/20/04 [pdf, doc]
Arabic STT GLM Rule Set [ar20040907.glm]

Format Conversion Software

RTTMValidator, v04 - validates RTTM file
ag-to-rttm+uems, v20 - converts LDC's AG format into RTTM format and generates the rt04f scoring uem

Scoring Software

md-eval, v17 - scores the system's MDE performance
sctk Version 2.0-Beta 4 - update required for Arabic STT scoring.

Workshop Presentations

PDF version of NIST's Workshop Presentation on MDE
PDF version of NIST's Workshop Presentation on STT

Detailed MDE Results

PDF file with Results of the NIST MDE evaluation

Evaluation Schedule

	Julian Days from Start	NIST Action	Site Action
16-Aug	0	NIST releases Current Test Set (CT) -- All languages
10-Sep	25		Sites submit CT English STT system outputs 8am EDT
10-Sep	25	NIST releases Progress Test Set (PT) to EARS contractors
13-Sep	29	NIST releases CT English STT system outputs and MDE ref transcripts
21-Sep	37	NIST releases CT English STT results
21-Sep	37		Sites submit PT STT system outputs 5pm EDT (applies only to EARS contractors)
24-Sep	39	NIST releases PT STT results
24-Sep	39		Sites submit Non-English CT STT systems outputs 8am EDT
1-Oct	46	NIST releases Non-English CT STT results
1-Oct	47		Sites submit CT MDE system outputs 8am EDT
8-Oct	53	NIST releases CT MDE results
15-Oct	60		Sites submit CT English Mothballed RT-03 system outputs 8am EDT (optional for both RT participants and EARS contractors)
19-Oct	65	NIST releases CT, Mothballed results
		NIST and all RT Evaluation Participants and EARS Contractors
26-Oct		Slides and papers for RT-04 and EARS notebook due
7-10 Nov		RT-04F Workshop
11 Nov		EARS PI Meeting

Update History

February 3, 2004: Initial page debut
April 2, 2004: Added AG to RTTM conversion script and RTTM validator script
April 16, 2004: Added MDE evaluation script
April 28, 2004: Updated with the latest MDE evaluation script, AG to RTTM conversion script, and RTTM validator script
May 19, 2004: Updated MDE evaluation script
June 10, 2004: Updated MDE evaluation script
June 17, 2004: Updated rt04f-eval-plan
June 25, 2004: Updated MDE evaluation script
July 19, 2004: Added evaluation schedule
August 30, 2004: Updated rt04f-eval-plan
August 31, 2004: Updated MDE evaluation script
September 2, 2004: Added Arabic STT Scoring Appendix
December 16, 2004: Added MDE Results, and Workshop presentation on MDE
January 10, 2005: Added Workshop presentation on STT results

[ RT Home ]

Page Created: Feb. 4, 2004r
Last Updated: December 19, 2007

Speech Group is part of IAD and ITL
NIST is an agency of the U.S. Department of Commerce