Evaluation Software

Current Version:

• mteval-v11b.pl
  • Release date: May 20 2005.
  • Digits join was removed from the text normalization process.
• splitUTF8Characters.c
  • Release date: November 23 2007.
  • Version 1.1: C program that will enclose any non-ASCII (UTF8-encoded) character between two spaces
• splitUTF8Characters.pl
  • Release date: November 23 2007.
  • Version 1.0: Perl script that will enclose any non-ASCII (UTF-8 encoded) character between two spaces

History:

• mteval-v11a.pl
  • Release date: June 15 2004.
  • Version 11 fixes a bug with joining numbers and makes the filehandle operate in binary mode.
  • Version 11a fixes a bug in the reporting of individual n-gram scores for the BLEU metric.
• mteval-v10.pl
  • Release date: November 19 2003.
  • Version 10 has been updated to include more detailed score reports. By default both NIST and BLEU scores are reported. Use "-b" for just BLEU scores, "-n" for just NIST scores.
    
  • New reports include:
    • Cumulative scores for n-grams 1 to 9.
    • Individual scores for n-grams 1 to 9.
  • The MT DTD did not change.
• mteval-v09c.pl
  • Release date: December 3 2002.
• Bug Fix: During the calculation of ngram information, each ngram was being counted only once for each segment. This has been fixed so that each ngram is counted correctly in each segment.
• Download the MT DTD version 1.1.
• The code for version 9 of mteval is based on findings in the paper: Automatic Evaluation of Machine Translation Quality using N-ram Co-Occurrence Statistics.
• mteval-v09.pl
  • Release date: April 26 2002.
    Important Modifications: Modified to accommodate SGML tag and attribute names revised to conform to default SGML conventions.
  • Download the MT DTD version 1.1.
  • The code for version 9 of mteval is based on findings in the paper: Automatic Evaluation of Machine Translation Quality using N-ram Co-Occurrence Statistics.
• mteval-v08.pl
  • Release date: April 16 2002.
    Modified the NIST metric in accordance with the findings on the 2001 Chinese-English dry run corpus. Also incorporates the BLEU metric as an option and supports the output of ngram detail. View the mteval-08-performance summary (pdf).