NIST Metrics for Machine Translation Challenge
(MetricsMATR)
Introduction
NIST is pleased to introduce the MetricsMATR Challenge, a new series of research challenge events
for machine translation (MT)
metrology promoting the development of innovative, even revolutionary, MT metrics. MetricsMATR
focuses entirely on MT metrics.
NIST provides the evaluation infrastructure, the source files being MT system output
The participants develop MT metrics to assess the quality of the source files
The goal is to create intuitively interpretable automatic metrics which correlate highly with human assessment of MT quality.
Motivation and Goals
There are several drawbacks to the current methods employed for the evaluation of MT technology:
- Automatic metrics have not yet been proved able to predict the usefulness and reliability
of MT technologies with respect to real applications with confidence.
- Automatic metrics have not demonstrated that they are meaningful in target languages
other than English.
- Human assessments are expensive, slow, subjective, and difficult to standardize.
These problems, and the need to overcome them through the development of improved automatic
(or even semi-automatic) metrics, have been a constant point of discussion at past NIST MT
evaluation events.
MetricsMATR aims to provide a platform to address these shortcomings. Specifically, the goals of MetricsMATR are:
- To inform other MT technology evaluation campaigns and conferences with regard to improved metrology.
- To establish an infrastructure that encourages the development of innovative metrics.
- To build a diverse community which will bring new perspectives to MT metrology research.
- To provide a forum for MT metrology discussion and for establishing future directions of MT metrology.
Target Audience
The MetricsMATR challenge is designed appeal to a wide and varied audience including
researchers of MT technology and metrology, acquisition programs such as SEQUOYAH,
and commercial vendors. We welcome submissions from a wide range of disciplines including
computer science, statistics, mathematics, linguistics, and psychology.
NIST encourages submissions from participants not currently active in the field of MT.
Current and Recent NIST MetricsMATR Activities
The first NIST MetricsMATR Challenge is MetricsMATR08 and concludes with a workshop at
AMTA 2008.
MetricsMATR will be held at regular intervals.
You can subscribe to the MetricsMATR mailing list hosted by NIST,
metricsmt_list@nist.gov,
by sending e-mail to
listproc@nist.gov.
Include "subscribe metricsmt_list" in the body.
Find more information (including registration and documentation for the current challenge) by clicking a specific year in the tabs below.
Contact mt_poc@nist.gov for more general information or to pose related questions.
[ 2008 ]
Page Created: January 29, 2008
Last Updated:
April 16, 2008
|
