<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> NIST Speech Group Website
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Group Home

  • Benchmark Tests

  • Tools

  • Test Beds

  • Publications

  • Links

  • Contacts
    •

    MADCAT Phase 2 Evaluation

    Phase 2 evaluation continues what was done for Phase 1. Three tasks identified in Phase 1 as the main components to be measured are:

    1. Document image translation (measuring the end-to-end system)
    2. Document image recognition (measuring the recognition capability)
    3. Document text translation (measuring the translation capability)

    Documentation

    • Evaluation plan - this document describes the evaluation protocol for Phase 2.
    • Data plan - this document describes the data proposed for Phase 2.
    • Development data list - the file lists the documents selected as Phase 2 development data
    • File status list - the file lists all Phase 1 eval documents and indicates which documents have been selected for the progress test set and for the control set. Note that no development and/or training is allowed on the documents selected as part of the progress set.

    Schedule

    Date
    Go/No Go Track Milestones
    Feb 9
    		 1st training data release
    May 11
    		 2nd training data and development data release
    Jul 9
    		 3rd training data release
    Oct 15
    		 Eval data arrives at NIST for final QC'ing
    Oct 28
    		 Eval data (w/o and w/ segmentation) for tasks 1 & 2 arrives at BBN
    Nov 18 at 10am
    		 System output for tasks 1 & 2 due at NIST
    Nov 18 at 3pm
    		 Eval data for task 3 arrives at BBN
    Nov 30
    		 System output for task 3 due at NIST
    Nov 30
    		 Post-editing starts
    Jan 27
    		 Post-editing completes
    Jan 29
    		 Final results to DARPA
    Dates
    		 Go/No-Go Dry Run
    Sep 11
    		 Sample test data arrives at BBN
    Sep 18
    		 Output of sample test data due to NIST
    Sep 22
    		 Post-editing kit of sample test data due to LDC
    Sep 25
    		 Post-editing results due to NIST
    Sep 28
    		 NIST reports results and/or any issues
    Dates
    		 Challenge Track Milestones
    Mar 19
    		 Gold standard eval data to NIST
    Mar 26
    		 Test set to BBN
    Apr 16
    		 System output to NIST
    Apr 20
    		 Auto scoring
    Apr 23
    		 Post-editing begins
    Jun 11
    		 Post-editing ends
    Jun 16
    		 Final results to DARPA
    Dates
    		 Challenge Scenario Dry Run
    Oct 23
    		 Gold standard eval data to NIST
    Oct 28
    		 Eval data (w/o and w/ segmentation) for tasks 1 & 2 arrives at BBN
    Nov 18 at 10am
    		 System output for tasks 1 & 2 due at NIST
    Nov 18 at 3pm
    		 Eval data for task 3 arrives at BBN
    Nov 30
    		 System output for task 3 due at NIST
    Nov 30
    		 Post-editing starts
    Jan 27
    		 Post-editing completes
    Jan 29
    		 Final results to DARPA

    [ MADCAT home ]

     

     

    Page Created: Jan 9, 2009
    Last Updated: October 8, 2009

    Multimodal Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
         		Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA