Home

Competitions

Reference Datasets

Reality Arena

Virtual Arenas/Simulation

Publications

Standards

Contact

Performance Metrics and Test Arenas
for Autonomous Mobile Robots

Reproducible and widely known challenges can help evolving fields by providing reference problems with measures of performance which allow researchers to compare implementations, communicate results, and leverage each other’s work. The National Institute of Standards and Technology (NIST) developed the Reference Test Arenas for Autonomous Mobile Robots to focus research efforts, provide direction, and accelerate the advancement of mobile robot capabilities. These arenas, modeled from buildings in various stages of collapse, allow objective performance evaluation of robots as they perform a variety of urban search and rescue (USAR) tasks in both physical and virtual domains.

NIST Reference Test Arenas

A reality arena, composed of an above and below ground structure covering approximately 300 square meters, further challenges those implementations that demonstrate successful performance in the physical arenas by forcing them to encounter a more rigorous and real-world environment.

Additional arenas are currently being designed to complete the spectrum of reference test environments. A “Confined Space Maneuver” arena is proposed that will be capable of testing shape shifting robotic implementations and other necessary robotic capabilities, along with evaluating an operator’s situational awareness. An outdoor “Scout Arena” is also in development that will challenge lightweight Unmanned Ground Vehicles (UGVs < 225 kg) by presenting an equestrian style obstacle course complete with ditches, tall grass, woods, rocks, and other outdoor obstacles (both natural and man-made). It is the intent that these test arenas will comprise a standardized reference facility that will assist in the development and evaluation of various mobile robotic systems.

Since 2000, the NIST USAR Arenas have inspired not only replicas to be fabricated, but they have provided the foundation for the annual Rescue Robot Competitions that are currently part of the RoboCup – USOpen, RoboCup– Japan Open, RoboCup, and the International Joint Conference on Artificial Intelligence (IJCAI)/American Association for Artificial Intelligence (AAAI) conferences. They have traveled throughout North America hosting the AAAI/IJCAI Robot Rescue Competitions for the past four years while the RoboCup community has constructed Japanese and Italian replicas to host their Robot Rescue Competitions for the past two years. NIST will be will be involved in the administration and/or support of the 2004 events including the RoboCupRescue - USOpen (New Orleans, USA), RoboCupRescue – Japan Open (Osaka, Japan), RoboCupRescue2004 (Lisbon, Portugal) and AAAI Rescue (San Jose, USA) Robot Competitions.

(Click on image to download the RoboCupRescue Robot League Arenas)

NIST Disclaimer:

Certain commercial equipment, instruments, or materials are identified in this site in order to adequately demonstrate the experimental procedure or capability. Such identification does not imply recommendation or endorsement by NIST, nor does it imply that the materials or equipment identified are necessarily best for the purpose.

isd-webmaster@cme.nist.gov
Last updated: February 11, 2008