Intelligent Planning and Modeling for Autonomous Systems

This project will develop the measurement science and standards to enable advancements in the autonomous decision-making of robots through application to specific scenarios relevant to manufacturing. The ability to create and execute plans in real-world scenarios is what separates a machine-tool from an intelligent manufacturing system. Plans enable a robot to change its actions to deal with uncertainty in its environment and to rapidly switch to new tasks. These plans are based on models of the current environment, predictions about the future, and a priori knowledge of causal relationships between current actions and results. The breadth and usability of knowledge in these models is one of the main factors that constrains the flexibility and performance of manufacturing planning systems. Currently, however, there is no accepted standardized way to represent this knowledge, to reason with this knowledge, or to measure the performance of these systems. 

Objective:

To develop the measurement science and standards for planning and modeling by robots so that they are able to be more quickly re-tasked and are more flexible and adaptive in manufacturing applications.

What is the new technical idea?

Current manufacturing robots have little or no understanding of the world around them and no capability to dynamically change their actions if the environment changes. This restricts them to operate in highly constrained environments and makes it difficult to change from one task to another. Systems that are capable of modifying their behavior by generating and executing plans will require detailed knowledge of and a consistent representation for both the manufacturing environment and the plans. This project will develop standard data representations for maintaining and sharing knowledge of the world and how to plan actions in it. Performance measurement techniques and artifacts will be developed to enable manufacturers to use these systems and representations in a cost-effective manner. Initially, a high-fidelity physics-based simulation engine will be used to simulate plan execution in content rich worlds that include semantic labels and virtual sensors capable of detecting and reporting the entirety of the environment. Repeatable scenarios will allow for the comparison of planning systems that are capable of exploiting this rich content. Later, work in the project will migrate to real robotic hardware.

What is the research plan?

While this project presents a very ambitious research agenda, it is designed to be implemented with a crawl, walk, run strategy. During the crawl period, we will develop an understanding of the state-of-the-art of knowledge representation and planning in intelligent autonomous systems that address the general class of manufacturing problems related to component placement, for example, the construction of assembly kits (kitting) and packaging. This will transition to our walking phase where we will provide leadership to the newly formed IEEE working group on knowledge representation (KR) and develop a generic KR specifically aimed at manufacturing problems in the area of component placement (i.e. kitting and packaging). The goal of this KR is to allow for the development of planning systems capable of rapid retasking. We will also develop the simulation tools, evaluation tools, test methods, and metrics necessary to fully evaluate both the KR and planning algorithms in manufacturing environments that require rapid retasking (i.e. construction of multiple different kits at the same assembly station or packaging multiple products). Finally, once we are fully running, we will continue to work with the IEEE to promote new KR standards and will look to generalize the previous work to allow rapid retasking in general assembly tasks requiring dexterous manipulation and multi-modal sensing. The plan includes tasks in the following areas:

Knowledge Representation: Develop standard representations for world knowledge and plan knowledge, and the related performance evaluation criteria. We will design a comprehensive model that is able to represent all of the knowledge for the general class of manufacturing problems in the area of rapid retasking for general assembly tasks. We will work in cooperation with the IEEE Working Group on Knowledge Representation for Robotics and Automation in order to create a standard for knowledge representation and will provide a test implementation of the proposed standard in order to validate its usability. We will also derive methods of defining the performance requirements for the knowledge and will develop performance methods and metrics. For example, we can develop a methodology for determining the necessary resolution of knowledge, the update rates needed, and other characteristics (e.g., is color perception necessary and how do we evaluate this?). Out-years will expand on the representation and examine the inclusion of unstructured environments and agile assembly processes where semantic knowledge of world objects is critical for successful planning and execution of manufacturing tasks.

Planning: Develop techniques to compare planning algorithms that utilize the representations developed above to address next generation robotics for the class of manufacturing problems in the area of component placement. The need to accurately place components is common to a large segment of the manufacturing industry for tasks ranging from kitting to packaging. Companies such as Proctor and Gamble have expressed a need to rapidly retask packaging robots to cope with small lots of products whose characteristics (size, shape, weight, etc.) change from lot to lot. Companies such as GM have expressed the need to work with a wide variety of assembly kits. Under this task, we will examine input/output standards and performance measures for planning systems.

In order to validate the utility of these interfaces and performance measures, a simulated manufacturing test method will be developed that is capable of demonstrating rapid retasking and evaluation using our performance measures.

Simulation: In order to populate the semantic world representation, the simulation system (called USARSim) will need to be augmented and expanded to include new virtual sensing and representation capabilities. The simulation system will provide representations that are not yet possible to obtain from the real sensing systems within the early part of this project. It therefore provides a high-fidelity but more advanced testbed for developing performance requirements and validating test methods.

Recent Results:
Outputs:

1. Development of XML prototype information model for order input and plan output for the construction of mixed pallets.
2. Development of performance metric prototypes for judging the quality of mixed pallet plans and construction.
3. Development of a consortium of robot vendors, researchers, and end-users of mixed palletizing technologies.
4. Pallet viewer tool: A tool for visualizing mixed-palletizing plans based on our emerging standard formats has been developed. This tool is currently being utilized by C&S Wholesale Grocers to evaluate their planning software.

Papers:

1. F.H. Riddick, Y.T. Lee, D. Kibira, S. Balakirsky. A Component-based Approach for Manufacturing Simulation. 3-2-2011. Boston, MA. Symposium for Emerging Applications of Modeling and Simulation in Industry and Academia. 4-4-2011.
   
2. C. Schlenoff, H.B.S. Scott, S. Balakirsky, Performance Evaluation of Intelligent Systems at the National Institute of Standards and Technology (NIST), The ITEA Journal. 32(2011) 59-67.
3. S. Balakirsky, H. Christensen, Kramer.T., P. Kolhe, F. Proctor. Mixed Pallet Stacking: An Overview and Summary of the 2010 PerMIS Special Session. 9-30-2010. Proceedings of Performance Metrics for Intelligent Systems (PERMIS) Workshop.
4. S. Balakirsky, F. Proctor, T. Kramer, P. Kolhe, H. Christensen. Using Simulation to Assess The Effectiveness of Pallet Stacking Methods. 2010. Proceedings of the 2nd International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR 2010).
5. S. Balakirsky, T. Kramer, F. Proctor. Metrics for Mixed Pallet Stacking. 9-30-2010. Proceedings of Performance Metrics for Intelligent Systems (PERMIS) Workshop.
6. S. Balakirsky. The Mobile Manipulation Challenge. IEEE Robotics Automation Magazine 17[4], 10-12. 1-2-2010.
7. S. Balakirsky, D. Dimitrov. Single-query, Bi-directional, lazy Roadmap Planner Applied to Car-like Robots. 2010. IEEE International Conference on Robotics and Automation (ICRA).
8. S. Balakirsky, S. Carpin, G. Dimitoglou, B. Balaguer From Simulation to Real Robots with Predictable Results: Methods and Examples. in: R. Madhavan, E. Messina, and E. Tunstel (Eds.), Performance Evaluation and Benchmarking of Intelligent Systems, Springer, 2009.
9. S. Balakirsky, J. Falco, F. Proctor, P. Velagapudi. USARSim - Porting to Unreal Tournament 3. 2009. IROS Workshop on Robots, Games, and Research: Success stories in USARSim. 10-15-2009.
10. S. Balakirsky, R. Madhavan. Advancing manufacturing Research Through Competitions. 7332, 73320G-73320G-9. 2009. Proceedings of the SPIE Unmanned Systems Technology XI Conference. 2009.
11. S. Nunnally, S. Balakirsky. Acoustic Sensing in UT3 Based USARSim. 2009. IROS Workshop on Robots, Games, and Research: Success stories in USARSim. 10-15-2009. 

Outcomes:

1. Administration of the Virtual Manufacturing and Automation Challenge as part of the IEEE ICRA Robot Challenge series. This competition utilized our metrics and control system (MOAST) software.
2. 2010 Bronze medal award for developing open source software.
3. 2009 Government Open Source Conference Agency Award.
4. Release of Unified System for Automation and Robot Simulation (USARSim) based on the Unreal Development Kit (UDK). USARSim has developed a large user base amongst industry, researchers, and government institutions.
   

Standards and Codes: 

We have started to work with the newly formed and NIST-led IEEE Study Group on Knowledge Representation for Robotics and Automation. As part of this project’s efforts, we are planning to work with industry to formalize the study group and turn it into a formal IEEE Working Group. Once formed, this working group will produce new standards and performance measures for Knowledge Representation.