NA-MIC Project Weeks

Back to Projects List

Training Robot Motions Using Slicer Images

Key Investigators

• Taewoo Yoon (AIRS Inc, Republic of Korea)
• Joonho Seo (Korea Institute of Machinery Materials, Republic of Korea)

Project Description

3D Slicer can function as a tool to display real-time object movements in a 3D view. We intend to utilize this 3D view as training data for robotic motion. The 3D Slicer view will function just like a real camera mounted on the robot. Furthermore, it will allow us to build custom simulation environments to generate and train on virtual data.

Objective

• Establishing a pipeline to integrate 3D Slicer’s visual data with robot joint data for advanced policy learning and real-time inference.
• Establishing an environment dedicated to generating and training on synthetic simulation data.

Approach and Plan

1. Establishing an environment to track an object rigidly coupled with the robot in the 3D view, ensuring its pose updates dynamically in accordance with the robot’s motion.
2. Capture the 3D view for a specified number of frames while simultaneously acquiring the corresponding robot joint data.
3. Train the policy based on the training dataset created in step 2.
4. After training the policy, inferred joint positions are fed as control commands into either a simulation tool or physical robot.
5. Input the 3D Slicer view into the trained model for inference.
6. Feed the inferred joint position into either the simulation tool or physical robot, and sequentially input the updated 3D Slicer view as the next state input as the robot moves.

Progress and Next Steps

1. Describe specific steps you have actually done.

Illustrations

No response

Background and References

No response