Lecture Notes: Digital Twin Interface in Mixed Reality and VR

Overview

• Digital twin representation of the world
• Users explore in Mixed Reality (MR) & Virtual Reality (VR)
  • Physical movement
  • VR controller teleportation
• Integration of perception inputs
  • Depth cameras
  • Proprioception sensors & state estimation
• Third-person view of the robot

Interface Features

Camera Feed Panel

• Displays robot's point of view
• User can adjust position

Robot Operation

• VR trackers & controllers for operation
• Kinematic streaming
  • Yellow virtual ghost robot overlay
  • Represents Inverse Kinematics (IK) algorithm's solution based on current VR input

Teleoperation Assistance

• Information panel on right VR controller
  • Activates assistance with left controller button
• Prompts user to start task
  • Robot analyzes initial inputs
  • Determines trajectories for each body part
• Preview of computed motion
  • Gray ghost visualisation in spline trajectories
  • Accept/reject suggested motion

Control & Execution

• Post-validation control of robot
  • User directs movements.
• Learning Phase
  • Simulated environment demonstrations
  • ProMPs (Probabilistic Movement Primitives) learned
• During teleoperation
  • Recognizes current task
  • Updates ProMPs using initial user input and estimated object position
• Affordance blending for smooth transitions

Example Task: Punching

• User initiates part of punching motion
• Robot recognizes & completes action
  • Initial input suffices for accurate execution
• Preview feature disabled for quicker execution