HUMAN-IN-LOOP ROBOT TRAINING AND TESTING SYSTEM WITH GENERATIVE ARTIFICIAL INTELLIGENCE (AI)

Organization Name

Inventor(s)

Patrick McKinley Jarvis of Redmond WA (US)

HUMAN-IN-LOOP ROBOT TRAINING AND TESTING SYSTEM WITH GENERATIVE ARTIFICIAL INTELLIGENCE (AI)

This abstract first appeared for US patent application 20240408757 titled 'HUMAN-IN-LOOP ROBOT TRAINING AND TESTING SYSTEM WITH GENERATIVE ARTIFICIAL INTELLIGENCE (AI)

Original Abstract Submitted

a robot teaching and testing system and method that performs human-operated robot tasks according to instructions generated from generative ai models. the process starts with a user prompt and combines the user prompt with predefined prompt templates to generate well-formatted text prompts. generative ai models take the text prompts and convert them into high-level instructions or control codes that can be deployed on a robot. the high-level instructions are then converted into human-operated robot tasks for a human data collector using a mixed reality (mr) device. the human data collector will attempt to follow the instructions to complete the human-operated robot tasks and may overwrite the suggested instructions by performing a different action, demonstrate a task without instructions, or leave feedback or comments regarding the tasks. feedback data will be captured and saved for improving the robot system.