NATURAL LANGUAGE INTERACTIONS WITH INTERACTIVE VISUAL CONTENT

Organization Name

amazon technologies, inc.

Inventor(s)

Amitabh Saikia of Mountain View CA US

Devesh Mohan Pandey of Redwood City CA US

Tagyoung Chung of Sunnyvale CA US

Shanchan Wu of Foster City CA US

Chien-Wei Lin of Sunnyvale CA US

Govindarajan Sundaram Thattai of Fremont CA US

Aishwarya Naresh Reganti of San Jose CA US

Arindam Mandal of Redwood City CA US

Prakash Krishnan of Santa Clara CA US

Raefer Christopher Gabriel of San Jose CA US

Meyyappan Sundaram of Bellevue WA US

NATURAL LANGUAGE INTERACTIONS WITH INTERACTIVE VISUAL CONTENT

This abstract first appeared for US patent application 20250149028 titled 'NATURAL LANGUAGE INTERACTIONS WITH INTERACTIVE VISUAL CONTENT

Original Abstract Submitted

techniques for facilitating natural language interactions with visual interactive content are described. during a build time, a system analyzes various websites and applications relating to a particular user goal to understand website and application navigation and information relating to the user goal. the learned information is used to store configuration data. during runtime, when a user request performance of an action, the system engages in a dialog with the user to complete the user's goal. the system uses the stored configuration data to determine actions to be performed at a website or application to complete the user's goal, and determines system responses to present to the user to facilitate completion of the goal. such system responses may request information from the user, may inform the user of information displayed at the website or application, etc.