18626959. SYSTEMS AND METHODS FOR GESTURE GENERATION FROM TEXT AND NON-SPEECH simplified abstract (Datum Point Labs Inc.)
Contents
SYSTEMS AND METHODS FOR GESTURE GENERATION FROM TEXT AND NON-SPEECH
Organization Name
Inventor(s)
SYSTEMS AND METHODS FOR GESTURE GENERATION FROM TEXT AND NON-SPEECH - A simplified explanation of the abstract
This abstract first appeared for US patent application 18626959 titled 'SYSTEMS AND METHODS FOR GESTURE GENERATION FROM TEXT AND NON-SPEECH
The abstract describes a system and method for generating gestures from multimodal input, involving masking input, generating embeddings, extracting features, generating output, computing loss, and updating parameters.
- System and method for gesture generation from multimodal input
- Involves masking input, generating embeddings, extracting features, generating output, computing loss, and updating parameters
- Utilizes an embedder, encoder with attention layers, and generator for processing multimodal data
- Loss is computed based on input and output, guiding parameter updates for the encoder
Potential Applications: - Human-computer interaction - Virtual reality and augmented reality systems - Sign language recognition and translation - Gesture-based control interfaces - Multimodal communication platforms
Problems Solved: - Enhances the interpretation of complex multimodal input - Improves the generation of gestures from diverse input sources - Facilitates seamless communication between humans and machines
Benefits: - Enhanced user experience in interactive systems - Improved accuracy in gesture recognition and generation - Increased efficiency in multimodal data processing - Enables natural and intuitive communication interfaces
Commercial Applications: Title: Multimodal Gesture Generation System This technology can be applied in various industries such as: - Gaming and entertainment - Education and training - Healthcare and telemedicine - Automotive and robotics - Communication and collaboration tools
Questions about Multimodal Gesture Generation: 1. How does this technology improve human-computer interaction? 2. What are the potential challenges in implementing this system in real-world applications?
Frequently Updated Research: Stay updated on advancements in multimodal data processing, gesture recognition, and human-machine interaction for insights into the latest developments in the field.
Original Abstract Submitted
Embodiments described herein provide systems and methods for gesture generation from multimodal input. A method includes receiving a multimodal input. The method may further include masking a subset of the multimodal input; generating, via an embedder, a multimodal embedding based on the masked multimodal input; generating, via an encoder, multimodal features based on the multimodal embedding, wherein the encoder includes one or more attention layers connecting different modalities; generating, via a generator, multimodal output based on the multimodal features; computing a loss based on the multimodal input and the multimodal output. The method may further include updating parameters of the encoder based on the loss.
