20230154104. UNCERTAINTY-AWARE FUSION TOWARDS LARGE-SCALE NeRF simplified abstract (NEC Laboratories America, Inc.)
Contents
UNCERTAINTY-AWARE FUSION TOWARDS LARGE-SCALE NeRF
Organization Name
NEC Laboratories America, Inc.
Inventor(s)
Bingbing Zhuang of San Jose CA (US)
Samuel Schulter of New York NY (US)
Yi-Hsuan Tsai of Santa Clara CA (US)
UNCERTAINTY-AWARE FUSION TOWARDS LARGE-SCALE NeRF - A simplified explanation of the abstract
This abstract first appeared for US patent application 20230154104 titled 'UNCERTAINTY-AWARE FUSION TOWARDS LARGE-SCALE NeRF
Simplified Explanation
The patent application describes a method for achieving high-quality 3D reconstruction and novel view synthesis for large-scale scenes using multiple video image capturing devices.
- The method involves obtaining images from a video stream captured by multiple devices.
- The images are grouped into different clusters representing a large-scale 3D scene.
- Neural Radiance Field (NeRF) and Uncertainty Multilayer Perceptron (MLP) models are trained for each image cluster to generate NeRFs and uncertainty MLPs.
- Rendering loss and entropy loss are applied to the NeRFs to improve their quality.
- Uncertainty-based fusion is performed to combine the NeRFs into a fused NeRF.
- The NeRFs and uncertainty MLPs are jointly fine-tuned to enhance their performance.
- During inference, the fused NeRF is used for generating novel views of the large-scale 3D scene.
Potential applications of this technology:
- Virtual reality (VR) and augmented reality (AR) applications
- Gaming and entertainment industry
- Architectural visualization and design
- Virtual tours and simulations
- Training and education in various fields
Problems solved by this technology:
- Achieving high-fidelity 3D reconstruction and novel view synthesis for large-scale scenes
- Overcoming limitations of single-camera systems in capturing detailed and realistic 3D scenes
- Improving the quality and accuracy of virtual views generated from captured images
Benefits of this technology:
- High-quality and realistic virtual views of large-scale scenes
- Enhanced immersion and user experience in VR and AR applications
- Improved accuracy and detail in 3D reconstruction
- Efficient utilization of multiple video image capturing devices for better scene representation
Original Abstract Submitted
a method for achieving high-fidelity novel view synthesis and 3d reconstruction for large-scale scenes is presented. the method includes obtaining images from a video stream received from a plurality of video image capturing devices, grouping the images into different image clusters representing a large-scale 3d scene, training a neural radiance field (nerf) and an uncertainty multilayer perceptron (mlp) for each of the image clusters to generate a plurality of nerfs and a plurality of uncertainty mlps for the large-scale 3d scene, applying a rendering loss and an entropy loss to the plurality of nerfs, performing uncertainty-based fusion to the plurality of nerfs to define a fused nerf, and jointly fine-tuning the plurality of nerfs and the plurality of uncertainty mlps, and during inference, applying the fused nerf for novel view synthesis of the large-scale 3d scene.
