Neuro Radiance Fields (NeRFs) - Lecture Notes

Introduction

• Speaker: Torsten
• Topic: Introduction to Neuro Radiance Fields (NeRFs)
• Background: Computer Vision, 3D reconstructions from images

Key Concepts

Explicit Scene Representations

• Point Clouds & 3D Meshes: Traditionally used in multi-view stereo methods
• Issues: Memory intensive, assume Lambertian surfaces (color consistency from various viewpoints)

Implicit Scene Representations

• Signed Distance Functions (SDF): Used for 3D reconstructions, represent surfaces with zero-crossing isocontours
• Voxel Grids: Use SDF stored in a voxelized manner, memory intensive

NeRFs Introduction

• Input: Set of images with known camera calibrations and poses
• Output: Faithful 3D reconstruction with implicit representation
• Advantages: Models complex lighting effects, compact representation (~1MB for a room)

Volume Rendering

• Concept: Compute color for each pixel by shooting a ray into the scene and sampling along it
• Steps:
  1. Evaluate volume at sampled points
  2. Accumulate colors and volume densities
• Result: Linear combination of colors based on volume density and visibility

NeRFs Detailed

• Neural Network Input: Five parameters (3D position, 2 viewing angles)
• Output: Color (RGB) and volume density
• Training:
  • Images and known poses
  • Volume rendering to match predicted color to actual image color
• Sample Generation: Randomized sampling within intervals to approximate surfaces
• Position Encoding: Higher dimensional representation of coordinates improves fine detail recovery

Results

• Synthetic Data: High detail and view-dependent effects
• Real Scenes: Effective in handling complex real-world scenes

Issues and Solutions

• Speed: Original NeRFs are slow (days for training, minutes for rendering)
• PlenOctrees: Spherical harmonics for fast training and rendering
• Multi-resolution Hash Encoding: Efficient training with hash tables for feature storage

Geometric Extraction from NeRF

• Direct Extraction Issues: Noise in reconstructed geometry
• Signed Distance Functions: Better definition of surface geometry
• WSDF: Volume rendering using signed distance function for improved geometry

Challenges and Alternatives

• Sparse Data Training: Difficult with limited viewpoint coverage
• Monocular Cues: Using depth and normal predictions to aid training
• Transients and Appearance Vectors: Training NeRF with varying illumination and transient objects

Scalability

• Large Scenes: Training multiple NeRFs for large scenes (e.g., downtown San Francisco)
• Run-Time and Memory: Trade-offs in run-time speed and memory usage

Tools and Software

• Nvidia's Instant NGP: Efficient implementation of NeRFs
• Nerf Studio: Tool for training and visualizing NeRFs

Conclusion

• Current State: NeRFs are promising but still in infancy
• Future Directions: Reduce training times, improve scalability, potential for real-time mobile applications
• Resources: Blog posts by Frank Dard, survey papers