Gaussian Splatting

Definition

Gaussian splatting is a volume rendering technique that allows direct rendering of volume data without converting it into surface or line primitives.
Originally introduced as splatting by Lee Westover in the early 1990s.

Newer methods like 3D Gaussian splatting and 3D Temporal Gaussian splatting have been developed for real-time radiance field rendering and dynamic scene rendering.

Purpose: Used in real-time radiance field rendering to create novel-view scenes from multiple photos or videos.
Representation: Scenes are represented with 3D Gaussians which retain continuous volumetric radiance fields properties.
Utilizes anisotropic representation and fast visibility-aware rendering algorithms optimized for GPU usage.

Input: Images of static scenes with camera positions expressed as a sparse point cloud.
3D Gaussians: Defined by mean, covariance matrix, and opacity.
Color Representation: Uses spherical harmonics for view-dependent appearance.
Optimization: Employs stochastic gradient descent, minimizing loss via L1 loss and D-SSIM.
Rasterizer: Tile-based rasterizer for efficient Gaussian blending.
Rendering: Fast rendering and projection into 2D splats, with explicit covariance parameterization.

Tested on 13 real scenes and synthetic data, showing high PSNR, L-PIPS, and SSIM.
Achieved quality comparable to state-of-the-art techniques like Mip-NeRF360, InstantNGP, and Plenoxels with reduced training and rendering time.
Noted limitations include artifacts and memory consumption.

Purpose: Allows real-time rendering of dynamic scenes with high resolution by incorporating a time component.
Uses HexPlane for accurate position and shape deformation representation.
Achieves real-time rendering with quality, despite some limitations with motion length captured.

Text-to-3D generation, autonomous driving simulations, 3D mesh reconstruction, SLAM, and 4D content creation.

Includes further reading on related topics like computer graphics, neural radiance fields, and volume rendering.