Machine-readable content: Complete portfolio Markdown · Content index · This page as Markdown.

Gaussian Splatting

Rendering novel views with Gaussian Splatting, with the mathematics behind the technique.

Read Markdown

Overview

3D Gaussian Splatting represents a scene with learned Gaussians for novel-view rendering. The 2023 method by Kerbl and colleagues starts with calibrated images and sparse points, then optimizes the scene representation with a fast splat renderer.

Gaussian Splatting video 1

Steps of Gaussian Splatting

1. Data Collection

Collect overlapping images of the scene. Estimate camera parameters and a sparse point cloud, typically using Structure from Motion.

2. Preprocess the Point Cloud

  • Check the camera poses and remove unreliable points.
  • Use the remaining points to initialize Gaussian centers.

3. Define the Gaussian Function

Each Gaussian has a center and covariance, along with learned opacity and view-dependent color. Its spatial shape is:

G(x) = exp(-½ * (x - μ)^T * Σ⁻¹ * (x - μ))
  • μ is the mean or center of the Gaussian (the location of the data point).
  • Σ is the covariance matrix that determines the shape of the Gaussian (how spread out or narrow it is).
  • x is the 3D point in space.

The covariance matrix controls the width and orientation of the Gaussian splat. If the covariance is small, the splat will be narrow and sharp; if it is large, the splat will be broader and softer.

4. Splatting Process

Project each visible 3D Gaussian onto the image plane as a 2D footprint.

5. Blending Overlapping Gaussians

Sort contributions by depth and use alpha compositing. With contributions ordered front to back:

C = Σ_i c_i α_i ∏_{j<i} (1 − α_j)

Here, c_i is color and α_i combines the Gaussian footprint with learned opacity. Earlier contributions reduce the visibility of later ones.

6. Rendering the Scene

The visibility-aware rasterizer evaluates these contributions to produce the image. The original method uses splat rasterization rather than tracing a ray through every Gaussian.

7. Optimization

Optimize Gaussian parameters against the training images, with density control to add or remove Gaussians as needed.