# Gaussian Splatting

Rendering novel views with Gaussian Splatting, with the mathematics behind the technique.

Canonical: https://aasmanbashyal.com.np/projects/gaussian

## Overview

3D Gaussian Splatting represents a scene with learned Gaussians for novel-view rendering. The [2023 method by Kerbl and colleagues](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) starts with calibrated images and sparse points, then optimizes the scene representation with a fast splat renderer.

[Watch video: Gaussian Splatting video 1](https://www.youtube.com/watch?v=-6xx_EPm7YA)

## Steps of Gaussian Splatting

### 1. Data Collection

Collect overlapping images of the scene. Estimate camera parameters and a sparse point cloud, typically using Structure from Motion.

### 2. Preprocess the Point Cloud

- Check the camera poses and remove unreliable points.

- Use the remaining points to initialize Gaussian centers.

### 3. Define the Gaussian Function

Each Gaussian has a center and covariance, along with learned opacity and view-dependent color. Its spatial shape is:

G(x) = exp(-½ \* (x - μ)^T \* Σ⁻¹ \* (x - μ))

- **μ** is the mean or center of the Gaussian (the location of the data point).

- **Σ** is the covariance matrix that determines the shape of the Gaussian (how spread out or narrow it is).

- **x** is the 3D point in space.

The covariance matrix controls the **width** and **orientation** of the Gaussian splat. If the covariance is small, the splat will be narrow and sharp; if it is large, the splat will be broader and softer.

### 4. Splatting Process

Project each visible 3D Gaussian onto the image plane as a 2D footprint.

### 5. Blending Overlapping Gaussians

Sort contributions by depth and use alpha compositing. With contributions ordered front to back:

C = Σ\_i c\_i α\_i ∏\_{j<i} (1 − α\_j)

Here, c\_i is color and α\_i combines the Gaussian footprint with learned opacity. Earlier contributions reduce the visibility of later ones.

### 6. Rendering the Scene

The visibility-aware rasterizer evaluates these contributions to produce the image. The original method uses splat rasterization rather than tracing a ray through every Gaussian.

### 7. Optimization

Optimize Gaussian parameters against the training images, with density control to add or remove Gaussians as needed.