HybridSplat: Fast Reflection-baked Gaussian Tracing using Hybrid Splatting

Abstract

Rendering complex reflection of real-world scenes using 3D Gaussian splatting has been a quite promising solution for photorealistic novel view synthesis, but still faces bottlenecks especially in rendering speed and memory storage. This paper proposes a new Hybrid Splatting(HybridSplat) mechanism for Gaussian primitives. Our key idea is a new reflection-baked Gaussian tracing, which bakes the view-dependent reflection within each Gaussian primitive while rendering the reflection using tile-based Gaussian splatting. Then we integrate the reflective Gaussian primitives with base Gaussian primitives using a unified hybrid splatting framework for high-fidelity scene reconstruction. Moreover, we further introduce a pipeline-level acceleration for the hybrid splatting, and reflection-sensitive Gaussian pruning to reduce the model size, thus achieving much faster rendering speed and lower memory storage while preserving the reflection rendering quality. By extensive evaluation, our HybridSplat accelerates about 7x rendering speed across complex reflective scenes from Ref-NeRF, NeRF-Casting with 4x fewer Gaussian primitives than similar ray-tracing based Gaussian splatting baselines, serving as a new state-of-the-art method especially for complex reflective scenes.

Methodology

Figure 3: Pipeline of HybridSplat. We first reconstruct complex reflective scenes with base Gaussians \( G_{base} \) and reflective Gaussians \( G_{ref} \) , where the latter bake the reflection using a novel reflection-baked Gaussian tracing (left). Then these Gaussians are efficiently rendered using our hybrid splatting (middle), with specific pipeline acceleration at the precossing (more precise AABB-tiles) and rendering (more efficient execution flow) stages, for the final image rendering (right).

Comparative Results

Qualitative comparison on Ref-NeRF and NeRF-Casting Shiny Scenes datasets.

Comparison on datasets with complex reflections 1

Comparison on datasets with complex reflections 2

Evaluation on grinder scene of NeRF-Casting dataset

Evaluation on toaster scene of NeRF-Casting dataset

On datasets with complex reflections, our method achieves much faster rendering speed and few Gaussian primitives than EnvGS, with high image quality, better than other 3DGS baselines.

Qualitative comparison on Mip NeRF 360, Tanks & Temples and Deepblending datasets.

Comparison on datasets without complex reflections 1

Comparison on datasets without complex reflections 2

Evaluation on bonsai scene of Mip NeRF 360 dataset

On datasets without complex reflections, our method achieves much faster rendering speed and few Gaussian primitives than EnvGS, with competitive image quality with other 3DGS baselines.

Quantitative comparison on Ref-NeRF and NeRF-Casting Shiny Scenes datasets.

Methods	Ref-NeRF Real Scenes			NeRF-Casting Shiny Scenes
Methods	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	FPS↑	speedup	GS number↓
3DGS	23.94	0.641	0.265	28.05	0.847	0.191	227.78	1x	3033253
Speedy-Splat	24.30	0.658	0.324	27.94	0.832	0.238	2305.03	10.12x	234714
MetaSapiens	24.28	0.629	0.330	28.43	0.845	0.205	1358.98	5.97x	212662
FlashGS	22.93	0.535	0.317	27.10	0.818	0.240	714.97	3.14x	3033253
PUP 3D-GS	23.78	0.644	0.289	28.33	0.858	0.180	653.56	2.87x	606650
EnvGS	24.55	0.666	0.282	30.21	0.872	0.158	15.05	1x	1408937
Ref-Gaussian	24.61	0.685	0.252	30.17	0.873	0.161	65.77	4.37x	592099
Ours	24.40	0.664	0.280	29.87	0.864	0.175	106.98	7.11x	385722

Our method realizes high-quality rendering, while renders the fastest among tracing methods. More results can be found in our paper.