PanDWM: Panoramic 4D Driving World Models

1Shenzhen University 2Peking University 3The University of Melbourne 4La Trobe University

*Equal contribution. †Project lead. ‡Corresponding author.

PanDWM: Panoramic 4D Driving World Models

PanDWM predicts about 20 seconds of panoramic driving video from three observed frames, and decodes the prediction into renderable 3D and 4D Gaussian scenes.

Panoramic video generation and novel-view rendering with PanDWM. Given panoramic history, 3D scene layouts and text, PanDWM predicts future video in a unified panoramic representation and uses a feed-forward Gaussian decoder to obtain a scene representation for novel-view rendering. The ring visualizes a generated panorama, and the bottom row shows its forward-facing region over time; the right column renders the Gaussian scene of the same time step from the reference viewpoint, 5 m forward and turned 30° left. The panorama spans 360° horizontally and approximately 33° vertically.

Abstract

Generative driving world models typically represent the surround scene as six pinhole-camera videos, whose different optical centres and small overlaps force the model to learn cross-view consistency. PanDWM instead generates a single 360° panoramic band, split into two overlapping halves so that every camera boundary lies inside one image, and conditions it on 3D boxes and HD maps stitched into the band in the same way. The band is generated causally, one window at a time, by a four-step student distilled from a block-causal video diffusion model and trained on its own generated history. A feed-forward decoder lifts each generated frame into 3D Gaussians placed along the calibrated camera rays of its pixels; 4D Gaussians add static Gaussians from neighbouring frames, aligned by ego pose, to fill regions unseen in the current frame. On the nuScenes validation set, the student matches the FID of its 50-step causal teacher (10.1 vs. 9.93) and lowers FVD-1 from 47.7 to 37.4, at 1.96 s instead of 33.7 s per frame.

Long-Horizon Panoramic Rollout

Each scene shows 19.2 s of generated future (frames 3–194 at 10 fps) after three recorded frames, with seed 42.

Recorded Ground Truth (GT)
Causal Teacher (50 Steps)
Causal Student (4 Steps)
3D Gaussian Rendering (3DGS)
4D Gaussian Rendering (4DGS)
0.0 / 19.2 s

Rows from top: recorded ground truth, the 50-step teacher, our 4-step student, and 3D and 4D Gaussian renderings decoded from the student rollout, seen from 1.5 m behind and 0.3 m above the recording rig. All rows show the same scene time.

3 selected scenes (3 held-out scenes). All 3 have the full five-video comparison.

Interactive 3D Gaussian Scene (student rollout decoded at t = 10 s; drag to orbit, right-drag to pan, scroll to zoom)

Loading 3D scene ...

Method Overview

PanDWM architecture: panoramic video generation and feed-forward Gaussian lifting
PanDWM overview. (a) Panoramic history is split into overlapping halves and encoded by a VAE. Conditioned on layouts, text and ray maps, a causal diffusion transformer predicts 16 frames from 3 history frames in four steps; its final 3 frames seed the next window. (b) A feed-forward decoder lifts frame latents into pixel-aligned Gaussians using calibrated source-camera rays. Static geometry from neighbouring frames supports 4D completion. Panoramas are displayed as cylindrical bands. All image inserts come from one development scene; latent cards, attention masks, cameras, rays and enlarged ellipsoids are schematic.
Gaussian geometry: calibrated camera-ray origins, Gaussian placement and static completion across time

Gaussian geometry and completion.

(a) Panoramic pixels retain their calibrated source-camera origins.

(b) Range and offset locate Gaussians relative to these rays (μ = oc + d · r + Δ); both halves' predictions are retained in the overlap.

(c) Ego-motion-aligned neighbouring static Gaussians fill uncovered pixels, while dynamic content comes from the current frame.

Scene elements are schematic.

Performance and Efficiency

Per-window FID and FVD comparing the four-step PanDWM student with the 50-step teacher
Per-window FID and FVD on Test-L (83 scenes, seed 42), each computed on the 83 windows of one position.
Band FVD-1 versus seconds per frame, comparing PanDWM with the causal teacher and other driving world models
Band FVD-1 against time per frame on Val-150; lower left is better.