From one real drive to a simulation world you can vary.
We turn a recorded drive, multi-camera video plus LiDAR, into a simulation-ready, labelled OpenUSD scene: the street as a 3D Gaussian-splat background, the traffic regenerated and driven back along its LiDAR tracks. The same drive then comes back in rain, dust, dusk or night, or with a pothole that a simulated vehicle drives over.
The pipeline is not tied to one country: it has run unchanged on US city, US freeway and European clips. The reference clip shown here is a US intersection.
Every step below ran end to end on that clip, and every number in the steps comes straight from that run's own record.
Front camera, same moment and pose: the real recording and the reconstructed scene. Masked PSNR 19.5 dB, scored on the 41% of the frame the render draws.
What you get from one drive
Each step shows real output from the reference drive and one number from its run.
Multi-camera + LiDAR footage in
A recorded drive with surround cameras and a roof LiDAR, converted to NVIDIA's NCore format. The reconstruction is rendered and scored against the real footage on 6 cameras.
A 3D Gaussian-splat background of the street
The static street, reconstructed as 3D Gaussian splats with NVIDIA's NuRec after the moving and the regenerated vehicles are masked out; parked cars stay part of the street. 1.8 million Gaussians. Shown here from a side camera; the front camera is at the top of the page.
The real vehicles, regenerated and driven back along their LiDAR tracks
These are generated look-alikes, not reconstructions. NVIDIA's Asset Harvester generates each vehicle from a few camera crops; it is sized to its LiDAR box, drives its recorded track and is visible only while it was observed. Shape and colour can differ from the real car.
real crop · generated
real crop · generated
real crop · generated
real crop · generated
real crop · generated
real crop · generatedOne OpenUSD scene that opens in Isaac Sim
The background and all 15 vehicles in one OpenUSD file. It passes OpenUSD's own validators, and it opens and plays in Isaac Sim.
Variations of the same drive: rain, dust, dusk, night
Procedural weather and light, applied to the rendered scene (rendered effects, not a generative model). The scene's labels stay byte-identical to the base scene's, and each variant must pass a flicker check before it is shown. Each video: the plain render on the left, the variant on the right.
A pothole carved into road and splats, driven over with vehicle physics
A 2.0 × 1.5 m pothole, 18 cm deep with a worn, sloped rim, is cut into the LiDAR ground mesh and into the splat road. A simulated vehicle (MuJoCo) drives over it at 28 km/h. The video is the dashcam view: the real street re-rendered with the camera moved by the vehicle's body motion (left: the plain road, right: the pothole). The pothole is visible from about 17 m. The plot shows the body's height, pitch and roll over time.
What is not right yet
Milestones
- 2026-09-18
One command from a raw drive to a validated OpenUSD scene
Every stage ran on real data in a single pipeline run.
- 2026-10-11
A fully trained background
A 40,000-step NuRec training background composed with the vehicles, and the first frame-by-frame review of renders against the real cameras.
- 2026-10-28
Vehicles checked against the real cameras
Placed vehicles face their direction of travel and appear only while they were observed. Scores count only the pixels the render draws, with that coverage shown next to them.
- 2026-10-29
Variations with physics
Rain, dust, dusk and night applied to the reconstructed drive with its labels unchanged; a ground mesh from the accumulated LiDAR that agrees with the reconstructed road to 2.7 cm (median); a pothole carved into mesh and scene and driven over by a simulated vehicle whose body motion moves the camera, also in Isaac Sim. See the run.
- 2026-10-30
Rebuilt from code on a fresh cloud GPU and reproduced the result
A new cloud GPU machine was set up from the repository alone in about 17 minutes and ran the clip: the same ground mesh, pothole and physics numbers, and front-camera masked PSNR 19.39 dB against 19.42 dB before. See the run.
- 2026-10-31
Variations on the fully trained background
The variation stages ran on the 40,000-step background too: road agreement with the LiDAR ground mesh 1.1 mm (median), pothole footprint IoU 0.935, the same physics as on the fast background. See the run.
- 2026-11-02
Parked cars stay in the street
Parked cars that are detected but not regenerated as 3D vehicles now stay in the background instead of leaving black, car-shaped holes. Missing-car time in the front view fell from 71 to 10 car-seconds on the reference clip, and the front-camera score held (masked PSNR 19.42 dB before, 19.54 dB after). See the run.
- 2026-11-02
A pothole you can feel
A 2.0 × 1.5 m, 18 cm deep pothole with a worn rim: body dip 9.4 cm, pitch 2.6°, roll 3.2° (the earlier 1.2 × 0.9 m pothole: 3.5 cm, 1.0°, 1.3°), visible on the dashcam from about 17 m. Every physics check passes, including the one that the tyre does not sink into the road edge. See the run.
Technical appendix
The full review of each clip: both reconstruction modes side by side, every camera, how every score is computed, and earlier runs. Large 3D files (OpenUSD scenes, splat PLYs) are not hosted here; they are shared on request.
US intersection, about 20 s 9fc15f8c
One clip from NVIDIA's PhysicalAI Autonomous Vehicles dataset: 7 cameras and a roof LiDAR, driving through an intersection.
Full review → Instant NuRec and Full NuRec: every camera, every metric with its coverage, every variation that passed its checks.
Earlier runs: 2026-10-30 10:27 UTC (Instant NuRec), 2026-10-29 14:40 UTC (Instant NuRec), 2026-10-29 09:41 UTC (Instant NuRec).