Masked scores only the pixels the render actually draws: render opacity ≥ 0.9 inside the ego-valid area. Coverage is that share of the ego-valid frame. Read every masked number next to its coverage, because a mode that draws less would otherwise score better. Exposure-matched fits one gain per colour channel per frame on the covered pixels (least squares; the means are shown) and scores again, which separates colour and exposure error from geometry error. A gain above 1 means the render is darker than the real frame. Raw and static-only are the page's earlier scores, kept for comparison.
These frames were also training frames for the reconstruction, which flatters every score on this page. There is no held-out view here; the lane-shift section is the only novel-view test.
Static-only masks out the ego vehicle and every detected track's projected 3D box, so it scores the background alone. Raw full frame also counts the cars, including the many the detector missed or the QA gate rejected, which were never placed back. The renders leave the sky black. The reconstructions do model a sky, but this pipeline drops it (see Known limits under About). Raw and static-only count those black pixels; masked does not.
Real camera vs reconstruction
Every camera at its full frame rate (render budget: 0.23 s/frame measured on the first 20 front frames, 22.0 min per mode). Boxes: cyan = composed actor (labelled with its track id), thin grey = detected by the LiDAR detector but not composed (no usable Step 1 crop, Step 2 failed, or rejected by the QA gate). Space plays/pauses, ←/→ steps one frame, F flips A/B.
Metrics per camera
Camera
Frames
Instant NuRec
Cov
PSNR m
SSIM m
LPIPS m
PSNR m+e
LPIPS m+e
PSNR raw
SSIM raw
LPIPS raw
PSNR s
Gain r/g/b
Front wide 120°
599
0.418
19.40
0.544
0.218
19.70
0.230
12.52
0.365
0.592
12.52
0.96/0.95/0.93
Cross left 120°
599
0.365
17.88
0.440
0.248
18.39
0.234
10.74
0.295
0.705
10.61
1.06/1.01/0.90
Cross right 120°
599
0.380
19.52
0.508
0.236
19.93
0.237
12.87
0.321
0.683
12.88
0.93/0.94/0.91
Rear left 70°
599
0.267
18.09
0.594
0.202
19.57
0.187
9.25
0.224
0.832
9.11
1.08/1.06/0.99
Rear right 70°
599
0.337
17.13
0.539
0.308
19.08
0.299
9.85
0.219
0.832
9.76
1.00/1.01/0.98
Front tele 30°
599
0.867
18.91
0.487
0.606
19.79
0.584
15.98
0.461
0.684
17.02
1.08/1.10/1.13
Cov = coverage; "m" = masked (covered pixels only); "m+e" = masked + exposure-matched; "s" = static-only (ego vehicle and every detected box masked); raw = full frame. PSNR is over the scored pixels. SSIM is the mean of the SSIM map over them. LPIPS (AlexNet) is computed after filling unscored pixels of the render with the real ones. Gain is the mean per-channel exposure gain. 6 cameras, 960 px wide.
Alignment check (before any metric)
Each render uses the camera's own calibration and its rolling-shutter start/end pose at that frame. Orange = edges in the real frame, cyan = edges in the render, white = both. The shift is the sub-pixel phase-correlation offset between the two frames (ego vehicle masked).
Instant NuRec
t=4.868 s · shift (-0.42, +0.18) px · 50% of real edges have a render edge within 2 pxt=9.868 s · shift (+0.56, -0.06) px · 49% of real edges have a render edge within 2 pxt=14.868 s · shift (+0.04, +0.09) px · 44% of real edges have a render edge within 2 px
Lane shift: front camera, ego moved left
The same drive with the camera moved 1.5 m and 3.0 m to the left (rig +y). There is no real footage for these views, since the car never drove there. This is the novel-view test: the scene has to hold up away from the path it was trained on. Instant NuRec: a point 15 m ahead moved from u=483.3 px to u=535.6 px for 1.5 m left — the expected direction.
Removed-actor check
For each mode the scene was composed again without the actor that covers the most of the front camera (into a scratch directory; the real scene is untouched), and rendered at the frames where that car was closest. If the background behind it was reconstructed, the right image shows road; if not, a hole or a smear.
Hole fraction: share of the victim's projected-box pixels whose removed-render opacity < 0.9. Smear ratio: mean Sobel gradient magnitude (grey 0-255) inside the box / the same over a ring 6-24 px outside it, covered pixels only; ~1 = texture continuous with the surroundings, << 1 = smooth smear.
Instant NuRec — removed ext_0026 (automobile)
t (s)
Box px
Hole fraction
Gradient in box
Gradient in ring
Smear ratio
4.601
6,274
0.028
55.5
54.8
1.01
4.968
16,901
0.369
48.8
61.4
0.80
5.334
56,115
0.660
51.2
72.0
0.71
Real · t=4.601 sInstant NuRec, all actorsInstant NuRec, without ext_0026 (6,274 px box)
Real · t=4.968 sInstant NuRec, all actorsInstant NuRec, without ext_0026 (16,901 px box)
Real · t=5.334 sInstant NuRec, all actorsInstant NuRec, without ext_0026 (56,115 px box)
Actors (18 harvested tracks, Step 2 seed 0)
Left: real Step 1 crops the diffusion model saw. Right: the generated asset, fitted to the LiDAR cuboid, from the front, three-quarter, side and rear. "Front" is where the pipeline puts the car's front: the asset's −x end, unless the LiDAR box was reversed at the moment the asset was built (then +x). This was checked against the real cameras.
Instant NuRec: 15 of 18 harvested actors passed the 1.8× anisotropy gate; rejected ext_0019 (2.36×), ext_0121 (4.29×), ext_0221 (3.99×). Validator: PASS (UsdValidation (28 validators), 0 issues, 15 actor prims). Scene 597.9 MB.
Variations of this same reconstructed clip, made by the pipeline's variation stages (08-12): weather and light (appearance, applied to the rendered pixels; labels unchanged) and a pothole (geometry: carved into the ground mesh and the splats, driven over by a physics vehicle whose body motion moves the camera). Every item names its recipe, engine and seed; the run record is the scene manifest.
Weather and light (Instant NuRec)
rain_heavy: recipe rain_heavy, engine procedural, seed 23. Left: the original render; right: the variant. Labels hash-identical: yes. Flicker excess (deterministic part, /255): 0.339.dust_haze: recipe dust_haze, engine procedural, seed 31. Left: the original render; right: the variant. Labels hash-identical: yes. Flicker excess (deterministic part, /255): 0.05.dusk: recipe dusk, engine procedural, seed 41. Left: the original render; right: the variant. Labels hash-identical: yes. Flicker excess (deterministic part, /255): 0.023.night_headlights: recipe night_headlights, engine procedural, seed 53. Left: the original render; right: the variant. Labels hash-identical: yes. Flicker excess (deterministic part, /255): 0.377.
Pothole on the dashcam (pothole_left_track, Instant NuRec)
Left: base scene at the recorded camera pose. Right: pothole_left_track, camera pose = recorded pose + the physics body motion (recorded x physics delta (10)); renderer 3dgrut.Physics (mujoco 3.3.7): body height, pitch and roll of the pothole run minus the flat run, vertical acceleration and wheel loads; shaded = wheels inside the pothole.
Dip -3.5 cm at 12.49 s, rebound 5.0 cm, peak |Δpitch| 0.99°, |Δroll| 1.25°, peak vertical acceleration 6.2 m/s² (flat run 1.6); flat run ride-height std 4.8 mm, pitch std 0.16° (the ground mesh is not bumpy on its own).
Edit pothole_0: recipe pothole_left_track, seed 1023, 1.2 × 0.9 m, 0.12 m deep. Mesh vs splat projected-footprint IoU 0.979; pixels changed outside footprint + margin 0.000177; floaters 0.
Approach: base | edited, recorded poses (yellow: mesh footprint)Close-up at ~5 m: base | edited
Isaac Sim view (pothole_left_track)
Isaac Sim PhysX (isaac), chase camera: a programmatic vehicle (box chassis, four sprung wheels) driven along the recorded path over the carved ground collider (invisible) in the pothole-edited scene. Splats render hazier in Isaac than in 3DGRUT.Isaac Sim PhysX (isaac), roadside camera: a programmatic vehicle (box chassis, four sprung wheels) driven along the recorded path over the carved ground collider (invisible) in the pothole-edited scene. Splats render hazier in Isaac than in 3DGRUT.
Cross-check
MuJoCo
Isaac PhysX
dip_time_s
12.492
12.477
dip_dz_m
-0.0352
-0.0269
peak_abs_dpitch_deg
0.994
0.699
peak_abs_droll_deg
1.247
0.663
accel_peak_abs_ms2
6.197
7.495
Same input to both (ground + edits + vehicle + recorded path), same output format. Reported, not gated: the Isaac tyre is the PhysX sphere contact itself (no separate tyre spring).
pothole_left_track+rain_heavy: geometry pothole_left_track + appearance rain_heavy (procedural, seed 23). Left: the pothole render with physics poses; right: the same with rain. Labels (base + the pothole) hash-identical: yes.
Ground mesh (08, from the accumulated LiDAR, actors removed): 1,225,920 triangles at 0.1 m; agreement with the splat road median |Δz| 0.0274 m, p90 0.0532 m; corridor coverage 0.9726.
Run record: variations_instant/scene_manifest.json.
Explore it yourself
File (not hosted; shared on request)
MB
What
instant/scene_instant.usdz
597.9
Composed scene (instant background + actors), OpenUSD, for Isaac Sim
instant/static_scene_instant.ply
147.3
Static background only (instant), 3DGS PLY, SH degree 0
instant/combined_busiest_instant.ply
169.7
Static + actors at the busiest instant (instant), 3DGS PLY, for SuperSplat
Isaac Sim opens the usdz and plays the timeline, but renders these splats with a known milky haze compared to 3DGRUT. 3DGRUT is the canonical renderer for this pipeline; every image on this page is a 3DGRUT render.
SuperSplat (superspl.at/editor) opens the PLYs in a browser. It could not be tested headless here; instead the combined PLY was read back with plyfile. Instant NuRec: 2,495,534 Gaussians at t=0.0 s with 6 actors; standard 3DGS properties present: yes (checked with plyfile).
About
Input: one clip from NVIDIA's PhysicalAI Autonomous Vehicles dataset, with 7 cameras and a roof LiDAR, about 20 s at an intersection. These are not Indian roads and not EdgeRecon's own rig.
Cameras on this page: 6 of the clip's 7. The one left out is the rear telephoto (camera_rear_tele_30fov): full NuRec's six-camera recipe (car2sim_6cam) does not train on it, and every review uses the same cameras so that instant and full stay comparable. Instant NuRec ran its default pa-front model, which sees only the front wide camera, so the five other cameras are views Instant NuRec never saw.
02 convert: stream the clip from Hugging Face into NCore V4.
01 cuboids: VoxelNeXt (nuScenes weights) detects vehicles in the LiDAR, a tracker links them, and the tracks are injected into the clip as EXTERNAL cuboids.
03 background: Instant NuRec (feed-forward, about a minute), or full NuRec training (40,000 steps in NVIDIA's nre-ga container) with the background's Fourier colour collapsed to its constant term. The tracked cars are masked out of both.
04 harvest: Asset Harvester segments each tracked car in a few camera crops and generates a 3D Gaussian asset with a multiview diffusion model (Step 2 seed 0).
05 compose: an anisotropy QA gate, then each asset is fitted to its cuboid and animated along its LiDAR track in one OpenUSD scene.
06 validate: OpenUSD compliance plus a dynamic-actor count check.
08 ground mesh: a road-surface mesh from the accumulated LiDAR (actors removed), checked against the reconstructed road.
09 scene edits: geometry edits such as a pothole, carved into the ground mesh and the splats from one shape.
10 physics: a simulated vehicle (MuJoCo) drives the recorded path over the edited ground; its body motion moves the camera. Cross-checked in Isaac Sim PhysX.
11 variant renders: 3DGRUT renders of each variant along the recorded or physics-moved camera path.
12 appearance: procedural, seeded weather and light (rain, dust, dusk, night) applied to the rendered pixels; labels unchanged.
07 review: this page.
Measured vs generated. The background is reconstructed from this clip's own camera and LiDAR data. The vehicles are not reconstructed: Asset Harvester's diffusion model generates them from a few camera crops, then they are placed on their LiDAR-tracked trajectories. A generated car can look plausible and still not match the real one.
Known limits. The detector misses many vehicles (grey and unboxed cars in the overlay). Without a fixed Step 2 seed the actor count varies between runs (15–17 seen); this run used Step 2 seed 0. Both reconstructions model the sky, but this pipeline drops it. Full NuRec trains a sky cubemap (nre-ga's sky-env-map) that its PLY export does not write. Instant NuRec writes its sky as a cubemap file next to its PLY, which the compose stage does not read. Isaac Sim renders the splats with a haze. Each actor is visible only inside its observed time window: the USD authors its visibility as invisible before its first and after its last LiDAR observation, and these renders read that.
live Raw clip → NCore conversion · aux masking/depth · Instant NuRec · full NuRec training · Asset Harvester · USD composition + validation · scene variations: weather and light, LiDAR ground mesh, pothole with vehicle physics roadmap EdgeRecon's own rig and Indian-road capture · generative variations (Cosmos Transfer) · NuRec's native dynamic-actor layer (not functional in the public release) gap close-range LiDAR cuboid fitting in dense traffic (diagnosed, not solved)
LIVE: has run end to end in this pipeline. ROADMAP: planned, not built yet. GAP: a known problem, not solved yet.
This run's GPU steps (from the watchdog logs)
Step
Wall (m:s)
Card peak MiB
Runs
01_detect_all_vehicles_nuscenes
0:25
1,248
1
02_ncore_convert
2:46
571
1
03_reconstruction_instant
2:11
11,981
1
04_harvest_step1_ncore_parser
—
5,778
2
04_harvest_step2
—
19,418
2
07_review_render_instant
18:08
3,634
1
09_scene_edits
—
1,480
4
10_physics
—
4,358
4
11_variant_render
—
2,714
5
b23_orientation
0:26
2,762
1
b23_part_a_gate
0:25
1,448
1
b23_part_c_overlay
0:45
1,448
1
b23_sky_850f6b5
1:00
11,981
1
render
0:40
610
1
render_fix
0:30
610
1
render_fix2
0:35
874
1
render_t16
0:33
1,288
1
"—": the docker steps' watchdog is stopped before it writes its peak line, and a re-run appends to the same log, so no single wall time or peak can be read from it.
