regolith

rock-IoU: intersection-over-union on the rock class (synthetic hold-out). Real false-positive: share of pixels mislabelled as rock across 21 NASA Apollo/Surveyor photographs, the same images for every build.

A rover or lander has to spot rocks and hazards on the Moon, but there's almost no labelled real lunar imagery to learn from. So I made the data. A domain-randomized synthetic Moon, authored in OpenUSD and generated with NVIDIA Omniverse Replicator, becomes the training set for a SegFormer hazard segmenter. The whole loop, from synthetic world to trained model to cinematic render, runs end-to-end on a single DGX Spark.

The obvious first move was to make the synthetic rocks photoreal. The synthetic score rose (rock-IoU 0.815 → 0.852), but real-world transfer got worse: on real Apollo photographs the model flagged most of the surface as rock, a 72.6% false-positive flood. Photoreal rocks had handed it a shortcut (rough grey means rock), and real lunar regolith is rough and grey too. The failure was a diagnosis: the fidelity was on the wrong surface.

So I moved the realism to the ground: a cratered, dark, displaced regolith floor as rough as the rocks themselves. With the texture shortcut gone, the model had to learn the cues that actually transfer: shape, shadow, and scale. It worked: on the same 21 NASA images the false-positive flood roughly halved to 35.7% (below even the crude first build) while synthetic rock-IoU climbed to a best-ever 0.887. For the first time, the synthetic and real arrows point the same way.

The series ends with a film: NASA's VIPER rover crossing the v3 boulder field in a cinematic RTX flythrough, its forward hazard-cam carrying the live overlay the segmenter produces, the model's-eye view of the terrain it learned to read.

Watch the ground: in v3 it is as cratered and rough as the rocks, so “rough grey” stops meaning “rock.”