Perception/ML lead · Apr 2026 → now
Soil-moisture perception for an unmanned ground vehicle
A camera and lidar system that classifies the soil under a moving vehicle as dry, semi-wet or saturated. It has to run on a 6-watt onboard computer, so a lot of the work is fitting inside that budget.
- PyTorch
- Blender / Cycles
- OpenCV
- ONNX → OpenVINO
- ROS 2
- Ouster lidar
System design & embedded constraints
- Fuses RGB, near-infrared and a 64-beam Ouster OS1 lidar, budgeted for a 6 W x86 SBC decoding two 5 MP streams.
- Derived imaging geometry from first principles (GSD, motion blur, per-frame observation count), correcting a 2–5× overstatement of the temporal-voting budget and setting camera mode (5 MP @ 30 fps over VGA @ 112 fps).
- Built the ONNX → OpenVINO export path and an fp32 / fp16 / int8 latency harness.
Synthetic data & sim-to-real
- Built a ~13k-line, 201-test Blender/Cycles renderer producing 10,000+ labeled frames per batch with per-pixel moisture, depth, segmentation and simulated lidar, calibrated to LUCAS soil spectra.
- Label-blind domain randomization: photoscanned soil textures, a 336-HDRI sky bank with sun extraction, dusk/dawn and overcast regimes, canopy shadows and vehicle self-shadowing.
- Measured lighting randomization swamping the class signal (3.55× vs. 1.7× in brightness); fixed it with irradiance-tracked exposure and a regression validator; verified exposure linearity to 0.01%.
- Diagnosed a domain shift between dataset generations with PCA and random-forest probes and fixed it with a white-balance-relative illuminant model.
Experiments & training
- A controlled A/B pilot (1,200 synthetic images, one variable) showed a brightness shortcut was capping the model; widening photometric jitter gained +0.12 macro-F1 and cut saturated→dry confusions from 21 to 0.
- Anchored simulation parameters in bench measurements: drying-tray soil series, lidar-return modeling, camera characterization.
- Built the PyTorch training framework (RGB / NIR / early-fusion arms; motion-blur and rolling-shutter augmentation from vehicle speed and exposure).
Status: simulation pipeline and pilot done; final model and field results still in progress.