042026
Synthetic Data Pipeline for Grasp Detection
Labelling grasp data by hand does not scale, and no public dataset shows wheelchair push handles from a humanoid's point of view. So the data gets generated instead — randomized scenes in Isaac Sim, rendered and annotated straight into a format a detector can train on.
- Where
- Intelligent Systems & Robotics / Micro Manufacturing (ISRMM) Laboratory, TMU
- Stack
- Isaac Sim, Omniverse Replicator, YOLOv8, Python
- 2,000+images generated
- 2annotated classes
- 0.887 uhandle separation
Randomising the scene
Omniverse Replicator inside Isaac Sim does the generating. It randomizes object pose, surface materials, lighting through dome-light HDRI environments and camera placement, shooting from top-down and perspective angles both.
A custom writer dumps annotations straight into YOLOv8 format across two classes, handle and wheelchair, so a generated set is ready to train against with no intermediate conversion and no manual pass. Two thousand plus images so far.
Camera placement out of the target's geometry
Arm's length is the range the robot actually grasps from, and at that distance the handles fill a good chunk of the wrist camera's view. So the close-range half of the pipeline is built around that one viewpoint, pulling camera placement out of the target's geometry instead of eyeballing it.
The wheelchair USD is read directly for the real numbers: two handle prims, 0.185 units apiece, roughly 0.887 units apart. Those figures anchor every close-range camera, not the object's overall bounding box. Cameras aim at the midpoint between the handles — the spot the robot goes for — so handles land centred and full-size in every frame that comes out.
Standoff distance scales off the target too. Camera distances are multiples of that 0.887-unit separation, which turns handle size in the training images into something set deliberately rather than something that just happens. Viewpoints get sampled from a horizontal wedge on the handle side of the chair, matching the way the robot actually walks up to it.
Checking the data before spending an epoch on it
The pipeline is instrumented to report bounding-box scale statistics across the whole generated set. That makes it possible to check training data against the box sizes the deployment camera actually produces before spending a single epoch on it.
Material randomization got wider along the way: a plain white and grey pass alongside the wood and coloured-handle pass already there, closer to the finishes real chairs come in.
What it feeds
The trained detector is exported to ONNX and runs under ONNX Runtime inside a C++ ROS 2 node, with Python kept out of the runtime path entirely.
On the wrist-camera frame the robot deploys from, it picks up both grips at roughly 0.81 confidence apiece and the wheelchair body at 0.85, each with a metric depth attached — two real grasp targets in the robot's own frame, handed straight to the planner.
