HomeBody uses spatial targets to connect task reasoning to physical execution. The VLM selects a skill and its target from the current ego view, map context, gripper state, recalled observations and the previous result. It passes this selection through a structured tool call, leaving the skill to plan and execute the motion. The VLM therefore does not need to know the skill’s low-level implementation.

For picking, the call specifies an image point normalized to 0–1000 and which hand to use. The point prompts segmentation [3], while Fast-FoundationStereo [4] estimates depth from D435i stereo images. Camera calibration projects the masked geometry into 3D, where we predict the grasp analytically. To reach that pose, the arm planner builds a spline reference with minimum-jerk timing, solves inverse kinematics along the path and checks the swept motion for collision clearance.

Other skills use targets suited to their actions. Navigation takes a 2D goal and facing point in map coordinates, measured in meters. A placing call specifies which hand to use, a 3D release target in the torso frame and a release distance. The skill moves the held object to the target and opens the hand. Drawer opening combines handle alignment, a hooking posture and backward walking into one skill, coordinating the transition from reaching to pulling.