Four Claude agents turn one sentence and a depth image into controller skills. The whole stack runs in your browser.
MuJoCo compiled to WebAssembly, driving a Franka Panda in a web worker on your machine. The built-in demo planner needs no account.
Planning with Claude? Open the studio and connect either a Claude API key or Amazon Bedrock credentials. Requests go from your browser straight to the provider you choose, and are billed to you.
Each agent has one job and a typed output. Their answers compile into a stack of task-axis controllers that runs at control rate, with no model in the loop.
Everything below is copied from a recorded episode: Claude Sonnet 4.5 on Amazon Bedrock, asked to "put the bottle into the box".
"The bottle is standing upright on the table to the right of the box. The box appears empty with no obstructions inside. The bottle's cap end is at the top, making it suitable for top-down grasping."
For the first step it defined a skill called approach_bottle_pregrasp and asked the Spotter for one keypoint, the centre of the bottle's cap.
# subtask 1: Approach bottle for pre-grasp [p1] PosAlign(eef_tcp, bottle_cap, axis=x, o_x) [p1] PosAlign(eef_tcp, bottle_cap, axis=y, o_y) [p1] PosAlign(eef_tcp, bottle_cap, axis=z, o_z) [p2] AxisAlign(eef_z, bottle_cap_normal, o_theta)
o_z0.12 mHover 12 cm above the bottle cap for pre-grasp approacho_theta180°Gripper points down onto upward-facing cap surfaceo_d-0.015 mFingertip centre goes half the cap height below the cap top to centre pads on the capf3 NGentle contact force threshold for placing the bottle
bottle_cap at pixel 377, 1882.2 mm from the simulator's ground truthbox_inside at pixel 265, 28529 mm off, still inside the trayIn this recorded run the bottle was knocked over beside the box. The Composer asked for a keypoint that is in no object list, a free spot on the table, and the Spotter found one. A grasp was missed along the way and the scene was grounded again before the run finished.
table_free_spot marked bottom left.
Every run is seeded and saves the camera frames, 3D keypoints, controller script and a ground-truth success label. That is the raw material for training a manipulation policy.
Swap the Claude model with one flag and read off success rate, cost, latency and grounding error. Ablations turn single stages off.
An arm is a profile: joints, gripper, workspace. A Franka Panda and an SO-ARM101 ship today, and the SO-101 can be driven for real over USB.
A skill is a short list of controllers, each aligning one point or axis with another, or pushing until contact. The runtime stacks them by priority.
# subtask 5: Lower the bottle into the box [p1] PosAlign(bottle_cap, box_inside, axis=x, offset=+0.000 m) [p1] PosAlign(bottle_cap, box_inside, axis=y, offset=+0.000 m) [p1] Force(axis=-z, contact>=3.0 N) [p2] AxisAlign(eef_z, box_inside_normal, angle=180 deg)
A 7-joint Franka Panda and the 5-joint SO-ARM101, which has no force sensor. Same pipeline, a different robot profile.
Recipes that succeed are stored by role, not by object name, and offered again with the numbers that worked.
A failed subtask first triggers a second look at the scene. Only if that fails does the Scout plan again from where things now lie.
The Python stack was ported to TypeScript and a web worker. Claude is called from the page with your own Claude API key or AWS credentials, and skills are kept in IndexedDB. Hosting is static files.
Claude Sonnet 4.5 on Amazon Bedrock plans, a Franka Panda acts. Five tasks, 30 random seeds each, scored by the simulator's own ground truth. Each square is one trial.
No demonstrations and no training: each trial is a fresh plan from one sentence and one camera frame. Accuracy is the share of trials where the object ended up where the task asked: 142 of 150, with a 95% interval of 89.8% to 97.3%. In every miss the arm finished its plan but the object was not in place. Simulation only; nothing here was measured on physical hardware.
| Task | Accuracy | Model calls | Cost | Planning time | Grounding error |
|---|---|---|---|---|---|
| Put the bottle in the box | 100% 30 / 30 | 8.3 | $0.131 | 60.3 s | 12.4 mm |
| Put the red cube in the box | 97% 29 / 30 | 8.0 | $0.133 | 61.9 s | 18.5 mm |
| Stack red on blue | 97% 29 / 30 | 8.0 | $0.131 | 61.0 s | 7.0 mm |
| Blue cube in the box, cluttered table | 97% 29 / 30 | 8.2 | $0.136 | 63.7 s | 24.1 mm |
| Stack blue on red, cluttered table | 83% 25 / 30 | 8.0 | $0.130 | 59.9 s | 17.8 mm |
Averages per trial. Cost is computed from token counts at list price. With the scripted reference planner in place of Claude, the same five Franka tasks solve 250 of 250 trials (50 seeds each), which isolates grounding, control and execution from planning.
Apache-2.0, one repository: the engine you just ran, the Python reference stack and the benchmark harness.
# get the code git clone https://github.com/jay7828/Dooms-inference cd Dooms-inference # build the browser engine npm install && npm run build # or run the Python stack, no AWS needed uv venv .venv --python 3.11 uv pip install --python .venv/bin/python -e ".[dev]" .venv/bin/python scripts/run_task.py \ --task put_bottle_in_box --seed 0 --llm scripted
Our inference engine for Apple Silicon. SmolVLA runs from three cameras to 50 robot actions in 267 ms, 2.3x faster than PyTorch, with C++, C and Python APIs.
Bike, auto and cab rides in Indore. See the fare before you book, and five nearby drivers get your request at once.
Bedtime stories in two languages that children can read, hear and tap along to.