Shape assembly
The robot picks up a circle, square or triangle block, turns it to line up, and pushes it through a hole in a plate. Setups change the fit, the start position, clutter, lighting, camera, delay or bumps.
Shape assembly detailsCompare VLA models on the same robot and the same tasks.
VLA models look at a camera image, read an instruction and move a robot arm. We run each one on the same simulated Franka Panda arm, with the same 30 task setups and the same starting positions. Only the model changes, so the scores compare fairly.
Octo-base and Octo-small didn't fit a single block in 3,000 test tries.
The task can be done: a scripted controller given the exact block positions succeeds in 1,490 of 1,500 test tries. OpenVLA-7B and OpenVLA-OFT have not been tested on this task yet.
Leaderboard
Shape assembly on the Franka Panda, test split. Each model ran the same 1,500 tries. The score is the average success rate over the 7 task groups, with a 95% range. bc-state was trained on this task and the scripted controller knows where the blocks are, so we show them for comparison but don't rank them.
| Model | Track | Score, % [95% range] | Tries | Status |
|---|---|---|---|---|
| Released VLA models, not trained on this task | ||||
| octo-base-1.5 | frozen | 0.0 [0.0, 0.0] | 1,500 | Complete |
| octo-small-1.5 | frozen | 0.0 [0.0, 0.0] | 1,500 | Complete |
| openvla-7b | frozen | not tested yet | - | Set up, not tested on this task yet |
| openvla-oft (LIBERO) | frozen | not tested yet | - | Code written, not tested on this task yet |
| For comparison, not ranked | ||||
| bc-state | adapted | 92.1 [91.1, 92.9] | 1,500 | Complete. Trained on this task, so not ranked. |
| scripted | reference | 99.2 [98.7, 99.7] | 1,500 | Complete. Knows the exact block positions. Shows the task can be done. |
| randomIncomplete: 10 of 50 tries per setup | sanity | 0.0 [0.0, 0.0] | 300 | Incomplete: 10 of 50 tries per setup. Moves at random. |
| holdIncomplete: 10 of 50 tries per setup | sanity | 0.0 [0.0, 0.0] | 300 | Incomplete: 10 of 50 tries per setup. Holds the arm still. |
frozen: released model, not trained on this task; the only ranked track. adapted: trained on this task, so listed apart from the released models. reference: scripted controller that reads the exact part positions from the simulator; shows the task can be done. sanity: moves at random or holds still, with no camera input; checks the task cannot be passed by chance. OpenVLA-OFT (LIBERO) was fine-tuned on LIBERO, a public set of simulated tabletop tasks.
Source: shape-assembly-bench/results/reports/jetson_v1_test/REPORT.mdAll shape assembly results
Benchmarks
Each task setup changes one thing, like lighting or where the parts start, so you can see what breaks a model.
PC assembly
The robot installs parts in a PC case, one part per task: RAM, CPU, cooler, graphics card and motherboard. Latches, levers and keyed slots work like the real ones.
PC assembly detailsA chemistry lab benchmark is in development, with no results yet.
How we test
Every number on this site comes from our benchmark code. If something isn't finished, we say so.
- Same test for every model
Every model gets the exact same scenes, in the same order.
- Every attempt is recorded
We save each run, so any failure can be replayed and watched.
- The test is locked first
We fix the task list before any scored run and never change it after.
- Only fresh models are ranked
The leaderboard ranks released models that were not trained on this task. Everything else is shown for comparison.
- Unfinished runs are marked
If a run is cut short, we say so and show how many attempts it had.
- Scores come with error bars
Each score shows a range, so you can tell a real gap from noise.
Launch video, 60 s
All footage is simulation output from the benchmark environments, ray traced on a Jetson AGX Orin.