Benchmark

RoboDojo – Real

Real-world builds of ten tasks from the RoboDojo simulation benchmark, two each for generalization, memory, precision, long-horizon and instruction-conditioned manipulation, run on a bimanual station and scored by milestone.

Fine-tuned
Fine-tuned with our 🤗 500 released teleoperation episodes
Zero-shot
Run as released, with no training on our data

Leaderboard

#ModelMethodScore
1AnthropicFable 5.2 High FastZero-shotLLM38.5
2Dream LabsDream-1Fine-tunedWAM22.0
3Physical Intelligenceπ0.5Fine-tunedVLA20.6
4Dream LabsDream-1Zero-shotWAM8.5

By task

Task
AnthropicFable 5.2 High FastZero-shot
Dream LabsDream-1Fine-tuned
Physical Intelligenceπ0.5Fine-tuned
Dream LabsDream-1Zero-shot
Generalization
39.054.551.018.5
15.513.011.510.0
Memory
64.09.07.01.5
23.05.01.04.0
Precision
37.017.09.00.0
37.014.516.010.5
Long-Horizon
62.510.038.021.0
31.035.015.010.0
Instruction-Conditioned
40.041.021.51.0
36.521.036.08.5

Generalization

Pack Objects Into Box

Task definition
Zero-shotFable 5.2 High Fast
Fine-tunedDream-1
Fine-tunedπ0.5
Zero-shotDream-1
Zero-shot
Fable 5.2 High Fast

Episode 1 of 10

15/100

Recording
Download data · 833 MB
Instruction
Place the car, shoe, toothbrush, and dual-face hammer into the open cardboard box. Point the car nose, shoe toe, toothbrush head, and hammer's yellow striking face to the left.
Layout
Sampled start layout, seen from the overhead camera; arms at the bottom. Hover a shape for its name.
Scoring
ScoreMilestoneThis episode
0No milestone below is reached.✓
5A task object is grasped and lifted clear of the table.✓
15One object is packed and the gripper is open.Reached
30Two objects are packed.
45Three objects are packed.
60All four objects are packed.
80All four objects are packed and at least two are oriented.
100All four objects are packed and oriented, and the box is within its initial footprint.

Contact SE3 Labs.

Tell us what you are building and where real-world evaluation could help.

conan@se3labs.ai