Dream in the Real World: Strong Zero-Shot Skills, Despite Limited Data Diversity

What we evaluated

We ran the closed-source Dream model API from Dream Labs zero-shot on a bimanual robot across RoboDojo – Real: our real-world builds of ten tasks from the RoboDojo simulation benchmark, two each for generalization, memory, precision, long-horizon, and instruction-conditioned manipulation.

Every episode samples a fresh table layout and, where a task has them, fresh assignments such as target pegs, button counts, or bins. The robot starts at rest with 180 seconds to act. AI agents reviewed every recording against the task’s scoring sheet: a ladder of milestones from a first grasp up to the full task.

No episode completes its full task, and the mean milestone score is 8 out of 100. 57 episodes reach at least one milestone. Half stall until the timeout; the rest fail on a grasp, a placement, or the wrong object or target.

RoboDojo simulation of Pack Objects Into Box — RoboDojo simulation

Our real-world Pack Objects Into Box setup — RoboDojo simulation

Score	Milestone	Episodes
0	No milestone below is reached.	3
5	A task object is grasped and lifted clear of the table.	1
15	One object is packed and the gripper is open.	3
30	Two objects are packed.	0
45	Three objects are packed.	3
60	All four objects are packed.	0
80	All four objects are packed and at least two are oriented.	0
100	All four objects are packed and oriented, and the box is within its initial footprint. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	1
10	A mug is grasped and lifted clear of the table.	9
25	A held mug's handle reaches its assigned peg.	0
50	One mug hangs and the gripper is open.	0
100	Both mugs hang and the rack is upright. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	7
5	A cup is lifted clear of the table.	3
15	dark blue is covered first by an upright cup.	0
25	Then light blue is covered second.	0
35	Then green is covered third; all three blocks are covered.	0
55	Then light blue was uncovered first; the other two stay covered.	0
75	Then green was uncovered second; dark blue stays covered.	0
100	Then dark blue was uncovered third, and every cup is upright. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	7
10	The first press is on the middle count button.	2
30	The middle count button was pressed exactly 9 times with no other press.	0
45	Then the right confirm button was pressed once.	1
75	Then the left count button was pressed exactly 4 times with no other press.	0
100	Then the right confirm button was pressed once, with no extra, missing, or out-of-order press. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	10
15	The adapter is grasped and lifted clear of the table.	0
35	The adapter's prongs touch the face of outlet 6.	0
60	The prongs are partly inside outlet 6.	0
100	The adapter is seated flush and upright in outlet 6, no prong is exposed beyond the strip's normal fit, and the strip is within its allowed footprint. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	1
10	A tube is grasped and lifted clear of its holder.	9
25	A held tube's tip enters well 6 or 3.	0
50	One tube is seated and the gripper is open.	0
100	Both tubes are seated and the rack is within its allowed footprint. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	2
10	One object is grouped and the gripper is open.	5
25	One bin counts.	2
50	Two bins count.	1
100	All three bins count and all six objects are settled. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	2
5	A loose piece is grasped and lifted clear of the table.	2
15	A held piece is carried to the peg board.	6
30	One piece is seated and the gripper is open.	0
50	Two pieces are seated.	0
75	Three pieces are seated.	0
100	All four pieces are seated. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	9
10	The ruler is grasped.	1
25	The ruler pushes a cube, and no cube was ever lifted above the no-lift height.	0
50	Two cubes rest flush against the same ruler edge with faces parallel within tolerance, and no cube was ever lifted above the no-lift height.	0
100	The three cube centers are collinear and their faces parallel within tolerance, and no cube was ever lifted above the no-lift height. Both arms are back at origin.	0

Score	Milestone	Episodes
0	No milestone below is reached.	1
5	One object is in its instructed bin and the gripper is open.	5
15	Two objects are in their instructed bins.	4
25	Three objects are in their instructed bins.	0
40	One category counts.	0
70	Two categories count.	0
100	All three categories count and all eight objects are assigned. Both arms are back at origin.	0

A low task score hides a lot of competent motion, so we look past it at what the policy does with each object.

Strong zero-shot primitives

With no training on these tasks, Dream grasps, carries, and releases objects of very different shapes. It packs a shoe, a hammer, and a toy car into a box, sorts a car and pens into bins, and drops a pepper and a watch into their instructed bins. It also corrects itself: in the sorting episode below, it places a white pen with the car, then picks it back out and moves it to the black pen.

Car placement and pen sorting with a correction. Car released into blue at 64 s; white pen placed with the car at 123–125 s, retrieved at 148–153 s, and placed with the black pen in orange at 159–160 s. Two category bins completed; camera equipment unfinished.

Packing differently shaped objects. The shoe goes into the box at about 62 s and the hammer at about 100 s; the car follows at about 120 s while the toothbrush stays outside.

Correct placements into two instructed bins. Yellow pepper released into its instructed yellow bin at 58.3 s; black watch released into its instructed orange bin at 135.8 s.

Across ten language-sorting episodes, 12 of 30 peppers finish in their instructed bin and only one in a wrong bin; the rest are never placed. When Dream commits to a placement, it is usually a clean one.

Right motion, wrong goal

A failed grasp says little about what a policy understood. A clean transport to the wrong place says a lot, and Dream produces many of those. Told “Without picking up any cube, use the clear ruler…”, it picks up a cube and carries it; the ruler is never touched. Told to put peppers in blue, figurines in orange, and watches in yellow, it puts the pepper in blue, then the watch and the figurine in blue as well.

Violating an explicit no-lift instruction. At 149–155 s the robot lifts and carries a purple cube to another cube while the ruler remains unused.

Successful transport to explicitly wrong bins. Green pepper is correctly released into blue at 61 s. Yellow watch at 100.5 s and lion figurine at 170.5 s are also released into blue, although their instructed targets are yellow and orange.

Four of the ten language-sorting episodes contain at least one wrong-bin release, five events in total. Dream knows how to put an object in a bin; it does not reliably know which bin. That gap between skill and instruction is the clearest weakness in these episodes.

Skill is concentrated where the data likely is

The success map has a clear shape. Peppers that start in the middle third, in front of and between the two arms, finish in the instructed bin in 9/14 cases (64%), versus 3/16 (19%) from the outer thirds. The peg board shows the same pattern.

0%100% success, smoothedSuccessWrong binNot placed / no approachFill = score

Each dot is one pepper (30, from ten sorting episodes) or one peg board (ten stacking episodes) at its recorded starting position on the 1.2 × 0.65 m table; dashed lines split the table into thirds and grey blocks mark the arm bases. Pie color is the object’s outcome and its fill the episode’s milestone score. Shading is a Gaussian-kernel (σ = 0.1 m) estimate of the success rate, faded where few objects started. Hover a dot for a preview; select it to watch the episode.

We read this as a coverage effect. Robot training data is mostly teleoperated demonstrations, and those tend to cluster where operators work most comfortably: the center of the table, within easy reach of both arms. Dream Labs has not published its data mix, so we cannot confirm this, but a policy trained that way would learn its primitives densely at the center and thinly at the edges. The same model that places a pepper cleanly at the center then stalls or never approaches one 30 cm to the side.

This is the larger lesson. Dream’s zero-shot skills are real, but they appear to be bounded by where the training data reached. More tasks and objects do not by themselves fill in the workspace; coverage of positions and layouts may matter as much as coverage of skills.

The peg board: stacking without the board

With the peg board near the center, Dream carries a loose piece toward a peg. With the board far right, it stacks the loose pieces onto each other on the other side of the table instead: the motions of the task, without the board.

Central board receives a peg-directed transfer. Orange is carried to the board and released over the purple starter around 62 s. Target is wrong and no correct insertion completes.

Far-right board receives no approach. Red is transported onto loose green around 39.6 s and blue onto loose orange around 69.2 s, both on the left tabletop. No loose piece approaches the peg base during the active episode.

Even when it reaches the board, none of the ten episodes seats a piece correctly.

Next: fine-tuning, and a comparison with π0.5

This round measured Dream zero-shot. If limited data coverage is what holds it back, task-specific data should help most where the zero-shot policy is weakest: objects away from the center of the table, and instructions that single out one bin or method over another.

In the next round we fine-tune Dream on demonstrations from our own station and rerun the same ten tasks with fresh layouts and assignments. We will put a fine-tuned π0.5 through the same protocol on the same data, so the two models are compared on equal tasks, data, and scoring.

Explore all 100 episodes

We publish every episode, not just the highlights. Pick one to read its instruction and the reviewer’s account, watch it at 10x speed, see its sampled start layout and which milestone it reached on the task’s scoring sheet, and download the full recording: top-down and wrist cameras plus joint and gripper telemetry for both arms, as a Rerun 0.36 file you can open with rerun recording.rrd.

Generalization

Pack Objects Into Box

Hang Mugs

Memory

Cover Blocks

Press By Number

Precision

Plug In Charger

Insert Tubes

Long-Horizon

Classify Objects

Play Stacking Toy

Instruction-Conditioned

Align Blocks

Classify Objects By Language

Long-Horizon · Episode 6 of 10 · 28__20261026-024050__ep_0003

Classify Objects

50/100 · Timed out

Sort the objects by category into the three bins. Keep all writing tools together, the toy vehicle in its own bin, and all camera equipment together. Any category may use any bin.

Sampled start layout, seen from the overhead camera; arms at the bottom. Hover a shape for its name.

Generated by SE3 evaluation engine.

Two bins count (car in blue, both pens in orange) after one corrected mixed-bin placement; equipment remains unsorted at timeout.

Score	Milestone	This episode
0	No milestone below is reached.	✓
10	One object is grouped and the gripper is open.	✓
25	One bin counts.	✓
50	Two bins count.	Reached
100	All three bins count and all six objects are settled. Both arms are back at origin.

Download recording · .rrd · 716 MB Task sheet

Each pie is one episode: the fill is its milestone score and the color its primary failure mode. Select one to watch it, see its start layout and scoring sheet, and download its recording.