Where Should a Robot’s Intelligence Live? Code-as-Policy versus Agentic Control

The task and how we measure it

The instruction is simple: put the red LEGO brick in the dark gray box. The robot must pick the target out from colored distractors, grasp and lift it, carry it over the box, and release it inside. Carrying out that sequence takes more than a plan for the expected case. The controller also needs a way to decide what to do when the next observation does not match that plan.

Progress, completion, and quality

Progress records task milestones on a 0–100 scale. The score sheet below defines the values used throughout the post.

A Progress value of 100 means the brick is inside the box, where it may still be held by the gripper. Completion requires the brick to be released, fully inside the box, and at rest for at least two seconds at the end of the episode. A human operator scored every episode on-site against this rule.

A separate 100-point quality score describes how the robot got there: grasping contributes up to 30 points, transport 30, and placement 40. A recovered drop can lead to a completed task with a lower quality score. This keeps the outcome separate from the smoothness of the execution.

Two places to put the reasoning

We used GPT6-Astra to build two controllers for a bimanual YAM station. In code-as-policy, the model develops and tests a Python program, then we freeze that program for evaluation. The program reads observations and controls the robot without further model calls. In agentic control, the model stays in the execution loop: it requests camera snapshots, interprets them, and issues the next commands through a control harness.

Both systems use feedback. The architectural choice is where new evidence gets interpreted: by logic written before the episode, or by the model during the episode.

One task, 20 episodes per controller

A human operator repositioned the target brick, colored distractors, and box between episodes. These layouts varied reach, clutter, and occlusion.

Getting hold of the brick is not enough

6 of 20 placements completed with code; 18 of 20 with agentic control

The code policy lifted the brick in 14 of 20 episodes, but completed only 6 placements. Agentic control retained a lift in 18 of 20 episodes and completed all of those placements. Much of the gap appeared after the brick was already in the gripper: the code policy often acquired the target without carrying the task through to release.

Half of the agentic completions involved grasp retries. 9 of its 18 completed episodes needed more than one grasp attempt; one also included a recovered drop. The partially filled dots in the Released column capture that distinction: the task finished, but the execution earned less than full quality credit. Of the code policy’s 14 retained lifts, four stopped short of the release pose and four reached it without letting go.

What happens after a failed attempt?

Retries carried nine agentic episodes through to completion

A missed grasp changes the scene: the target may move, rotate, or become harder to separate from its neighbors. In the successful retry episodes, the agentic controller repositioned and grasped again. The code policy made no second grasp attempt in this evaluation. Its failures included eight episodes in which it held the target but never completed placement. The figure groups each episode by the path it took to its final outcome.

A stalled placement versus a recovered drop

These two trajectories illustrate the difference. Code-as-policy - episode 15 carries a stable grasp to the box but never releases it. Agentic Policy - episode 8 shows repeated attempts to acquire the target, followed by a drop and a successful recovery. The videos play at 10× speed; the captions report elapsed time in the source recordings.

Persistence does not resolve every perception problem

The two agentic failures expose different limits. In Agentic Policy - episode 5, three attempts fail to retain the target, displacing it out of the visible work area on the last attempt. In Agentic Policy - episode 7, the gripper stays open and never purposefully contacts the partly occluded brick. Repeated action can make a scene harder, while continued observation need not produce an actionable grasp.

The price of staying in the loop

The additional completions came with more time and ongoing model use. To understand that tradeoff, we separate time to release from total recording length, and model costs during development from those during execution.

Successful code-policy placements finished sooner

Among completed placements, the median observed release time was about 131 seconds for code-as-policy and 175 seconds for agentic control. Those medians cover six and eighteen completions, respectively. The timeline shows the full sample, including the agentic recovery that reached release at about 419 seconds.

An incomplete episode ends at its recorded duration with no observed release.

Model cost moves from development into each episode

The code policy made no model calls during evaluation. Agentic control used $53.60 across its 20 episodes, averaging $2.68 per episode. It used an existing generic harness with no task-specific development, so its development cost for this task was $0. Counting task-specific development and evaluation model use, cost per completed placement was $8.31 for code-as-policy and $2.98 for agentic control.

To project cost over repeated episodes, we divide total task-specific model spending by the expected number of completed placements:

N

Number of evaluation episodes

D

Task-specific development model cost (USD)

c

Mean runtime model cost per episode (USD)

p

Observed completion rate, held constant

Code-as-policy’s cost per completion declines as its development cost is spread across more episodes. Agentic control’s projected cost is constant: $2.68 per episode divided by a 90% completion rate gives $2.98 per completed placement.

With the observed completion rates and mean runtime model costs held constant, code-as-policy would become cheaper per completion at about 56 total episodes.

The reusable program also takes work to build

Code-as-policy development included stereo depth estimation, camera geometry, and compensation for joint-tracking error, as well as task logic. The milestones below show how that implementation progressed from control primitives to a frozen policy. Agentic control used an existing generic harness with Cartesian motion tools and required no task-specific development.

Limitations

Scenes were arranged by hand

A human operator arranged each layout rather than generating it procedurally, so the two controllers did not face an identical, controlled set of scenes.

Motor heat may be a confounding factor

In the code-as-policy evaluation, time to first gripper closure increased by about 0.53 seconds per episode across the 14 episodes with a detected closure. Heat accumulating in the motors over the run may have contributed to this drift and affected outcomes.

Where should the intelligence live?

In the agentic recordings, the model could use new observations to choose another attempt, sometimes recovering from its own earlier actions. That flexibility was useful, and it required time and inference on each episode.

A reusable program can spread its development cost across repeated executions. Its recovery behavior depends on the logic built into it. Agentic control can make new decisions during execution.

A next experiment could combine the two: use a tested program for routine motion, check explicit milestones such as retained lift and confirmed release, and ask the model to intervene when progress stops. It would test the practical question raised by these recordings: how much online reasoning is needed, and at which moments does it earn its cost?

Explore all 40 episodes