Lego-RL: Harness-Native Reinforcement Learning for Coding Agents

Abstract

The ideal way to run reinforcement learning on a coding agent is simple to state: whatever agent you deploy in production, train with that same agent. Follow that road in practice, though, and you hit an engineering wall that has nothing to do with the algorithm:

• the agent framework (the harness) quietly rewrites its own conversation history while it runs;
• the model finds shortcuts that reach the reward while bypassing the task itself;
• more than 90% of a rollout's time is spent inside the agent's interaction loop, while the trainer can do nothing but wait.

We took that wall apart brick by brick and built Lego-RL. Without modifying a single line of the native control flow of OpenHands SDK, Claude Code, or OpenCode, we plugged all three directly into GSPO (Group Sequence Policy Optimization) training. The same Qwen3.5-35B-A3B weights (a sparse MoE) go from 64.0, 62.4, and 57.2 on SWE-bench Verified to 70.4, 68.2, and 66.6.

Lego-RL has three pillars, one for each problem:

• Faithful optimization: capture rollouts at the serving boundary and replay expert routing, holding the median train-inference log-probability correlation at 0.998 or above.
• Reliable execution: screen the task pool, admit trajectories into the loss by termination reason, defend the verifier phase by phase, and overlap the 91% of wall-clock owned by agent execution with the optimization step.
• Observable training: validate configs and resources automatically before a run launches, and drill any run down to the level of a single trajectory.

Why insist on training inside the native harness?

The mainstream recipe for agentic RL reshapes the agent into a form the training framework accepts: rewrite its initialization logic, swap in the framework's own tool set, then bolt on termination logic so the reward can be read back out. The recipe works, but it quietly changes the optimization target. The policy you train is optimal under that remodeled control flow, not necessarily under the harness users actually deploy.

How much does the harness matter? The starting model is the cleanest evidence we have: the same Qwen3.5-35B-A3B weights, dropped into three different harnesses, score 64.0, 62.4, and 57.2 on the same SWE-bench Verified suite. Swapping the harness alone opens a gap of nearly 7 points, larger than the headline gains of many post-training methods.

Lego-RL takes the top score in every column, beating the newer Qwen3.6-35B-A3B by more than the entire 3.5→3.6 generational jump. So we trained on unmodified harnesses: same checkpoint, 2,699 tasks, 200k context, 126 steps each. Reward climbs on all three curves and entropy never collapses, while mean response length nearly doubles under OpenHands SDK (43.5k → 90.9k) but only grows from 41k to 51k under Claude Code. The harness shapes the model's behavioral personality.

Overall architecture: one infrastructure, shared by all harnesses

Lego-RL builds its trainer on verl and its sandboxed execution on Harbor. A new harness needs one lightweight adapter that launches the agent, points it at the inference service, and passes the interaction data back; everything downstream is shared.

The optimization objective: a formal definition

A task instance $x=(q_x,R_x,V_x)$ consists of a problem statement, an initialized repository environment, and a task-specific executable verifier. The harness $\mathcal{H}$ belongs to the environment, not to the policy. At turn $t$, the harness maps the current interaction and repository state $s_t$ to a context $c_t=\mathcal{H}(s_t)$, the policy generates an action $a_t\sim {\pi }_{\theta }(\cdot \mid c_t)$, and the harness executes whatever tool actions were requested, producing $s_{t+1}$. A rollout is the sequence of prompt/response pairs actually exchanged at the model API, and the verifier ultimately outputs a single bit:

Only policy-generated tokens participate in training. Writing $\mathcal{M}(\tau )$ for those positions:

Every turn conditions on the harness-supplied context $c_t$, not on the raw history. We maximize the expected verifier reward $J(\theta )={\mathbb{E}}_{x\sim \mathcal{D},\tau \sim \mathcal{H}({\pi }_{\theta })}[r(x,\tau )]$ with group-relative advantages. Each task samples $G$ trajectories and takes ${\hat{A}}_i=(r_i-\bar{r})/(\mathrm{std}(r_{1:G})+\delta )$, with $\delta =10^{-6}$. All three production runs use GSPO's sequence-level surrogate:

where ${\theta }_{k^{\prime }}$ is the policy version that generated the group, and $w({\tau }_i)\in \{0,1\}$ zeroes out trajectories terminated by infrastructure failures. The asymmetric bounds ${ϵ}_{\mathrm{low}}<{ϵ}_{\mathrm{high}}$ give the sequence-level ratio more room to move up than down. Replacing ${\sigma }_i$ with the per-token ratio recovers PPO or GRPO, which the trainer also supports.

Two properties decide the engineering that follows. When rewards within a group are identical, ${\hat{A}}_i\equiv 0$: the group stays in the batch but contributes no gradient, so task difficulty relative to the current policy is on the critical path. And $r$ comes from executing code, so the signal is only as credible as the sandbox that produced it.

Pillar 1: Faithful optimization

The core difficulty: the archived transcript ≠ the token sequence at sampling time

The most intuitive way to log training data for an agent is to save the conversation transcript and re-tokenize it at training time. That is enough for SFT. On-policy RL needs the log-probabilities of the exact token sequence generated at sampling time, and the quantity recomputed from a transcript is not that quantity.

Real harnesses do rewrite their own history:

• Claude Code injects <system-reminder> blocks mid-conversation;
• OpenHands performs history compaction once the context window fills;
• tool-call arguments get re-serialized, sometimes with keys in a different order;
• sub-agents share a session with their parent agent.

Any one of these plants an error in the importance-sampling ratio that never raises an exception. It just drifts.

Lego-RL's answer: an in-process proxy at the serving boundary

The proxy speaks both the Anthropic and the OpenAI protocol, so the harness sees nothing but a changed base URL. At the moment of generation it records token IDs, response masks, and log-probabilities along with MoE expert routing. Context alignment then runs turn by turn:

• match tool calls by ID, not by serialized arguments;
• isolate sub-agent sessions to avoid cross-contamination;
• discard history the harness compacted away rather than reconstructing it, since reconstruction would itself introduce bias.

MoE routing replay: a detail that is easy to miss

MoE (mixture-of-experts) models add one further requirement: identical token IDs are not sufficient. If vLLM routes token $t$ to experts {3, 17} at sampling time but the training-time forward pass selects {3, 41}, the two are not computing the same probability.

Replaying rollout-time routing (R3) lifts the correlation from 0.9946 to 0.9993. Two silent bugs showed why this has to be checked, not assumed: a one-position routing misalignment scored worse than no replay (0.750 vs 0.995), and an undersized capture buffer for hybrid-attention models wrote out-of-bounds entries as 0, so coverage decayed to 24% before the guard started raising. After the fix, coverage stays above 99.8%.

Formally, write ${\ell }_{i,(t,j)}^{\mathrm{roll}}$ for the log-probability recorded at generation time and ${\ell }_{i,(t,j)}^{\mathrm{train}}(\theta )$ for the value the trainer recomputes. Faithful optimization requires:

That identity is on the same weights ${\theta }_{k^{\prime }}$. Under async training the trainer's ${\theta }_k$ runs ahead of ${\theta }_{k^{\prime }}$ — bounded off-policyness, which ${\sigma }_i$ corrects. A violation of the identity is a capture defect, and nothing corrects for that. Across the three production runs, median train/inference correlation never falls below 0.998.

Pillar 2: Reliable execution

Reward integrity

Reward is the task's own tests: 1.0 resolved, 0.0 otherwise. No reward model, so no reward-model drift — and every shortcut to a score without a fix has to be sealed off, because a strong coding model will find them.

Defenses sit in the task environment, on by default, and flip with Harbor's phases: denied while the agent runs, restored for grading.

Difficulty is relative to the current policy

Each task samples 8 rollouts. All-success or all-fail groups contribute no gradient, so a fixed pool gets less informative as the policy improves: on OpenHands SDK, zero-variance groups climb from 44.7% to 51.4%; OpenCode holds around 43.3%.

Screening 36,884 OpenSWE candidates — validity, executability, then 1–3 solves out of 4 with Qwen3.6-27B on OpenHands SDK — leaves 2,699 tasks that also transfer to Claude Code and OpenCode. An ablation on four 951-task pools confirms the cut: the selected band and its upper half both improve; the lower half and an unscreened sample do not (72.7% of the unscreened pool was never solved).

Infrastructure failures are masked from the loss — 7.1% Claude Code, 2.4% OpenHands SDK, 6.4% OpenCode — and kept in the batch at zero weight. Trajectories that hit the turn or token ceiling still count.

91% of a trial is the agent executing

Mean OpenHands SDK trial: 920s, 91.3% in agent execution. Sync screening stalled 31 times at batch boundaries (median 38.7 min). Async cuts per-step time 2.5×, but the trainer still waits on generation 40.8–66.1% of the time.

Partial rollout syncs weights without discarding a ten-minute trial: interrupt vLLM, load, continue on the same replica with the prefix KV cache. The harness sees one HTTP request.

Pillar 3: Observable training

Failure attribution: quickly telling policy problems from infrastructure problems

One run died at step 3 because the tool-call parser was set to hermes instead of qwen3_coder. In another, validation reward fell from 0.556 to 0.150 — it looked like policy collapse, but only 60 of 172 trajectories had reached grading; the rest never finished environment setup. A third killed all 1,024 trajectories on turn one, again from an incompatible parser. One run did collapse for real, and trajectory-level analysis produced an early-stop condition that would have fired 8 steps earlier.

What did training actually change? Evidence at the behavioral level

A rising mean reward cannot say what the model learned; behavior can. We drew 420 trajectories from each end of the OpenHands SDK production run and analyzed the agent's behavioral patterns.

The clearest change is in self-verification:

The model picked up a working habit: look around before editing, and check the result after every change. Error recovery barely moved — among trajectories that hit a failed command, the share that still solves the task rises only from 63.9% to 66.8%. A terminal binary reward only sees the final outcome, so verification gets reinforced while mid-course correction stays flat.

The gains lean toward reliability rather than coverage. pass@8 improves 4.7 points (83.2 → 87.9); pass⁸ improves 11.1 (28.3 → 39.4). Malformed tool calls fall from 1.07% to 0.15%. Response length grows because there are more turns, not longer ones (46.6 → 83.1 turns, +17% tokens per turn). Budget the turn limit as well as the context window, or late-training trajectories will be cut off at the ceiling.

Quick start

git clone https://github.com/LegoX/Lego-RL
bash scripts/setup_env.sh
bash scripts/train/train.sh train/configs/my_run.env

To reproduce, start from a single-node 8-GPU sync_* template, and enable full async only once the single-node loop becomes the bottleneck. Typical signs: an MoE that cannot share GPUs with vLLM, a 200k context window, or a trainer whose idle ratio shows it waiting on rollouts.

Lego-RL is part of the LegoX series, alongside SWE-Lego (SFT recipes), Terminal-Lego (trajectory-quality screening), and SWE-Review (inference-time generate-review-revise).

BibTeX

@misc{du2026legorlharnessnativereinforcementlearning,
  title={LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents},
  author={Yiming Du and Yuxin Jiang and Tao Yuan and Jianbo Dai and Shaowei Wang and Jierun Chen and Chaofan Tao and Xianzhi Yu and Lifeng Shang and Kam-Fai Wong and Xiaohui Li and Haoli Bai},
  year={2026},
  eprint={2608.17393},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2608.17393},
}