Skip to content

Recovery, budgets and model routes ​

An attempt fails. The simplest harness runs it again until it passes or somebody notices. That hides the reason for the failure, spends the budget on the same mistake, and makes the harness's behaviour depend on how patient it was written to be.

In regulator, what S3 does with a failed attempt is written down as policy. The failure is normalized to a cause. A versioned recovery policy maps the cause, and how often it has recurred on the unit, to exactly one action. The decision is recorded together with the policy version that produced it. Budgets bound what one attempt may spend, and model routes bound what it may run on. Changing how the loop recovers therefore means changing a policy file, which is reviewed like any other part of the definition, and the loop itself stays as it is.

From failure to cause ​

When a unit is blocked, the recovery router reads what was recorded and names exactly one cause. The cause is the key the policy is indexed by, so settling on one is what turns the next step into a lookup:

CauseTypically from
check-failurethe technical verdict failed at closeout
invalid-reportreport_result was refused, or the report did not validate
no-reportthe session ended without a report
budget-exhaustedthe budget guard halted the attempt
timeout, environment, tool-erroran error the session observed, such as a missing command or dependency, no model in the route, or a timeout
dispatch-errorthe dispatch itself failed
conflictreintegration conflicted with the base
oscillationS2's thrash detector signalled that the unit is rewriting the same files
unknownnone of the above
ambiguitynothing yet. The vocabulary and the shipped policy name it, but the router never classifies a failure this way

The router reads three sources: the orchestrator's records (the attempt's outcome and the unit's block reason), S2's coordination signals, and the failure observations the session wrote. The failure observer normalizes every tool error and provider error in a session to a cause as it happens, so the record exists before the attempt ends. That is why an attempt that ends silently can still be routed by what went wrong, and never only by the fact that it went quiet.

When more than one source applies, a fixed precedence decides. Reintegration problems come first, then the attempt's recorded outcome, then an oscillation signal, then the session's last observation. The orchestrator's record wins over the session's, so a check failure caused by a missing tool is routed as check-failure.

From cause to action ​

The recovery policy lists, per cause, the actions for its first, second and third occurrence on one unit. The shipped policy reads as follows:

Cause1st2nd3rd
no-reportretryretryescalate
invalid-reportrepairrepairescalate
budget-exhaustedretryreplan
check-failurerepairrepairreplan
tool-errorretryremediateescalate
timeoutretryretryescalate
environmentremediateescalate
conflictrepairescalate
oscillationclarify
ambiguityclarify
dispatch-errorretryremediateescalate
anything elsepause (the fallback)

Three rules shape how the table is read. Occurrences count per cause, per unit, so the third check-failure on a unit takes the third action, however many other failures came between. The last action listed repeats, so a fourth check-failure is replan again. And the attempt ceiling is a hard stop: an action that needs another attempt, when the unit has used all its attempts, becomes escalate. The policy cannot talk the loop into a fourth try that the budget does not allow.

The decision is immutable. It records the cause, the occurrence, the evidence, the action, the policy's name and version, and a rationale. For retry and repair it also records the hint the next attempt is told. For clarify it records the question someone must answer.

The actions ​

ActionWhat happensWho acts
retryanother attempt, with a hint about what went wrongthe loop, at once
repairanother attempt, told to fix what the verdict or the report gate named; the work already in the worktree standsthe loop, at once
remediatethe environment needs fixing before the unit can run againS3
replanthe contract needs a new versionS3
clarifya question needs an answera person
pausethe unit waitsa person
escalatethe policy has nothing left to try, so an algedonic signal goes out with the evidencea person
abortthe lease is released, the worktree is removed, and the unit is abortedthe loop, at once

Retry and repair spend an attempt, and the loop applies them itself: regulator unit drive keeps running the unit for as long as the router answers retry or repair. It stops at the first decision it cannot apply.

Every other action except abort leaves the unit blocked and opens an obligation for the consumer the routing policy names for that action. Under the shipped policy, remediate and replan are owed to S3, while clarify, pause and escalate are owed to a person. The unit's open S3 obligations are escalated into that one, so the wait has a single owner. When someone dispositions it, the unit can run again. The loop never performs a remediation or writes a new contract by itself.

A unit paused on an unanswered question is not routed at all. It is waiting for a person, and the recovery policy has nothing to say until the person answers.

Budgets ​

The budget policy sets ceilings per unit type, with a default:

CeilingPerShipped default
tokensattempt400,000
costattempt2
wall-clockattempt20 minutes
turnsattempt60
attemptsunit3

Research, planning, verification and close units get smaller ceilings.

Inside the session, the budget guard meters each attempt's tokens, cost, wall-clock and turns. When a ceiling is crossed, the guard halts the attempt and refuses every tool with an effect from then on. The ledger is written to the execution store, so the orchestrator closes on the ledger and never reads the budget off the transcript. A halted attempt is routed as budget-exhausted. The attempt ceiling is the one the loop enforces itself: the dispatch gate refuses a further attempt once it is used up.

Model routes ​

The same policy names a route per unit type: a primary model and an ordered list of fallbacks. The dispatcher runs a unit on the first model in its route that is available. If a provider fails, the dispatcher moves to the next fallback in a fresh session. Nothing outside the route is ever tried, and every model an attempt ran on is recorded in its ledger, so its cost is attributable. If no model in the route is available, dispatch fails with an environment cause.

Compaction that keeps the contract ​

A long attempt fills the context window, and the host compacts it. A default summary is a model's judgement about what mattered, and it can drop a fixed decision. Contract-preserving compaction replaces the summary with a deterministic block first:

  • the unit, the attempt, and the contract's id, version and objective;
  • the fixed, delegated and unresolved decisions;
  • the evidence gathered so far, from run_tests and run_checks;
  • the files modified, and the budget used.

A model summary of the conversation follows when a model is available. When none is, the block says so and stands alone. Nothing the loop needs depends on the summary being good. The contract section in the system prompt is regenerated every turn anyway, so what compaction preserves is the unit's progress. It is never the only copy of the contract.

What is not there yet ​

The registry records for the recovery router, the failure observer, the budget guard, the model router and compaction state their limitations. In short:

  • Classification is a precedence over recorded facts. A failure with two causes is routed by the first one the precedence finds.
  • Counting per cause can hide a pattern. A unit that alternates between two causes never reaches the third action of either. The attempt ceiling stops it instead, and the alternation shows only in its decisions.
  • Observations are pattern matches. The observer classifies error text with a regular expression, and it sees only errors the tool layer reports as errors. A bash command that exits non-zero without the tool flagging it is invisible.
  • The budget guard checks between turns. The turn that crosses a ceiling completes. A provider that reports no usage meters as zero, and wall-clock counts from session start.
  • Failover starts from scratch. A fallback model does not see what the primary did, only the contract. "Available" means only that an API key is configured; the provider may still be down.
  • The compaction summary is unchecked. Only the deterministic block is guaranteed, and it sees evidence from run_tests and run_checks only, never from bash.
  • ambiguity has a rule and no source. The shipped policy routes it to clarify, but no classification produces it yet.
  • Nothing performs a waiting action. Remediate, replan, clarify and pause are recorded and owed to someone, and the loop waits until that someone acts.

Released under the Apache-2.0 License.