octopus-workflow/core/skills/headless-session-ops/reference/quota-wall-recovery.md

> Added from headless-session-ops/SKILL.md (triggers `quota wall` / `配额墙`,
> Post-launch lifecycle section, pitfall #17, References) — new runbook
> 2026-08-29, ticket [org-internal #3669] (evidence instance [org-internal #3627]). Companion runbooks:
> `lifecycle-ops.md` (hung rescue, fact-baseline structure) and
> `completion-report.md` §"Orchestrator side" (the quota self-report clause);
> §Prevention is this file's own closing section.

## Quota wall — worker killed mid-task by a provider usage limit

A **quota wall** is when a headless worker's turn is killed by a terminal
provider-quota error (usage limit / 429 family) mid-task. From the issue side
it looks EXACTLY like silent completion: no completion-report, no
`status=blocked` report, the ticket just stops moving. The stalled window
equals the orchestrator's polling interval, and without this runbook the
recovery was improvised.

**Evidence instance ([org-internal #3627], 2026-08-29)** — the shapes below are field-verified:

|时刻 (+08)| 事件 |
|---|---|
| 08-29 13:18 | review-code Round-1 synthesis (FAIL) posted on [org-internal #3627]; worker session `ses_fb46e705bffe9kYbSoHayoJ1OQ` (builder, opencode-go model, branch `workflow/session/3627-wedge-promptasync`) entered its revision loop |
| ~13:18–16:00 | worker hit the provider's `weekly usage limit reached` mid-revision — turn killed, **no** completion-report, **no** blocked report; issue side silent |
| detection | orchestrator noticed the `/session` state stall, then confirmed via the quota error in the session log (message tail carries the ⚠️ usage-limit part, [org-internal #2912]) |
| recovery | re-wake via `prompt_async` on a known-good provider (`zhipuai-coding-plan`/glm-5.3 — since renamed `zai-coding-plan`) **plus a fact-baseline injection** (what happened / what is done / where to resume) |
| 16:26 / 16:51 | Round-2 synthesis PASS; `status=done pr=[org-internal #3675]` report received |

## Detection — three signals, in this order

Set the usual variables first:

```sh
BASE=http://127.0.0.1:4096   # dev backend (:4180 prod)
DIR=/data/octopus
SID=<worker session id>
```

**Signal 1 — tail probe (primary; survives restarts, lives in the DB).** Since
[org-internal #2912]/[org-internal #3190] a quota-killed turn persists an assistant message with
`finish=error` plus a human-readable ⚠️ text part whose metadata names the
quota family. Probe the TAIL (`limit=1`, never the no-`limit` full transcript):

```sh
curl -s "$BASE/session/$SID/message?directory=$DIR&limit=1" | jq '
  {role: .[0].info.role, finish: .[0].info.finish,
   error_parts: [.[0].parts[]? | select(.type=="text" and .metadata?.error == true)
                 | {text: .text[0:160], metadata: .metadata}]}'
```

Quota-wall verdict: `role=assistant` AND `finish=error` AND an error part with
`metadata.reason` in `{account_rate_limit, free_tier_limit, account_usage_limit,
rate_limit}` (metadata also carries `provider`). The ⚠️ text is one of — all
strings observed in this repo's runtime (`session/retry.ts`):

- `` `weekly usage limit` reached. It will reset in … `` / `Go limit reached`
  (GoUsageLimitError — the [org-internal #3627] shape; `weekly` is the body's `limitName`)
- `Free limit reached` (FreeUsageLimitError)
- `Account-level usage limit — switching API keys does not reset it. You can
  continue after HH:mm.` ([org-internal #3407], zhipu/zai code 1308 — account-level)
- `Provider <id> rate limited (HTTP 429)` (generic 429)
- `Generation failed — …` with a 402 `insufficient_quota` body (balance, not
  window — recovery is top-up, not provider switch)

**Signal 2 — state (two shapes; use the lifecycle-ops dual-sample rule).**

```sh
curl -s "$BASE/session?directory=$DIR" | jq -r --arg s "$SID" \
  '.[] | select(.id==$s) | .state'
```

- **idle** + Signal-1 tail → the turn already died on the quota error. No
  abort needed; go straight to Recovery.
- **generating** with `progress`/newest-message-id frozen across two polls
  30–60 s apart → the 429 retry spin is still running (it retries with backoff
  until the schedule ends) OR the stream is hung — treat exactly like the hung
  shape in `lifecycle-ops.md`: abort FIRST, then re-wake. A single snapshot is
  never an abort basis ([org-internal #3215]).

**Signal 3 — provider-level confirmation ([org-internal #2911] quota markers).** `GET
/provider` overlays active usage-limit cooldowns, so the dead provider is
visible without reading logs:

```sh
curl -s "$BASE/provider?directory=$DIR" | jq '[.all[] | select(.quota) | {id, quota}]'
# quota: {markedAt: <epoch-ms>, resetAt?: <epoch-ms>} — resetAt absent = reset unknown
date -d @$(($(curl -s "$BASE/provider?directory=$DIR" | jq '[.all[].quota.resetAt // 0] | max') / 1000))  # earliest full-reset wall clock
```

**Log grep (fallback; the dev log is restart-truncated).** Patterns actually
seen in the wild — [org-internal #3627] produced the first:

```sh
grep -E "usage limit reached|Go limit reached|Free limit reached|Account-level usage limit|GoUsageLimitError|FreeUsageLimitError|rate limited \(HTTP 429\)|insufficient_quota" \
  /tmp/octopus-backend.log | tail -20
# the pool-failover WARN precedes the wall when same-name pools exist:
#   "Usage limit reached on <provider> — retrying <model> on <provider>"
```

The wall is only terminal after failover has exhausted the same-name pools
([org-internal #2911] failover); the `retrying … on …` WARNs above tell you the pool was
already draining before it died.

## Recovery — abort (only if needed), re-wake on a known-good provider, inject facts

**Step 0 — classify the state (Signal 2).** `idle` → skip abort.
`generating` + frozen across dual samples → abort first, exactly per
`lifecycle-ops.md` (queued prompts are never consumed by a dead stream):

```sh
curl -s -X POST "$BASE/session/$SID/abort?directory=$DIR" -o /dev/null -w "%{http_code}\n"   # → 200
curl -s "$BASE/session?directory=$DIR" | jq -r --arg s "$SID" '.[] | select(.id==$s) | .state' # → idle
```

**Step 1 — inventory surviving work.** The killed turn's filesystem writes
survive; list them so the worker re-validates instead of redoing:

```sh
git -C <worktree> status --short
git -C <worktree> log --oneline -3
```

**Step 2 — pick a known-good provider/model.** The re-wake payload MUST carry
an explicit `model` (SKILL.md Iron Law), and after a quota wall the CURRENT
provider is by definition NOT known-good. Pick a provider that is connected
AND not quota-marked:

```sh
curl -s "$BASE/provider?directory=$DIR" | jq -r \
  '.connected as $c | [.all[] | select(.id as $id | $c | index($id)) | select(.quota | not) | .id] | join("\n")'
# then pick a modelID you know works on that provider (e.g. the orchestrator's
# own current model — it is generating by definition).
```

([org-internal #3627]: opencode-go dead → re-woken on zhipuai-coding-plan/glm-5.3, since
renamed `zai-coding-plan`. Do NOT "wait for reset" as the default — weekly
windows mean days.)

**Empty list — every connected provider is quota-marked.** The loop needs an
explicit exit; never re-wake into a wall you can see. Find the earliest reset
among the marked connected providers:

```sh
curl -s "$BASE/provider?directory=$DIR" | jq -r \
  '.connected as $c
   | [.all[] | select(.id as $id | $c | index($id)) | select(.quota)]
   | min_by(.quota.resetAt // 9007199254740991)   # resetAt absent → sorts last
   | "\(.id): markedAt=\(.quota.markedAt) resetAt=\(.quota.resetAt // "unknown")"'
# resetAt is epoch-ms → wall clock: date -d @$((<resetAt> / 1000))
```

- Known `resetAt` within reach → suspend the re-wake (leave the worker down)
  and re-run this Step after the reset; weekly windows mean days — schedule
  the retry, do not poll in a tight loop.
- `resetAt` unknown, or the earliest window is unacceptable (402
  `insufficient_quota` is balance — reset means top-up, not time) →
  **escalate to the human orchestrator**: report every marked provider's
  `id`/`markedAt`/`resetAt` and stop. Top-up or onboarding a new provider is
  a human decision, not another re-wake.

**Step 3 — re-wake with a fact-baseline prompt.** Build the payload from a file
(`jq --rawfile`, SKILL.md pitfall #7), same wake shape as
`lifecycle-ops.md` but with the quota facts. The template also restates the
completion-report protocol ([org-internal #2374]) so the recovery is self-contained: it
assumes the original seed still contains the protocol block — if you are
unsure it does (e.g. the worker was dispatched before [org-internal #3669] added the quota
clause), the template's own block below carries it; fill
`<BASE>`/`<LAUNCHER_SID>`/`<DIR>` with the orchestrator session's values:

```sh
cat > wake.txt <<'EOF'
## 事实基线（配额墙恢复）
你上一回合被供应商配额错误杀死，世界观停留在被杀时刻。先读以下事实再行动：
1. 发生了什么：<HH:mm> 你的回合因 <provider> 配额耗尽终止（usage limit，
   预计重置 <HH:mm 或 未知>）；本消息由已知良好供应商重新唤醒，非你自行恢复。
2. 已做什么：<已完成步骤/commit/PR 状态>；worktree 未提交变更：
   <git status --short 摘要>——先 re-validate 这些文件，勿重做已完成工作。
3. 从哪继续：<一件具体的下一步任务>。
4. 证据规则：逐条到引用来源（issue 评论 / PR / 工件）核实后再行动，
   勿仅凭本基线行动。
## 完成回报协议（重申 [org-internal #2374]/[org-internal #3669]；不确定原始 seed 是否含协议块时以本节为准）
完成、阻塞或移交时，必须回报编排会话（你的唤醒者）：
  POST <BASE>/session/<LAUNCHER_SID>/prompt_async?directory=<DIR>
  body: {"agent":"builder","model":{...},"parts":[{"type":"text","text":"<report>"}]}
<report> 单行格式：[#<ticket>] [<node-or-slug>] status=done|blocked|handoff pr=#<N> summary=<一行>
约束：若再遇配额/用量上限错误，立即回报 status=blocked
reason=quota-exhausted summary=<provider 与重置提示>（pr=#<N> 有 PR 时带上）
后停止，勿自行换模型重试；回报失败 → 源 issue 评论兜底。
EOF
jq -n --rawfile p wake.txt '{agent:"builder",
  model:{providerID:"<known-good providerID>",modelID:"<known-good modelID>"},
  parts:[{type:"text",text:$p}]}' \
| curl -s -X POST "$BASE/session/$SID/prompt_async?directory=$DIR" \
  -H 'content-type: application/json' -d @- -o /dev/null -w "%{http_code}\n"   # → 204
```

Then poll the tail (`limit=1`) until an assistant message appears, and keep
expecting the completion-report per protocol — the re-woken worker owes you
`status=done|blocked` like any other (the restated block in the template
makes this hold even when the original seed predates [org-internal #3669]).

## Prevention — before dispatch, and in the seed

**Pre-dispatch provider check (30 seconds, catches most walls):**

```sh
# 1) no active quota markers on the dispatch provider (else pick another)
curl -s "$BASE/provider?directory=$DIR" | jq '[.all[] | select(.quota) | .id]'
# 2) dispatch provider is connected AND ≥1 other connected provider exists as fallback
curl -s "$BASE/provider?directory=$DIR" | jq '{connected, fallbacks: (.connected | length > 1)}'
```

If the intended provider is already quota-marked, dispatch on a different one
— do not launch into a wall you can see coming.

**Quota self-report clause (turns a silent death into a harvestable
blocked).** The canonical clause text lives in `completion-report.md`
§"Orchestrator side" (配额自报, [org-internal #3669]) — append it VERBATIM to the seed
prompt's completion-report block. Single source of truth: do not fork or
restate the clause here; its blocked payload already follows the one-line
key=value spec (`status=blocked reason=quota-exhausted summary=<provider +
reset hint> pr=#N`, completion-report.md `<report>` format).

With the clause in the seed, a worker that hits the wall reports `blocked`
instead of dying silently — the orchestrator harvests the report and runs the
Recovery section directly. The clause is also why the re-wake prompt's
constraint (Step 3) repeats it: the re-woken worker must know the rule still
holds on the new provider.