octopus-workflow/core/skills/headless-session-ops/reference/quota-wall-recovery.md

Added from headless-session-ops/SKILL.md (triggers quota wall / 配额墙, Post-launch lifecycle section, pitfall #17, References) — new runbook 2026-08-29, ticket [org-internal #3669] (evidence instance [org-internal #3627]). Companion runbooks: lifecycle-ops.md (hung rescue, fact-baseline structure) and completion-report.md §"Orchestrator side" (the quota self-report clause); §Prevention is this file's own closing section.

Quota wall — worker killed mid-task by a provider usage limit

A quota wall is when a headless worker's turn is killed by a terminal provider-quota error (usage limit / 429 family) mid-task. From the issue side it looks EXACTLY like silent completion: no completion-report, no status=blocked report, the ticket just stops moving. The stalled window equals the orchestrator's polling interval, and without this runbook the recovery was improvised.

Evidence instance ([org-internal #3627], 2026-08-29) — the shapes below are field-verified:

时刻 (+08)	事件
08-29 13:18	review-code Round-1 synthesis (FAIL) posted on [org-internal #3627]; worker session ses_fb46e705bffe9kYbSoHayoJ1OQ (builder, opencode-go model, branch workflow/session/3627-wedge-promptasync) entered its revision loop
~13:18–16:00	worker hit the provider's weekly usage limit reached mid-revision — turn killed, no completion-report, no blocked report; issue side silent
detection	orchestrator noticed the /session state stall, then confirmed via the quota error in the session log (message tail carries the ⚠️ usage-limit part, [org-internal #2912])
recovery	re-wake via prompt_async on a known-good provider (zhipuai-coding-plan/glm-5.3 — since renamed zai-coding-plan) plus a fact-baseline injection (what happened / what is done / where to resume)
16:26 / 16:51	Round-2 synthesis PASS; status=done pr=[org-internal #3675] report received

Detection — three signals, in this order

Set the usual variables first:

BASE=http://127.0.0.1:4096   # dev backend (:4180 prod)
DIR=/data/octopus
SID=<worker session id>

Signal 1 — tail probe (primary; survives restarts, lives in the DB). Since [org-internal #2912]/[org-internal #3190] a quota-killed turn persists an assistant message with finish=error plus a human-readable ⚠️ text part whose metadata names the quota family. Probe the TAIL (limit=1, never the no-limit full transcript):

curl -s "$BASE/session/$SID/message?directory=$DIR&limit=1" | jq '
  {role: .[0].info.role, finish: .[0].info.finish,
   error_parts: [.[0].parts[]? | select(.type=="text" and .metadata?.error == true)
                 | {text: .text[0:160], metadata: .metadata}]}'

Quota-wall verdict: role=assistant AND finish=error AND an error part with metadata.reason in {account_rate_limit, free_tier_limit, account_usage_limit, rate_limit} (metadata also carries provider). The ⚠️ text is one of — all strings observed in this repo's runtime (session/retry.ts):

• `weekly usage limit` reached. It will reset in … / Go limit reached (GoUsageLimitError — the [org-internal #3627] shape; weekly is the body's limitName)
• Free limit reached (FreeUsageLimitError)
• Account-level usage limit — switching API keys does not reset it. You can continue after HH:mm. ([org-internal #3407], zhipu/zai code 1308 — account-level)
• Provider <id> rate limited (HTTP 429) (generic 429)
• Generation failed — … with a 402 insufficient_quota body (balance, not window — recovery is top-up, not provider switch)

Signal 2 — state (two shapes; use the lifecycle-ops dual-sample rule).

curl -s "$BASE/session?directory=$DIR" | jq -r --arg s "$SID" \
  '.[] | select(.id==$s) | .state'

• idle + Signal-1 tail → the turn already died on the quota error. No abort needed; go straight to Recovery.
• generating with progress/newest-message-id frozen across two polls 30–60 s apart → the 429 retry spin is still running (it retries with backoff until the schedule ends) OR the stream is hung — treat exactly like the hung shape in lifecycle-ops.md: abort FIRST, then re-wake. A single snapshot is never an abort basis ([org-internal #3215]).

Signal 3 — provider-level confirmation ([org-internal #2911] quota markers). GET /provider overlays active usage-limit cooldowns, so the dead provider is visible without reading logs:

curl -s "$BASE/provider?directory=$DIR" | jq '[.all[] | select(.quota) | {id, quota}]'
# quota: {markedAt: <epoch-ms>, resetAt?: <epoch-ms>} — resetAt absent = reset unknown
date -d @$(($(curl -s "$BASE/provider?directory=$DIR" | jq '[.all[].quota.resetAt // 0] | max') / 1000))  # earliest full-reset wall clock

Log grep (fallback; the dev log is restart-truncated). Patterns actually seen in the wild — [org-internal #3627] produced the first:

grep -E "usage limit reached|Go limit reached|Free limit reached|Account-level usage limit|GoUsageLimitError|FreeUsageLimitError|rate limited \(HTTP 429\)|insufficient_quota" \
  /tmp/octopus-backend.log | tail -20
# the pool-failover WARN precedes the wall when same-name pools exist:
#   "Usage limit reached on <provider> — retrying <model> on <provider>"

The wall is only terminal after failover has exhausted the same-name pools ([org-internal #2911] failover); the retrying … on … WARNs above tell you the pool was already draining before it died.

Recovery — abort (only if needed), re-wake on a known-good provider, inject facts

Step 0 — classify the state (Signal 2). idle → skip abort. generating + frozen across dual samples → abort first, exactly per lifecycle-ops.md (queued prompts are never consumed by a dead stream):

curl -s -X POST "$BASE/session/$SID/abort?directory=$DIR" -o /dev/null -w "%{http_code}\n"   # → 200
curl -s "$BASE/session?directory=$DIR" | jq -r --arg s "$SID" '.[] | select(.id==$s) | .state' # → idle

Step 1 — inventory surviving work. The killed turn's filesystem writes survive; list them so the worker re-validates instead of redoing:

git -C <worktree> status --short
git -C <worktree> log --oneline -3

Step 2 — pick a known-good provider/model. The re-wake payload MUST carry an explicit model (SKILL.md Iron Law), and after a quota wall the CURRENT provider is by definition NOT known-good. Pick a provider that is connected AND not quota-marked:

curl -s "$BASE/provider?directory=$DIR" | jq -r \
  '.connected as $c | [.all[] | select(.id as $id | $c | index($id)) | select(.quota | not) | .id] | join("\n")'
# then pick a modelID you know works on that provider (e.g. the orchestrator's
# own current model — it is generating by definition).

([org-internal #3627]: opencode-go dead → re-woken on zhipuai-coding-plan/glm-5.3, since renamed zai-coding-plan. Do NOT "wait for reset" as the default — weekly windows mean days.)

Empty list — every connected provider is quota-marked. The loop needs an explicit exit; never re-wake into a wall you can see. Find the earliest reset among the marked connected providers:

curl -s "$BASE/provider?directory=$DIR" | jq -r \
  '.connected as $c
   | [.all[] | select(.id as $id | $c | index($id)) | select(.quota)]
   | min_by(.quota.resetAt // 9007199254740991)   # resetAt absent → sorts last
   | "\(.id): markedAt=\(.quota.markedAt) resetAt=\(.quota.resetAt // "unknown")"'
# resetAt is epoch-ms → wall clock: date -d @$((<resetAt> / 1000))

• Known resetAt within reach → suspend the re-wake (leave the worker down) and re-run this Step after the reset; weekly windows mean days — schedule the retry, do not poll in a tight loop.
• resetAt unknown, or the earliest window is unacceptable (402 insufficient_quota is balance — reset means top-up, not time) → escalate to the human orchestrator: report every marked provider's id/markedAt/resetAt and stop. Top-up or onboarding a new provider is a human decision, not another re-wake.

Step 3 — re-wake with a fact-baseline prompt. Build the payload from a file (jq --rawfile, SKILL.md pitfall #7), same wake shape as lifecycle-ops.md but with the quota facts. The template also restates the completion-report protocol ([org-internal #2374]) so the recovery is self-contained: it assumes the original seed still contains the protocol block — if you are unsure it does (e.g. the worker was dispatched before [org-internal #3669] added the quota clause), the template's own block below carries it; fill <BASE>/<LAUNCHER_SID>/<DIR> with the orchestrator session's values:

cat > wake.txt <<'EOF'
## 事实基线（配额墙恢复）
你上一回合被供应商配额错误杀死，世界观停留在被杀时刻。先读以下事实再行动：
1. 发生了什么：<HH:mm> 你的回合因 <provider> 配额耗尽终止（usage limit，
   预计重置 <HH:mm 或 未知>）；本消息由已知良好供应商重新唤醒，非你自行恢复。
2. 已做什么：<已完成步骤/commit/PR 状态>；worktree 未提交变更：
   <git status --short 摘要>——先 re-validate 这些文件，勿重做已完成工作。
3. 从哪继续：<一件具体的下一步任务>。
4. 证据规则：逐条到引用来源（issue 评论 / PR / 工件）核实后再行动，
   勿仅凭本基线行动。
## 完成回报协议（重申 [org-internal #2374]/[org-internal #3669]；不确定原始 seed 是否含协议块时以本节为准）
完成、阻塞或移交时，必须回报编排会话（你的唤醒者）：
  POST <BASE>/session/<LAUNCHER_SID>/prompt_async?directory=<DIR>
  body: {"agent":"builder","model":{...},"parts":[{"type":"text","text":"<report>"}]}
<report> 单行格式：[#<ticket>] [<node-or-slug>] status=done|blocked|handoff pr=#<N> summary=<一行>
约束：若再遇配额/用量上限错误，立即回报 status=blocked
reason=quota-exhausted summary=<provider 与重置提示>（pr=#<N> 有 PR 时带上）
后停止，勿自行换模型重试；回报失败 → 源 issue 评论兜底。
EOF
jq -n --rawfile p wake.txt '{agent:"builder",
  model:{providerID:"<known-good providerID>",modelID:"<known-good modelID>"},
  parts:[{type:"text",text:$p}]}' \
| curl -s -X POST "$BASE/session/$SID/prompt_async?directory=$DIR" \
  -H 'content-type: application/json' -d @- -o /dev/null -w "%{http_code}\n"   # → 204

Then poll the tail (limit=1) until an assistant message appears, and keep expecting the completion-report per protocol — the re-woken worker owes you status=done|blocked like any other (the restated block in the template makes this hold even when the original seed predates [org-internal #3669]).

Prevention — before dispatch, and in the seed

Pre-dispatch provider check (30 seconds, catches most walls):

# 1) no active quota markers on the dispatch provider (else pick another)
curl -s "$BASE/provider?directory=$DIR" | jq '[.all[] | select(.quota) | .id]'
# 2) dispatch provider is connected AND ≥1 other connected provider exists as fallback
curl -s "$BASE/provider?directory=$DIR" | jq '{connected, fallbacks: (.connected | length > 1)}'

If the intended provider is already quota-marked, dispatch on a different one — do not launch into a wall you can see coming.

Quota self-report clause (turns a silent death into a harvestable blocked). The canonical clause text lives in completion-report.md §"Orchestrator side" (配额自报, [org-internal #3669]) — append it VERBATIM to the seed prompt's completion-report block. Single source of truth: do not fork or restate the clause here; its blocked payload already follows the one-line key=value spec (status=blocked reason=quota-exhausted summary=<provider + reset hint> pr=#N, completion-report.md <report> format).

With the clause in the seed, a worker that hits the wall reports blocked instead of dying silently — the orchestrator harvests the report and runs the Recovery section directly. The clause is also why the re-wake prompt's constraint (Step 3) repeats it: the re-woken worker must know the rule still holds on the new provider.