forked from Octopus/octopus-workflow
v0.1.5: standalone-readability pass — delink --strip provenance mode ([TT-NNNN]), Tier-2 dangling refs neutralized, verify/review-code giant skills split into reference/ on-demand files; guards now runtime-neutral (bun|node)
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
> Added from headless-session-ops/SKILL.md (triggers `quota wall` / `配额墙`,
|
||||
> Post-launch lifecycle section, pitfall #17, References) — new runbook
|
||||
> 2026-08-29, ticket [org-internal #3669] (evidence instance [org-internal #3627]). Companion runbooks:
|
||||
> 2026-08-29, ticket [TT-3669] (evidence instance [TT-3627]). Companion runbooks:
|
||||
> `lifecycle-ops.md` (hung rescue, fact-baseline structure) and
|
||||
> `completion-report.md` §"Orchestrator side" (the quota self-report clause);
|
||||
> §Prevention is this file's own closing section.
|
||||
@@ -14,15 +14,15 @@ it looks EXACTLY like silent completion: no completion-report, no
|
||||
equals the orchestrator's polling interval, and without this runbook the
|
||||
recovery was improvised.
|
||||
|
||||
**Evidence instance ([org-internal #3627], 2026-08-29)** — the shapes below are field-verified:
|
||||
**Evidence instance ([TT-3627], 2026-08-29)** — the shapes below are field-verified:
|
||||
|
||||
|时刻 (+08)| 事件 |
|
||||
|---|---|
|
||||
| 08-29 13:18 | review-code Round-1 synthesis (FAIL) posted on [org-internal #3627]; worker session `ses_fb46e705bffe9kYbSoHayoJ1OQ` (builder, opencode-go model, branch `workflow/session/3627-wedge-promptasync`) entered its revision loop |
|
||||
| 08-29 13:18 | review-code Round-1 synthesis (FAIL) posted on [TT-3627]; worker session `ses_fb46e705bffe9kYbSoHayoJ1OQ` (builder, opencode-go model, branch `workflow/session/3627-wedge-promptasync`) entered its revision loop |
|
||||
| ~13:18–16:00 | worker hit the provider's `weekly usage limit reached` mid-revision — turn killed, **no** completion-report, **no** blocked report; issue side silent |
|
||||
| detection | orchestrator noticed the `/session` state stall, then confirmed via the quota error in the session log (message tail carries the ⚠️ usage-limit part, [org-internal #2912]) |
|
||||
| detection | orchestrator noticed the `/session` state stall, then confirmed via the quota error in the session log (message tail carries the ⚠️ usage-limit part, [TT-2912]) |
|
||||
| recovery | re-wake via `prompt_async` on a known-good provider (`zhipuai-coding-plan`/glm-5.3 — since renamed `zai-coding-plan`) **plus a fact-baseline injection** (what happened / what is done / where to resume) |
|
||||
| 16:26 / 16:51 | Round-2 synthesis PASS; `status=done pr=[org-internal #3675]` report received |
|
||||
| 16:26 / 16:51 | Round-2 synthesis PASS; `status=done pr=[TT-3675]` report received |
|
||||
|
||||
## Detection — three signals, in this order
|
||||
|
||||
@@ -35,7 +35,7 @@ SID=<worker session id>
|
||||
```
|
||||
|
||||
**Signal 1 — tail probe (primary; survives restarts, lives in the DB).** Since
|
||||
[org-internal #2912]/[org-internal #3190] a quota-killed turn persists an assistant message with
|
||||
[TT-2912]/[TT-3190] a quota-killed turn persists an assistant message with
|
||||
`finish=error` plus a human-readable ⚠️ text part whose metadata names the
|
||||
quota family. Probe the TAIL (`limit=1`, never the no-`limit` full transcript):
|
||||
|
||||
@@ -52,10 +52,10 @@ rate_limit}` (metadata also carries `provider`). The ⚠️ text is one of — a
|
||||
strings observed in this repo's runtime (`session/retry.ts`):
|
||||
|
||||
- `` `weekly usage limit` reached. It will reset in … `` / `Go limit reached`
|
||||
(GoUsageLimitError — the [org-internal #3627] shape; `weekly` is the body's `limitName`)
|
||||
(GoUsageLimitError — the [TT-3627] shape; `weekly` is the body's `limitName`)
|
||||
- `Free limit reached` (FreeUsageLimitError)
|
||||
- `Account-level usage limit — switching API keys does not reset it. You can
|
||||
continue after HH:mm.` ([org-internal #3407], zhipu/zai code 1308 — account-level)
|
||||
continue after HH:mm.` ([TT-3407], zhipu/zai code 1308 — account-level)
|
||||
- `Provider <id> rate limited (HTTP 429)` (generic 429)
|
||||
- `Generation failed — …` with a 402 `insufficient_quota` body (balance, not
|
||||
window — recovery is top-up, not provider switch)
|
||||
@@ -73,9 +73,9 @@ curl -s "$BASE/session?directory=$DIR" | jq -r --arg s "$SID" \
|
||||
30–60 s apart → the 429 retry spin is still running (it retries with backoff
|
||||
until the schedule ends) OR the stream is hung — treat exactly like the hung
|
||||
shape in `lifecycle-ops.md`: abort FIRST, then re-wake. A single snapshot is
|
||||
never an abort basis ([org-internal #3215]).
|
||||
never an abort basis ([TT-3215]).
|
||||
|
||||
**Signal 3 — provider-level confirmation ([org-internal #2911] quota markers).** `GET
|
||||
**Signal 3 — provider-level confirmation ([TT-2911] quota markers).** `GET
|
||||
/provider` overlays active usage-limit cooldowns, so the dead provider is
|
||||
visible without reading logs:
|
||||
|
||||
@@ -86,7 +86,7 @@ date -d @$(($(curl -s "$BASE/provider?directory=$DIR" | jq '[.all[].quota.resetA
|
||||
```
|
||||
|
||||
**Log grep (fallback; the dev log is restart-truncated).** Patterns actually
|
||||
seen in the wild — [org-internal #3627] produced the first:
|
||||
seen in the wild — [TT-3627] produced the first:
|
||||
|
||||
```sh
|
||||
grep -E "usage limit reached|Go limit reached|Free limit reached|Account-level usage limit|GoUsageLimitError|FreeUsageLimitError|rate limited \(HTTP 429\)|insufficient_quota" \
|
||||
@@ -96,7 +96,7 @@ grep -E "usage limit reached|Go limit reached|Free limit reached|Account-level u
|
||||
```
|
||||
|
||||
The wall is only terminal after failover has exhausted the same-name pools
|
||||
([org-internal #2911] failover); the `retrying … on …` WARNs above tell you the pool was
|
||||
([TT-2911] failover); the `retrying … on …` WARNs above tell you the pool was
|
||||
already draining before it died.
|
||||
|
||||
## Recovery — abort (only if needed), re-wake on a known-good provider, inject facts
|
||||
@@ -130,7 +130,7 @@ curl -s "$BASE/provider?directory=$DIR" | jq -r \
|
||||
# own current model — it is generating by definition).
|
||||
```
|
||||
|
||||
([org-internal #3627]: opencode-go dead → re-woken on zhipuai-coding-plan/glm-5.3, since
|
||||
([TT-3627]: opencode-go dead → re-woken on zhipuai-coding-plan/glm-5.3, since
|
||||
renamed `zai-coding-plan`. Do NOT "wait for reset" as the default — weekly
|
||||
windows mean days.)
|
||||
|
||||
@@ -159,9 +159,9 @@ curl -s "$BASE/provider?directory=$DIR" | jq -r \
|
||||
**Step 3 — re-wake with a fact-baseline prompt.** Build the payload from a file
|
||||
(`jq --rawfile`, SKILL.md pitfall #7), same wake shape as
|
||||
`lifecycle-ops.md` but with the quota facts. The template also restates the
|
||||
completion-report protocol ([org-internal #2374]) so the recovery is self-contained: it
|
||||
completion-report protocol ([TT-2374]) so the recovery is self-contained: it
|
||||
assumes the original seed still contains the protocol block — if you are
|
||||
unsure it does (e.g. the worker was dispatched before [org-internal #3669] added the quota
|
||||
unsure it does (e.g. the worker was dispatched before [TT-3669] added the quota
|
||||
clause), the template's own block below carries it; fill
|
||||
`<BASE>`/`<LAUNCHER_SID>`/`<DIR>` with the orchestrator session's values:
|
||||
|
||||
@@ -176,7 +176,7 @@ cat > wake.txt <<'EOF'
|
||||
3. 从哪继续：<一件具体的下一步任务>。
|
||||
4. 证据规则：逐条到引用来源（issue 评论 / PR / 工件）核实后再行动，
|
||||
勿仅凭本基线行动。
|
||||
## 完成回报协议（重申 [org-internal #2374]/[org-internal #3669]；不确定原始 seed 是否含协议块时以本节为准）
|
||||
## 完成回报协议（重申 [TT-2374]/[TT-3669]；不确定原始 seed 是否含协议块时以本节为准）
|
||||
完成、阻塞或移交时，必须回报编排会话（你的唤醒者）：
|
||||
POST <BASE>/session/<LAUNCHER_SID>/prompt_async?directory=<DIR>
|
||||
body: {"agent":"builder","model":{...},"parts":[{"type":"text","text":"<report>"}]}
|
||||
@@ -195,7 +195,7 @@ jq -n --rawfile p wake.txt '{agent:"builder",
|
||||
Then poll the tail (`limit=1`) until an assistant message appears, and keep
|
||||
expecting the completion-report per protocol — the re-woken worker owes you
|
||||
`status=done|blocked` like any other (the restated block in the template
|
||||
makes this hold even when the original seed predates [org-internal #3669]).
|
||||
makes this hold even when the original seed predates [TT-3669]).
|
||||
|
||||
## Prevention — before dispatch, and in the seed
|
||||
|
||||
@@ -213,7 +213,7 @@ If the intended provider is already quota-marked, dispatch on a different one
|
||||
|
||||
**Quota self-report clause (turns a silent death into a harvestable
|
||||
blocked).** The canonical clause text lives in `completion-report.md`
|
||||
§"Orchestrator side" (配额自报, [org-internal #3669]) — append it VERBATIM to the seed
|
||||
§"Orchestrator side" (配额自报, [TT-3669]) — append it VERBATIM to the seed
|
||||
prompt's completion-report block. Single source of truth: do not fork or
|
||||
restate the clause here; its blocked payload already follows the one-line
|
||||
key=value spec (`status=blocked reason=quota-exhausted summary=<provider +
|
||||
|
||||
Reference in New Issue
Block a user