feat(core,server,web): images reach every input — steering, goal objectives, one fold (#71)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
rank-Yu
@@ -87,7 +87,19 @@ Carry-over enters the model context only — it is never written to the Trace, w
|
||||
|
||||
## Mid-run steering
|
||||
|
||||
While a Task is running, the host can queue a user message with `session.steer(text)` without interrupting the loop: at the next input assembly the engine delivers it as a **standalone user text message** wrapped in `[user_steering]…[/user_steering]`, sent alongside that turn's tool outputs (or alone as the continuation input when the turn produced no tool calls — the Task keeps going instead of ending). Steering is real user input: written to Trace like any Prompt, yielded to the output stream, and replayed as ordinary turn input on resume; tool outputs are never rewritten. The queue is drained at **every** input assembly — including right after a mid-run compaction, so steering that arrives during the compaction request is delivered, never swallowed. `steer` returns `false` when no Task is running (hosts then submit a normal task); the queue is discarded only when the run exits (abort included).
|
||||
While a Task is running, the host can queue a user message with `session.steer(input)` — an OmniMessage list, the same shape `run` takes a Prompt in — without interrupting the loop: at the next input assembly the engine delivers it as a **standalone user text message** wrapped in `[user_steering]…[/user_steering]`, sent alongside that turn's tool outputs (or alone as the continuation input when the turn produced no tool calls — the Task keeps going instead of ending). The input's user text becomes the block's body; its images follow it as ordinary user image messages, so an image with no caption is a complete steering message; on a model without vision they fold into `[attached image: <path>]` lines **inside** the block instead, exactly as a Prompt's images do (the block must stay the whole text, or the message would lose its steering identity and read as a new Task). Steering is real user input: written to Trace like any Prompt, yielded to the output stream, and replayed as ordinary turn input on resume; tool outputs are never rewritten. The queue is drained at **every** input assembly — including right after a mid-run compaction, so steering that arrives during the compaction request is delivered, never swallowed. `steer` returns `false` when no Task is running (hosts then submit a normal task); the queue is discarded only when the run exits (abort included).
|
||||
|
||||
## Input images
|
||||
|
||||
An input image either rides the request as an image message or becomes an `[attached image: <path>]` line pointing at a file in the session scratchpad — the model then views it with `read_image` / `describe_image`, and the Web restores the thumbnail from the path. The conversion is one function bound once per Session (it is the only layer that knows both the scratchpad and the model's capability), and **each input path decides for itself whether to apply it**:
|
||||
|
||||
| Input | Folds when | Applied at |
|
||||
| --- | --- | --- |
|
||||
| Prompt (`run`) | the model has no vision | run entry, before Trace and title material |
|
||||
| Steering (`steer`) | the model has no vision | delivery, at the turn boundary — queuing must stay synchronous, and a queue discarded on abort would otherwise leave orphan files |
|
||||
| Goal objective | **always** | before the objective is extracted, so the path lines survive every round's re-injection |
|
||||
|
||||
Goal mode is the exception because its objective is re-injected as text every round: see [Goal mode](/docs/goal-mode).
|
||||
|
||||
## Automatic reconnect
|
||||
|
||||
|
||||
@@ -84,7 +84,19 @@ Task 由若干连续的 Request(轮)组成，每轮：
|
||||
|
||||
## 运行中插话(Steering)
|
||||
|
||||
Task 运行期间，宿主可通过 `session.steer(text)` 排队一条用户消息而不打断循环：引擎在下一次输入组装时把它作为**独立的用户文本消息**送出，内容包裹在 `[user_steering]…[/user_steering]` 中，与该轮工具输出一起进入下一次请求（该轮没有工具调用时则单独作为继续输入，Task 不会就此结束）。插话是真实的用户输入：像 Prompt 一样写入 Trace、推送到输出流，恢复重放时按普通轮次输入处理；工具输出本身从不被改写。队列在**每次**输入组装时排空——包括运行中压缩完成后的那次，压缩请求期间到达的插话不会被吞掉。没有 Task 运行时 `steer` 返回 `false`（宿主转为发起普通 Task）；仅在运行退出（含中断）时丢弃队列。
|
||||
Task 运行期间，宿主可通过 `session.steer(input)` 排队一条用户消息而不打断循环（`input` 是 OmniMessage 列表，与 `run` 接收 Prompt 的形状一致）：引擎在下一次输入组装时把它作为**独立的用户文本消息**送出，内容包裹在 `[user_steering]…[/user_steering]` 中，与该轮工具输出一起进入下一次请求（该轮没有工具调用时则单独作为继续输入，Task 不会就此结束）。输入中的用户文本成为标记块的正文，图片紧跟其后，作为普通用户图片消息送出，因此一张没有配文的图片本身就是一条完整的插话；模型不支持视觉时，图片改为折叠成 `[attached image: <path>]` 路径行写在标记块**内部**，与 Prompt 的图片走同一条路（标记块必须仍是整条文本，否则这条消息会丢掉插话身份、被当成新 Task）。插话是真实的用户输入：像 Prompt 一样写入 Trace、推送到输出流，恢复重放时按普通轮次输入处理；工具输出本身从不被改写。队列在**每次**输入组装时排空——包括运行中压缩完成后的那次，压缩请求期间到达的插话不会被吞掉。没有 Task 运行时 `steer` 返回 `false`（宿主转为发起普通 Task）；仅在运行退出（含中断）时丢弃队列。
|
||||
|
||||
## 输入图片
|
||||
|
||||
一张输入图片要么以图片消息的形态跟着请求走，要么变成一行 `[attached image: <路径>]`，指向会话 scratchpad 里的文件——模型再用 `read_image` / `describe_image` 去看，Web 则从路径还原出缩略图。这个转换是每个 Session 绑定一次的同一个函数（Session 是唯一同时知道 scratchpad 目录和模型能力的层），而**是否折叠由各输入路径自己决定**：
|
||||
|
||||
| 输入 | 何时折叠 | 折叠时机 |
|
||||
| --- | --- | --- |
|
||||
| Prompt(`run`) | 模型不支持图片 | `run` 入口，早于写 Trace 和取标题素材 |
|
||||
| 插话(`steer`) | 模型不支持图片 | 投递时，即 turn 边界——入队必须保持同步，而中断时被丢弃的队列若已折叠会留下没人读的孤儿文件 |
|
||||
| 目标(goal) | **总是** | 抽取 objective 之前，这样路径行才能活过每一轮的重新注入 |
|
||||
|
||||
目标模式是唯一的例外，因为它的目标每轮都作为文本重新注入：见[目标模式](/docs/goal-mode)。
|
||||
|
||||
## 自动重连
|
||||
|
||||
|
||||
@@ -44,6 +44,12 @@ Each round's user message is a `[goal]` protocol block followed by a plain body
|
||||
- `blocked` → the loop stops; what the model needs from you is in its final reply. The injected rules require the **same blocking condition to persist for three consecutive rounds** before the model may claim `blocked`, so a transient obstacle doesn't end the goal.
|
||||
- `active` → budget permitting, the next round fires.
|
||||
|
||||
### Images in an objective
|
||||
|
||||
An objective may carry attached images — "make the page match this mockup" is a goal, and a screenshot states it better than a paragraph. They are always saved to the session scratchpad and referenced from the objective as `[attached image: <path>]` lines, **whatever the model's vision**: the objective is re-injected as the text of every round's block, so an image cannot ride along as an image. Sending it in round 1 alone would leave every later round pointing at something compaction has since removed, while the objective still reads correct. As a path it survives every round and every compaction, and the model spends tokens on it only when it actually looks (`read_image`, or `describe_image` without vision). An image cannot stand in for the text — a picture alone states no objective, so a text-less goal input is rejected.
|
||||
|
||||
The chat page shows the attachments in full under round 1's bubble and collapses them into a one-line chip on later rounds (click to expand): they are part of every round's input, but a twenty-round goal shouldn't repeat the same picture twenty times.
|
||||
|
||||
A round that ends in an abort (user stop, LLM failure) ends the whole goal without re-firing — on-disk state stays `active`, so the workspace and goal file remain a clean resume point. In the Web App the regular stop button aborts the entire loop; in the CLI, Ctrl-C does. The same applies to a round the engine cut off at the per-Task turn cap (`max_turns`): the model never got to write the goal file, so the loop ends as `aborted` instead of re-firing the same cutoff forever.
|
||||
|
||||
## Token budget
|
||||
|
||||
@@ -44,6 +44,12 @@ status: active
|
||||
- `blocked` → 循环停止；模型缺什么写在它最后一条回复里。注入规则要求**同一阻塞条件持续三个连续轮次**后才允许声明 `blocked`，临时性障碍不会终结目标。
|
||||
- `active` → 预算允许则进入下一轮。
|
||||
|
||||
### 目标里的图片
|
||||
|
||||
目标可以附带图片——「把页面改成这张设计稿的样子」本身就是一个目标，而一张截图比一段描述说得清楚。图片一律写入会话 scratchpad，在目标文本里以 `[attached image: <路径>]` 行引用，**与模型是否支持视觉无关**：目标每轮都作为协议块的文本被重新注入，图片没法以图片的形态跟着走。只在第一轮发，后续每一轮的目标就指向了一个早已被压缩掉的东西，而目标文本读起来却毫无破绽。作为路径，它跨越每一轮、每一次压缩都稳定存在，而模型只在真的要看的时候才付出 token（有视觉用 `read_image`，没有则 `describe_image`）。图片不能替代文字——一张图说明不了目标，所以没有文字的目标输入会被拒绝。
|
||||
|
||||
聊天页在第一轮气泡下方完整展示附图，后续轮次收成一行 chip（点击展开）：它确实在每一轮的输入里，但二十轮的目标不该把同一张图重复二十次。
|
||||
|
||||
某一轮以中断结束（用户停止、LLM 故障）时整个目标随之结束、不再续推——磁盘上的状态保持 `active`，工作区与目标文件就是干净的断点。Web App 中常规停止按钮即中止整个循环；CLI 中是 Ctrl-C。被单 Task 轮次上限（`max_turns`）掐断的轮同理：模型没来得及写目标文件，循环以 `aborted` 结束，而不是永远重演同一次掐断。
|
||||
|
||||
## Token 预算
|
||||
|
||||
@@ -161,8 +161,8 @@ The paths below omit the `/api/sessions/:sessionId` prefix. For the storage mode
|
||||
| DELETE | / | Delete the Session (along with its Traces and scratch files) |
|
||||
| GET | /messages | Full OmniMessage history; while a Task runs the response also carries `live` (the in-progress stream tail, see below) |
|
||||
| GET | /stream | SSE event stream (next section) |
|
||||
| POST | /tasks | Start a Task: `{input: TaskInputPart[], thinkingLevel?, queueIfBusy?}` → 202. With `queueIfBusy`, a busy session holds the input as a follow-up (`queued: true`) and auto-starts it as an ordinary next task once idle; `task_state` events report the queued count. `file` input parts are written to the Session scratchpad and handed to the model as `[attached file: <path>]` lines (see the request body below) |
|
||||
| POST | /steer | Mid-run steering: `{text}` queues a message for the running Task (delivered between turns as a standalone `[user_steering]` user message) → 202; 409 `not_running` when no Task is in progress |
|
||||
| POST | /tasks | Start a Task: `{input: TaskInputPart[], thinkingLevel?, queueIfBusy?}` → 202. With `queueIfBusy`, a busy session holds the input as a follow-up (`queued: true`) and auto-starts it as an ordinary next task once idle; `task_state` events report the queued count. `file` input parts are written to the Session scratchpad and handed to the model as `[attached file: <path>]` lines (see the request body below). With `goal: {budget?}` the input starts a goal loop instead: it must carry non-empty text (an image alone states no objective), any images it carries fold into the objective as scratchpad path lines whatever the model's vision, and `file` parts are refused — nothing folds them into a re-injected objective — see [Goal mode](/docs/goal-mode) |
|
||||
| POST | /steer | Mid-run steering: `{text, images?}` queues a message for the running Task (delivered between turns as a standalone `[user_steering]` user message, with its images right behind it) → 202; either field can carry the message on its own, but a request with neither is a 400; 409 `not_running` when no Task is in progress |
|
||||
| POST | /approvals/:toolCallId | Approval decision: `{decision}` is `allow` or `deny` → 204 |
|
||||
| POST | /abort | Interrupt the current Task: 202 when triggered, 204 when idle |
|
||||
| POST | /retry-now | "Retry now" on the reconnect countdown: skips the in-progress backoff wait, firing the next retry immediately (attempt counter unchanged) → 200 `{skipped}` — `skipped:false` is the benign "no wait in progress" case, never an error |
|
||||
|
||||
@@ -161,8 +161,8 @@ Trace 下载对任意成员开放；导入仅限 owner（同 Agent 快照导入
|
||||
| DELETE | / | 删除 Session（连同 Trace 与暂存文件） |
|
||||
| GET | /messages | 完整 OmniMessage 历史；Task 运行期间响应额外携带 `live`（进行中的流式尾部，见下） |
|
||||
| GET | /stream | SSE 事件流（见下节） |
|
||||
| POST | /tasks | 发起 Task：`{input: TaskInputPart[], thinkingLevel?, queueIfBusy?}` → 202。带 `queueIfBusy` 时，运行中的 Session 会把输入暂存为跟进消息（`queued: true`），空闲后按序自动作为普通 Task 发出；`task_state` 事件携带排队数。`file` 输入部分会写入该 Session 的 scratchpad，并以 `[attached file: <path>]` 行交给模型（见下方请求体） |
|
||||
| POST | /steer | 运行中插话：`{text}` 为运行中的 Task 排队一条消息（作为独立的 `[user_steering]` 用户消息随下一轮送达）→ 202；无 Task 运行返回 409 `not_running` |
|
||||
| POST | /tasks | 发起 Task：`{input: TaskInputPart[], thinkingLevel?, queueIfBusy?}` → 202。带 `queueIfBusy` 时，运行中的 Session 会把输入暂存为跟进消息（`queued: true`），空闲后按序自动作为普通 Task 发出；`task_state` 事件携带排队数。`file` 类型的输入会写入 Session scratchpad，以 `[attached file: <路径>]` 行交给模型（见下方请求体）。带 `goal: {budget?}` 时该输入转为发起目标循环：必须含非空文字（一张图说明不了目标），随行的图片一律折叠成 scratchpad 路径行写入目标文本、与模型是否支持视觉无关，而 `file` 会被拒绝——没有东西能把它折进每轮重注入的目标里——见[目标模式](/docs/goal-mode) |
|
||||
| POST | /steer | 运行中插话：`{text, images?}` 为运行中的 Task 排队一条消息（作为独立的 `[user_steering]` 用户消息随下一轮送达，图片紧随其后）→ 202；两个字段任一非空即可成消息，都为空则 400；无 Task 运行返回 409 `not_running` |
|
||||
| POST | /approvals/:toolCallId | 审批决定：`{decision}` 取 `allow` 或 `deny` → 204 |
|
||||
| POST | /abort | 中断当前 Task：已触发返回 202，无任务返回 204 |
|
||||
| POST | /retry-now | 重连倒计时上的「立即重试」：跳过进行中的退避等待、立刻发起下一次重试（重试计数不变）→ 200 `{skipped}`——`skipped:false` 表示当前没有等待可跳过（良性空操作，非错误） |
|
||||
|
||||
@@ -49,7 +49,8 @@ There are four approval modes: `allow-all`, `deny-all`, `read-only` (only read-o
|
||||
- Enter sends, Shift+Enter inserts a newline, and images can be pasted;
|
||||
- The "+" menu holds the input add-ons: **image upload**, **file attachment** and goal mode. An attachment can be any type (up to 20 at a time, ≤ 10MB each and 12MB in total; an oversize pick is refused before it is read, so nothing is uploaded to earn the rejection); selected files show as removable chips above the text body in the order they were picked, and a message with attachments and no text is sendable. On send the files are written into the Session's scratchpad — deleted with the Session — and the message gains an `[attached file: <path>]` line per file, which the conversation renders as an "Attached files" notice: the bytes never enter the conversation, the model opens each file by path with its ordinary file tools;
|
||||
- Typing `/` opens the slash menu: trigger context compaction (`/compact`), hand the conversation over to another Agent (`/agent`), switch the model (`/model`) — both switch commands appear in an active session only, since a draft has nothing to switch and picks its Agent and model up front — or toggle installed Skills — chosen Skills are sent along with the message in a `[use_skills]` block;
|
||||
- While a Task is running the input stays live and the toolbar keeps a single action button: an empty composer shows **Stop**, and typing turns it into **Send**, whose behavior follows the **mid-run send mode** from the toolbar's More-settings popover (a compact extensible settings panel, also available in draft state; the choice is remembered): **Steer** (default) delivers the text mid-run as a `[user_steering]` user message with the next turn, **Queue** holds the whole message server-side as a follow-up and auto-sends it as an ordinary new message when the run finishes (an "N queued" hint shows near the input until then; the queue survives page reloads);
|
||||
- While a Task is running the input stays live and the toolbar keeps a single action button: an empty composer shows **Stop**, and typing turns it into **Send**, whose behavior follows the **mid-run send mode** from the toolbar's More-settings popover (a compact extensible settings panel, also available in draft state; the choice is remembered): **Steer** (default) delivers the text **and any attached images** mid-run as a `[user_steering]` user message with the next turn (an image with no caption sends on its own, and the delivered message renders as a steering chip with its images inside it), **Queue** holds the whole message server-side as a follow-up and auto-sends it as an ordinary new message when the run finishes (an "N queued" hint shows near the input until then; the queue survives page reloads). A draft steering cannot carry — selected skills, file attachments, or a staged `/agent` or `/model` pick, with no text and no image — falls back to the queue for that one send, so the button never sits disabled with Stop displaced;
|
||||
- **Goal mode** (the "+" menu, or `/goal`) turns the draft into an objective the system loops Tasks against. Attached images ride along and are always sent as scratchpad paths (a goal objective is re-injected as text every round — see [Goal mode](/docs/goal-mode)); text is still required, since a picture alone states no objective, and file attachments are refused. Round 1's bubble shows the attachments in full, later rounds collapse them into a chip that expands on click;
|
||||
- `/agent` and `/model` stage their pick instead of acting on it: the chosen Agent or model becomes a chip above the text body and nothing is sent yet, so you keep typing — Enter/Send is what hands the conversation over (a new chat for that Agent) or forks it onto the chosen model, carrying the text along; with an empty composer a default message is filled in, and the chip's × cancels. Both chips are cached with the draft, so they survive a reload or a trip to another conversation together with the text. A model fork additionally waits for the Session to be idle — it continues from the Session's Trace, which a running turn or a compaction is still writing — and a line above the composer says so while it waits;
|
||||
- When human approval is required, tool calls show inline allow/deny buttons in the message stream; the approval mode can be changed mid-Session;
|
||||
- While the engine waits out a reconnect backoff (≥2s), the retry line shows a live countdown to the next attempt with inline **Retry now** (skips the remaining wait) and **Give up** (the ordinary abort) controls;
|
||||
|
||||
Reference in New Issue
Block a user