Continue generation ("say more") affordance #12

New issue

Open

opened 2026-08-02 21:13:46 -04:00 by lyssieth · 0 comments

lyssieth commented

2026-08-02 21:13:46 -04:00

Owner

Problem

The user can stop a live stream and branch (regenerate_assistant), but cannot ask the model to continue a finished reply that ended with finish_reason: "length" (token cap) or otherwise feels cut short. regenerate_assistant branches a sibling assistant message under the same user message (crates/sermones-core/src/llm/conversation.rs:640), so it does not satisfy "say more" — the user's mental model is the same prompt, the same reply, with more appended.

Re-grounded 2026-08-03. The original design here was written against the TurnBody / Exchange model and is obsolete: f2216c6 refactor: conversation model to two-level sibling structure replaced it, and a60fa8c feat: multi-provider config… fixed the request-options gap this issue called its hard part. What follows is the design against the tree as it stands. The parts that changed are called out inline.

What changed underneath this issue

No Exchange, no TurnBody, no path_idx. The tree is now flat and strictly alternating: UserMessage and AssistantMessage nodes with parent / children (conversation.rs:124-149, :194-233). A user message's children are assistant replies; an assistant message's children are follow-up user messages. So "append a second assistant message to the same turn" is not expressible — a continuation must extend the existing AssistantMessage.content with more Block::Output. That is simpler than the old plan, not harder.
The request-options problem is solved. The old text said read_stream hardcodes ChatOptions::default(). It no longer does: read_stream(client, opts, context, cancel, tx) takes options (crates/sermones/src/streaming.rs:307,321,341), and begin_turn_stream resolves them per request from the selected model via chat_options_from_model (crates/sermones/src/bridge.rs:589,1065 → crates/sermones-runtime/src/providers/llama.rs:263-268). The only surviving ChatOptions::default() calls are in tests (streaming.rs:1036,1139).
Model attribution is already frozen per message. AssistantMessage.model: Option<ModelKey> is the immutable record of what produced this message, distinct from the conversation's mutable selected_model (conversation.rs:220-233, :455). A continuation must re-use the message's own model, not the conversation's current selection, or the second leg comes from a different model than the first.
max_tokens / temperature therefore need no new storage — they are derivable from the frozen ModelKey. The one knob still missing for a faithful continuation is seed: absent from both ChatOptions (llama.rs:241-249) and the wire ChatRequest (llama.rs:274-281).

The actual crux now

Two things fight the naive implementation:

1. context_messages drops the message being continued. It skips any assistant whose status == Streaming — "it is the response, not history" (crates/sermones-runtime/src/session/mod.rs:562-567). But a continuation's whole point is that the partial reply is in the prompt. Marking the target Streaming and then building context silently loses the text the model is meant to extend. Options, cheapest first:

Build the context before re-marking the target (begin_turn_stream calls conversations.context(...) at bridge.rs:568 — it would need the flag, or the ordering, threaded through).
Add an AssistantStatus::Continuing and teach context_messages to emit it. Costs a variant on a persisted enum, so it costs a config/conversation migration — check docs/components/conversation-migration.md before choosing this.

2. Chunks carry the whole fragment, not a delta. stream_chunk sends the fully re-rendered bubble and the frontend replaces wholesale (assistant.html = payload.html, frontend/src/lib/conversations.svelte.ts:365-371; compose_fragment at streaming.rs:188-196). If the continuation's reader starts from an empty assembler, the bubble visibly resets to just the new text mid-stream and only recovers on the terminal. So the reader must be seeded with the target message's existing blocks, and compose_fragment must render prior + new output as one fragment.

Design

1. IPC surface

ContinueAssistantParams { conversation_id: String, assistant_id: String } in crates/sermones-core/src/ipc.rs, mirroring RegenerateAssistantParams (ipc.rs:154-161); registered in IPC_TYPES. Codegen emits the TS type, schema, and hash.
New continue_assistant slot in crates/sermones/src/bridge.rs next to regenerate_assistant (bridge.rs:448). It resolves the existing assistant_id rather than branching a sibling, then runs begin_turn_stream against it.
TurnStreamResult needs no change — { sub_id, assistant_message_id } already says everything, and the id is the button's own message.

2. Engine

Conversation::extend_assistant_content(assistant_id, Vec<Block>) next to set_assistant_content (conversation.rs:774) — appends instead of replacing.
ConversationStore::continue_stream(id, assistant_id, message, model) next to finish_stream (session/mod.rs:407-431): same status/finish_reason/usage handling, but extend_assistant_content instead of set_assistant_content, and it must not overwrite model with a fresh snapshot.
TurnOutcome::Continued(Box<Message>) alongside Finished / PartialFinished / Errored, routed in settle_turn (bridge.rs:648-668). PartialFinished needs a continuation twin too — cancelling a continuation mid-flight must keep both legs, and abandon_with_content currently replaces (session/mod.rs:485-498).
The subscription carries the mode. Subscription/StreamTarget (bridge.rs:616-628) is where the continue: bool goes, set at allocation; the forwarder picks the terminal from it.
seed on ChatOptions and ChatRequest (llama.rs), dropped from the wire body when None (llama.cpp/gemma servers do not take it, and must tolerate its absence).

3. Frontend

bridge.ts: continueAssistant(bridge, { conversation_id, assistant_id }, handler) mirroring regenerateAssistant (bridge.ts:251-257).
conversations.svelte.ts: continue(assistantId) mirroring regenerate (:183). #onChunk and #onEnded need no change — the sub mapping already keys sub_id → (conversation, assistant message) and the message id is unchanged.
MessageView.svelte: a new button in the assistant toolbar (:350-380, e.g. MessageSquarePlus from lucide-svelte, which is already the icon source at :29-35), gated on !isStreaming (:158) and only when the message is the leaf of the current path — you cannot continue a message you have navigated away from. Wire ChatView.svelte → App.svelte alongside onRegenerate (App.svelte:207).
conversations.test.ts: terminal-routing + chunk-routing coverage for the continue path, including "the bubble does not lose its existing html on the first continuation chunk".

4. Not gating on `finish_reason`

Tempting: show the button only when the provider cut the reply — the label already exists (MessageView.svelte:164-171, non-stop reasons only, :423-424). Rejected for v1: a user can reasonably want Continue on a stop reply ("now go on"), and the affordance speaks for itself. Available on any Done message; gate later if usage says otherwise.

Files expected to change

crates/sermones-core/src/ipc.rs — ContinueAssistantParams, registration
crates/sermones-core/src/llm/conversation.rs — extend_assistant_content
crates/sermones-runtime/src/providers/llama.rs — seed on ChatOptions + ChatRequest (optional on the wire)
crates/sermones-runtime/src/session/mod.rs — continue_stream, and the context_messages fix above
crates/sermones/src/bridge.rs — continue_assistant slot, TurnOutcome::Continued, subscription mode, seeded reader
crates/sermones/src/streaming.rs — assembler seeded from existing blocks so compose_fragment emits prior + new
frontend/src/lib/bridge.ts, conversations.svelte.ts, MessageView.svelte, App.svelte, ChatView.svelte
frontend/src/lib/conversations.test.ts

Open questions

~~Where does RequestOptions live?~~ Resolved. Options resolve from the frozen AssistantMessage.model through chat_options_from_model; no per-turn options struct is needed. Only seed is genuinely missing, and it belongs on ModelConfig / ChatOptions like the other knobs.
~~How does the projection render multiple assistant exchanges in one turn?~~ Resolved by the tree refactor. A continuation extends one AssistantMessage, so AssistantMessageView keeps its single text / html (ipc.rs:242-257) and the projection is untouched.
Where does the continuation boundary live, if anywhere? Two Block::Output runs concatenate into one paragraph flow. That is right for a mid-sentence length cut and wrong for "now do it again, longer". Cheapest answer: no marker in the data, no separator in the render — revisit if it reads badly.
AssistantStatus::Continuing or context-before-mark? See the crux above; the migration cost is the deciding factor.

Out of scope

Tool-call continuations — RequestMessage still does not carry tool_call_id (llama.rs:199-205). Separate issue.
A continue affordance on user-initiated branching.
Per-request option overrides in the UI (#9 / #10 own the settings surface; this issue reads what is configured, it does not add controls).

#9, #10 — settings; where seed would become user-visible
docs/architecture/streaming.md — cancellation / partial-text semantics the continuation twin must respect

## Problem The user can stop a live stream and branch (`regenerate_assistant`), but cannot ask the model to *continue* a finished reply that ended with `finish_reason: "length"` (token cap) or otherwise feels cut short. `regenerate_assistant` branches a **sibling** assistant message under the same user message (`crates/sermones-core/src/llm/conversation.rs:640`), so it does not satisfy "say more" — the user's mental model is the same prompt, the same reply, with more appended. > **Re-grounded 2026-08-03.** The original design here was written against the `TurnBody` / `Exchange` model and is obsolete: `f2216c6 refactor: conversation model to two-level sibling structure` replaced it, and `a60fa8c feat: multi-provider config…` fixed the request-options gap this issue called its hard part. What follows is the design against the tree as it stands. The parts that changed are called out inline. ## What changed underneath this issue - **No `Exchange`, no `TurnBody`, no `path_idx`.** The tree is now flat and strictly alternating: `UserMessage` and `AssistantMessage` nodes with `parent` / `children` (`conversation.rs:124-149`, `:194-233`). A user message's children are assistant replies; **an assistant message's children are follow-up user messages**. So "append a second assistant message to the same turn" is not expressible — a continuation must **extend the existing `AssistantMessage.content` with more `Block::Output`**. That is simpler than the old plan, not harder. - **The request-options problem is solved.** The old text said `read_stream` hardcodes `ChatOptions::default()`. It no longer does: `read_stream(client, opts, context, cancel, tx)` takes options (`crates/sermones/src/streaming.rs:307,321,341`), and `begin_turn_stream` resolves them per request from the selected model via `chat_options_from_model` (`crates/sermones/src/bridge.rs:589,1065` → `crates/sermones-runtime/src/providers/llama.rs:263-268`). The only surviving `ChatOptions::default()` calls are in tests (`streaming.rs:1036,1139`). - **Model attribution is already frozen per message.** `AssistantMessage.model: Option<ModelKey>` is the immutable record of what produced *this* message, distinct from the conversation's mutable `selected_model` (`conversation.rs:220-233`, `:455`). A continuation must re-use the message's own `model`, not the conversation's current selection, or the second leg comes from a different model than the first. - **`max_tokens` / `temperature` therefore need no new storage** — they are derivable from the frozen `ModelKey`. The one knob still missing for a faithful continuation is **`seed`**: absent from both `ChatOptions` (`llama.rs:241-249`) and the wire `ChatRequest` (`llama.rs:274-281`). ## The actual crux now Two things fight the naive implementation: **1. `context_messages` drops the message being continued.** It skips any assistant whose `status == Streaming` — "it is the *response*, not history" (`crates/sermones-runtime/src/session/mod.rs:562-567`). But a continuation's whole point is that the partial reply *is* in the prompt. Marking the target `Streaming` and then building context silently loses the text the model is meant to extend. Options, cheapest first: - Build the context **before** re-marking the target (`begin_turn_stream` calls `conversations.context(...)` at `bridge.rs:568` — it would need the flag, or the ordering, threaded through). - Add an `AssistantStatus::Continuing` and teach `context_messages` to emit it. Costs a variant on a persisted enum, so it costs a config/conversation migration — check `docs/components/conversation-migration.md` before choosing this. **2. Chunks carry the whole fragment, not a delta.** `stream_chunk` sends the fully re-rendered bubble and the frontend replaces wholesale (`assistant.html = payload.html`, `frontend/src/lib/conversations.svelte.ts:365-371`; `compose_fragment` at `streaming.rs:188-196`). If the continuation's reader starts from an empty assembler, the bubble **visibly resets** to just the new text mid-stream and only recovers on the terminal. So the reader must be seeded with the target message's existing blocks, and `compose_fragment` must render prior + new output as one fragment. ## Design ### 1. IPC surface - `ContinueAssistantParams { conversation_id: String, assistant_id: String }` in `crates/sermones-core/src/ipc.rs`, mirroring `RegenerateAssistantParams` (`ipc.rs:154-161`); registered in `IPC_TYPES`. Codegen emits the TS type, schema, and hash. - New `continue_assistant` slot in `crates/sermones/src/bridge.rs` next to `regenerate_assistant` (`bridge.rs:448`). It resolves the *existing* `assistant_id` rather than branching a sibling, then runs `begin_turn_stream` against it. - `TurnStreamResult` needs **no change** — `{ sub_id, assistant_message_id }` already says everything, and the id is the button's own message. ### 2. Engine - `Conversation::extend_assistant_content(assistant_id, Vec<Block>)` next to `set_assistant_content` (`conversation.rs:774`) — appends instead of replacing. - `ConversationStore::continue_stream(id, assistant_id, message, model)` next to `finish_stream` (`session/mod.rs:407-431`): same status/finish_reason/usage handling, but `extend_assistant_content` instead of `set_assistant_content`, and it must **not** overwrite `model` with a fresh snapshot. - `TurnOutcome::Continued(Box<Message>)` alongside `Finished` / `PartialFinished` / `Errored`, routed in `settle_turn` (`bridge.rs:648-668`). `PartialFinished` needs a continuation twin too — cancelling a continuation mid-flight must keep both legs, and `abandon_with_content` currently replaces (`session/mod.rs:485-498`). - The subscription carries the mode. `Subscription`/`StreamTarget` (`bridge.rs:616-628`) is where the `continue: bool` goes, set at allocation; the forwarder picks the terminal from it. - `seed` on `ChatOptions` and `ChatRequest` (`llama.rs`), dropped from the wire body when `None` (llama.cpp/gemma servers do not take it, and must tolerate its absence). ### 3. Frontend - `bridge.ts`: `continueAssistant(bridge, { conversation_id, assistant_id }, handler)` mirroring `regenerateAssistant` (`bridge.ts:251-257`). - `conversations.svelte.ts`: `continue(assistantId)` mirroring `regenerate` (`:183`). `#onChunk` and `#onEnded` need no change — the sub mapping already keys `sub_id → (conversation, assistant message)` and the message id is unchanged. - `MessageView.svelte`: a new button in the assistant toolbar (`:350-380`, e.g. `MessageSquarePlus` from lucide-svelte, which is already the icon source at `:29-35`), gated on `!isStreaming` (`:158`) and only when the message is the leaf of the current path — you cannot continue a message you have navigated away from. Wire `ChatView.svelte` → `App.svelte` alongside `onRegenerate` (`App.svelte:207`). - `conversations.test.ts`: terminal-routing + chunk-routing coverage for the continue path, including "the bubble does not lose its existing html on the first continuation chunk". ### 4. Not gating on `finish_reason` Tempting: show the button only when the provider cut the reply — the label already exists (`MessageView.svelte:164-171`, non-`stop` reasons only, `:423-424`). Rejected for v1: a user can reasonably want Continue on a `stop` reply ("now go on"), and the affordance speaks for itself. Available on any `Done` message; gate later if usage says otherwise. ## Files expected to change - `crates/sermones-core/src/ipc.rs` — `ContinueAssistantParams`, registration - `crates/sermones-core/src/llm/conversation.rs` — `extend_assistant_content` - `crates/sermones-runtime/src/providers/llama.rs` — `seed` on `ChatOptions` + `ChatRequest` (optional on the wire) - `crates/sermones-runtime/src/session/mod.rs` — `continue_stream`, and the `context_messages` fix above - `crates/sermones/src/bridge.rs` — `continue_assistant` slot, `TurnOutcome::Continued`, subscription mode, seeded reader - `crates/sermones/src/streaming.rs` — assembler seeded from existing blocks so `compose_fragment` emits prior + new - `frontend/src/lib/bridge.ts`, `conversations.svelte.ts`, `MessageView.svelte`, `App.svelte`, `ChatView.svelte` - `frontend/src/lib/conversations.test.ts` ## Open questions 1. ~~Where does `RequestOptions` live?~~ **Resolved.** Options resolve from the frozen `AssistantMessage.model` through `chat_options_from_model`; no per-turn options struct is needed. Only `seed` is genuinely missing, and it belongs on `ModelConfig` / `ChatOptions` like the other knobs. 2. ~~How does the projection render multiple assistant exchanges in one turn?~~ **Resolved by the tree refactor.** A continuation extends one `AssistantMessage`, so `AssistantMessageView` keeps its single `text` / `html` (`ipc.rs:242-257`) and the projection is untouched. 3. **Where does the continuation boundary live, if anywhere?** Two `Block::Output` runs concatenate into one paragraph flow. That is right for a mid-sentence `length` cut and wrong for "now do it again, longer". Cheapest answer: no marker in the data, no separator in the render — revisit if it reads badly. 4. **`AssistantStatus::Continuing` or context-before-mark?** See the crux above; the migration cost is the deciding factor. ## Out of scope - Tool-call continuations — `RequestMessage` still does not carry `tool_call_id` (`llama.rs:199-205`). Separate issue. - A `continue` affordance on user-initiated branching. - Per-request option overrides in the UI (#9 / #10 own the settings surface; this issue reads what is configured, it does not add controls). ## Related - #9, #10 — settings; where `seed` would become user-visible - `docs/architecture/streaming.md` — cancellation / partial-text semantics the continuation twin must respect

lyssieth self-assigned this

2026-08-02 23:33:35 -04:00

lyssieth added this to the v1 milestone

2026-08-02 23:33:38 -04:00

lyssieth added the

Kind/Feature

Priority

Medium

labels

2026-08-02 23:33:49 -04:00