Provider configuration from the UI #10

New issue

Open

opened 2026-08-01 22:10:42 -04:00 by lyssieth · 0 comments

lyssieth commented

2026-08-01 22:10:42 -04:00

Owner

Status: complete. The shape changed twice underneath this issue — the multi-provider cutover (a60fa8c) replaced the single provider.* triple with Config.providers / Config.models maps, and the settings screen (63dd0b4) landed the CRUD over both. The last two items closed in the connection-test pass.

Checklist

Fields for the provider record (kind, base_url, api_key) and the model record (model_id_at_provider, display_name, context_window, max_output_tokens, temperature), bound to config via the get_config / set_config IPC from #3 — frontend/src/lib/Settings.svelte (add/remove providers and models, default-model select), wired through App.svelte
~~Env-var overrides shown as such: SERMONES_BASE_URL / SERMONES_MODEL win during the dev loop — the UI says why a field is currently ineffective instead of silently losing~~ — obsolete. Those env vars were retired; config is file-or-default with no env override (docs/components/runtime.md). Nothing can shadow a UI-set field, so there is nothing to explain. SERMONES_DATA_DIR still exists but is a persistence path, not a provider knob.
Connection-test affordance; failures surface via the error store (#4)
Sane empty state: no provider configured yet → guidance, not an error — landed with 63dd0b4; Settings.svelte renders first-run copy in both the Providers and Models sections (what a slug is, what the <slug>|<model_id> key shape is)
Engine-side config validation, so the panel is no longer the only enforcement point

What landed for the connection test

Probe — providers::llama::probe_provider(&ProviderRecord) -> Result<Option<u32>, SermonError> (crates/sermones-runtime/src/providers/llama.rs). 10s timeout; the cheapest request that exercises the whole path a chat request takes — DNS, TLS, the base URL's prefix, the Authorization header — while generating nothing and spending no tokens.

The endpoint is per kind, and that is the whole subtlety. GET /models looks universal, and for Llama and OpenAI it is. OpenRouter's /api/v1/models is a public catalogue — verified live: it answers 200 with all ~338 models to an anonymous request and to a garbage Bearer token. Probing it reported "connected" for a key that cannot spend a cent, on the one kind where the credential is billable. OpenRouter is therefore probed at GET /key, which 401s on both. Hence Option<u32>: /key says nothing about models, so a success there carries no count — not a weaker answer, the stronger one for that kind.

Status mapping: 401/403 → ProviderAuth, 429 → ProviderRateLimited, any other non-2xx and every transport failure → ProviderUnavailable, a /models body that is not a model list → ProviderProtocol. The last one is deliberate: a 200 that does not parse means the URL reached something that is not an OpenAI-compatible API — often a base_url missing its /v1 — and reporting "connected" there would cost the user exactly the debugging session this button exists to save. A /key body is deliberately not parsed: the 200-versus-401 carries the whole answer, and demanding a shape that cannot be exercised without a live key would risk failing the one case that matters most.

What a success does not prove. A server that ignores the Authorization header answers the same for any key or none — a bare llama-server without --api-key does exactly this. That is the server's semantics, not a gap: there is no credential to be wrong about. Same reason a Llama provider whose base_url omits /v1 still passes (llama.cpp aliases both); the same omission against OpenAI 404s.

Live verification (2026-08-04) after the per-kind fix: local llama.cpp → OK, 4 models; OpenRouter with no key and with a junk key → ProviderAuth; OpenAI with a junk key → ProviderAuth.

IPC — TestProviderParams { provider: ProviderRecord }, TestProviderResult { test_id }, ProviderTestPayload { test_id, ok, model_count, detail }, all registered in IPC_TYPES. The record travels inline rather than by slug so an unsaved edit can be tested, which is the case that matters. ProviderTestPayload is flat rather than an enum for the facet-json-schema reason in docs/architecture/ipc-pipeline.md; it mirrors StreamEndedPayload.

Bridge — a test_provider slot plus a provider_test_finished signal (crates/sermones/src/bridge.rs). The slot validates the record and answers a bad one synchronously as ConfigInvalid with nothing sent over the network; the probe runs on the tokio runtime and its outcome arrives on the signal, correlated by test_id from its own counter (not the subscription registry's — a probe has no cancellation and nothing to clean up, so minting from the same space would blur what a sub_id means).

Frontend — testProvider / onProviderTestFinished in lib/bridge.ts; App.svelte folds the dispatch and the signal back into one promise per call, so Settings.svelte only awaits an outcome. The Test connection button lives in each provider's open editor — before Save, since the question is whether these credentials work and answering it after committing them is too late. Results are per-row and transient, superseded by a per-row attempt counter so a slow probe cannot land on top of a later answer.

On error surfacing. The failure rides provider_test_finished rather than error_raised — it has a requesting call to answer, the same split as stream_ended's own detail — but the shell files every failure in the error store on arrival, per error-visibility.md ("No silent swallow"). The inline result beside the row is additional: the answer goes where the user is looking, the record goes where records go.

What landed for validation

ProviderRecord::validate and Config::validate in crates/sermones-core/src/config.rs, called by the set_config slot before anything is mutated (so a refused write leaves the running app on the config it already had) and by test_provider before it dials.

The rules mirror the panel's validateDraft: provider keys that parse as a Slug, a non-empty base_url, an api_key wherever ProviderKind::requires_api_key (blank counts as absent), model keys that parse and name a provider that exists, a non-empty wire name, and an active_model that is listed. The key checks matter most: Slug / ModelId / ModelKey are #[facet(transparent)], so their constructors never run on the way in and a | in a slug round-trips through both the file and the IPC boundary unremarked, only biting later when a ModelKey built from it splits in the wrong place.

Deliberately not run by the loader. A file that fails these rules still loads: falling back to Config::default() would discard the user's whole config over one bad row, and orphan model references in particular are designed to be tolerated at request time by the resolver. Validation is for the write path, where the user is present to be told.

setConfig in lib/bridge.ts grew an onRejected callback for the synchronous refusal — it was previously fire-and-forget and would have dropped it. The panel pre-validates the same rules and shows the problem inline, so in practice the engine's copy fires only for a config the UI did not author.

Tests

crates/sermones-core/src/config.rs — 9 validate cases, including the transparent-newtype key built by deserialising a config with lo|cal as a provider key.
crates/sermones-runtime/src/providers/llama.rs — 11 probe_provider cases against mockito: model count, trailing slash on the base_url, the Bearer header, each status mapping, an HTML 200, a refused connection, plus three pinning the per-kind endpoint (OpenRouter hits /key and not /models, a 401 there is ProviderAuth, a 200 there succeeds whatever its body). The model-list fixture is real llama-server output — its nested status object with an args array is what pins facet-json skipping nested unknown structures, which a flat fixture would have missed.
frontend/src/lib/providerTest.test.ts — 13 cases over the projection: the record travelling inline (and surviving TypeBox validation), an absent api_key omitted rather than nulled, the correlation id, a refused record, drift folding, both signal outcomes, foreign envelopes ignored on the shared channel, disconnect-by-identity, and set_config's rejection path.

Docs updated in lockstep: docs/components/bridge-qobject.md (slot + signal + the validation placement), docs/components/runtime.md (the probe, and why validation is not a load-time step), docs/components/frontend-ui.md (the connection-test UX and the shifted enforcement point).

Refs

frontend/src/lib/Settings.svelte — providers / models / default-model sections
frontend/src/App.svelte — onSettingsSave, onSettingsRestore, onTestProvider
crates/sermones-core/src/config.rs — Config, ProviderRecord, ModelConfig, lookup_model, validate
crates/sermones/src/bridge.rs — get_config / set_config / test_provider slots
docs/components/runtime.md — config dir resolution, retired env overrides, validation placement
#3 — configuration loader (landed, closed)
#4 — error store (landed, closed)
#9 — settings screen (its home; the panel exists)
docs/designs/draft/frontend.md — settings scaffold

Settings → provider section: make the provider configurable from the UI instead of env vars. All three blockers have landed: the config loader (#3), the settings screen (#9, panel exists), and the error store (#4). **Status:** complete. The shape changed twice underneath this issue — the multi-provider cutover (`a60fa8c`) replaced the single `provider.*` triple with `Config.providers` / `Config.models` maps, and the settings screen (`63dd0b4`) landed the CRUD over both. The last two items closed in the connection-test pass. ## Checklist - [x] Fields for the provider record (`kind`, `base_url`, `api_key`) and the model record (`model_id_at_provider`, `display_name`, `context_window`, `max_output_tokens`, `temperature`), bound to config via the `get_config` / `set_config` IPC from #3 — `frontend/src/lib/Settings.svelte` (add/remove providers and models, default-model select), wired through `App.svelte` - [x] ~~Env-var overrides shown as such: `SERMONES_BASE_URL` / `SERMONES_MODEL` win during the dev loop — the UI says why a field is currently ineffective instead of silently losing~~ — **obsolete.** Those env vars were retired; config is file-or-default with no env override (`docs/components/runtime.md`). Nothing can shadow a UI-set field, so there is nothing to explain. `SERMONES_DATA_DIR` still exists but is a persistence path, not a provider knob. - [x] Connection-test affordance; failures surface via the error store (#4) - [x] Sane empty state: no provider configured yet → guidance, not an error — landed with `63dd0b4`; `Settings.svelte` renders first-run copy in both the Providers and Models sections (what a slug is, what the `<slug>|<model_id>` key shape is) - [x] Engine-side config validation, so the panel is no longer the only enforcement point ## What landed for the connection test **Probe** — `providers::llama::probe_provider(&ProviderRecord) -> Result<Option<u32>, SermonError>` (`crates/sermones-runtime/src/providers/llama.rs`). 10s timeout; the cheapest request that exercises the whole path a chat request takes — DNS, TLS, the base URL's prefix, the `Authorization` header — while generating nothing and spending no tokens. **The endpoint is per kind, and that is the whole subtlety.** `GET /models` looks universal, and for `Llama` and `OpenAI` it is. **OpenRouter's `/api/v1/models` is a public catalogue** — verified live: it answers 200 with all ~338 models to an anonymous request *and* to a garbage Bearer token. Probing it reported "connected" for a key that cannot spend a cent, on the one kind where the credential is billable. OpenRouter is therefore probed at `GET /key`, which 401s on both. Hence `Option<u32>`: `/key` says nothing about models, so a success there carries no count — not a weaker answer, the stronger one for that kind. Status mapping: 401/403 → `ProviderAuth`, 429 → `ProviderRateLimited`, any other non-2xx and every transport failure → `ProviderUnavailable`, a `/models` body that is not a model list → `ProviderProtocol`. The last one is deliberate: a 200 that does not parse means the URL reached something that is not an OpenAI-compatible API — often a `base_url` missing its `/v1` — and reporting "connected" there would cost the user exactly the debugging session this button exists to save. A `/key` body is deliberately *not* parsed: the 200-versus-401 carries the whole answer, and demanding a shape that cannot be exercised without a live key would risk failing the one case that matters most. **What a success does not prove.** A server that ignores the `Authorization` header answers the same for any key or none — a bare `llama-server` without `--api-key` does exactly this. That is the server's semantics, not a gap: there is no credential to be wrong about. Same reason a `Llama` provider whose `base_url` omits `/v1` still passes (llama.cpp aliases both); the same omission against OpenAI 404s. Live verification (2026-08-04) after the per-kind fix: local llama.cpp → `OK, 4 models`; OpenRouter with no key and with a junk key → `ProviderAuth`; OpenAI with a junk key → `ProviderAuth`. **IPC** — `TestProviderParams { provider: ProviderRecord }`, `TestProviderResult { test_id }`, `ProviderTestPayload { test_id, ok, model_count, detail }`, all registered in `IPC_TYPES`. The record travels **inline** rather than by slug so an unsaved edit can be tested, which is the case that matters. `ProviderTestPayload` is flat rather than an enum for the `facet-json-schema` reason in `docs/architecture/ipc-pipeline.md`; it mirrors `StreamEndedPayload`. **Bridge** — a `test_provider` slot plus a `provider_test_finished` signal (`crates/sermones/src/bridge.rs`). The slot validates the record and answers a bad one synchronously as `ConfigInvalid` with nothing sent over the network; the probe runs on the tokio runtime and its outcome arrives on the signal, correlated by `test_id` from its own counter (not the subscription registry's — a probe has no cancellation and nothing to clean up, so minting from the same space would blur what a `sub_id` means). **Frontend** — `testProvider` / `onProviderTestFinished` in `lib/bridge.ts`; `App.svelte` folds the dispatch and the signal back into one promise per call, so `Settings.svelte` only awaits an outcome. The Test connection button lives in each provider's open editor — before Save, since the question is whether these credentials work and answering it after committing them is too late. Results are per-row and transient, superseded by a per-row attempt counter so a slow probe cannot land on top of a later answer. **On error surfacing.** The failure rides `provider_test_finished` rather than `error_raised` — it has a requesting call to answer, the same split as `stream_ended`'s own `detail` — but the shell files every failure in the error store on arrival, per `error-visibility.md` ("No silent swallow"). The inline result beside the row is *additional*: the answer goes where the user is looking, the record goes where records go. ## What landed for validation `ProviderRecord::validate` and `Config::validate` in `crates/sermones-core/src/config.rs`, called by the `set_config` slot before anything is mutated (so a refused write leaves the running app on the config it already had) and by `test_provider` before it dials. The rules mirror the panel's `validateDraft`: provider keys that parse as a `Slug`, a non-empty `base_url`, an `api_key` wherever `ProviderKind::requires_api_key` (blank counts as absent), model keys that parse and name a provider that exists, a non-empty wire name, and an `active_model` that is listed. The key checks matter most: `Slug` / `ModelId` / `ModelKey` are `#[facet(transparent)]`, so their constructors never run on the way in and a `|` in a slug round-trips through both the file and the IPC boundary unremarked, only biting later when a `ModelKey` built from it splits in the wrong place. Deliberately **not** run by the loader. A file that fails these rules still loads: falling back to `Config::default()` would discard the user's whole config over one bad row, and orphan model references in particular are designed to be tolerated at request time by the resolver. Validation is for the write path, where the user is present to be told. `setConfig` in `lib/bridge.ts` grew an `onRejected` callback for the synchronous refusal — it was previously fire-and-forget and would have dropped it. The panel pre-validates the same rules and shows the problem inline, so in practice the engine's copy fires only for a config the UI did not author. ## Tests - `crates/sermones-core/src/config.rs` — 9 `validate` cases, including the transparent-newtype key built by deserialising a config with `lo|cal` as a provider key. - `crates/sermones-runtime/src/providers/llama.rs` — 11 `probe_provider` cases against `mockito`: model count, trailing slash on the `base_url`, the Bearer header, each status mapping, an HTML 200, a refused connection, plus three pinning the per-kind endpoint (OpenRouter hits `/key` and *not* `/models`, a 401 there is `ProviderAuth`, a 200 there succeeds whatever its body). The model-list fixture is real `llama-server` output — its nested `status` object with an args array is what pins `facet-json` skipping nested unknown structures, which a flat fixture would have missed. - `frontend/src/lib/providerTest.test.ts` — 13 cases over the projection: the record travelling inline (and surviving TypeBox validation), an absent `api_key` omitted rather than nulled, the correlation id, a refused record, drift folding, both signal outcomes, foreign envelopes ignored on the shared channel, disconnect-by-identity, and `set_config`'s rejection path. Docs updated in lockstep: `docs/components/bridge-qobject.md` (slot + signal + the validation placement), `docs/components/runtime.md` (the probe, and why validation is not a load-time step), `docs/components/frontend-ui.md` (the connection-test UX and the shifted enforcement point). ## Refs - `frontend/src/lib/Settings.svelte` — providers / models / default-model sections - `frontend/src/App.svelte` — `onSettingsSave`, `onSettingsRestore`, `onTestProvider` - `crates/sermones-core/src/config.rs` — `Config`, `ProviderRecord`, `ModelConfig`, `lookup_model`, `validate` - `crates/sermones/src/bridge.rs` — `get_config` / `set_config` / `test_provider` slots - `docs/components/runtime.md` — config dir resolution, retired env overrides, validation placement - ~~#3~~ — configuration loader (landed, closed) - ~~#4~~ — error store (landed, closed) - #9 — settings screen (its home; the panel exists) - `docs/designs/draft/frontend.md` — settings scaffold

lyssieth added a new dependency

2026-08-01 22:12:01 -04:00

#9 Settings screen

lyssieth added the

Kind/Feature

Priority

High

labels

2026-08-01 22:12:03 -04:00

lyssieth added a new dependency

2026-08-01 22:12:06 -04:00

#3 Add configuration loader for sermones-runtime