Operating Hypnos

Rendered from docs/operating.md; edits go there.

For the person who runs Hypnos rather than builds it: how to size a machine for a lineup, which knob to turn for which symptom, how to read what the server tells you, what each error means, and how to keep it running. The README covers installation and the API; backend/hypnos.example.toml documents every option.

1. Sizing a machine#

Hypnos books memory before it loads anything, so the question is whether the lineup's bookings fit the budgets, not whether the machine "feels" big enough.

Tier 1 (VRAM) holds, per resident model, its weights plus the KV cache and compute workspace:

vram(model) = total_weight_bytes
            + slots x KV(default_context_len)      # chat models; KV per token from [models.arch]
            + arch.activation_workspace_bytes

POST /v1/memory/status reports the booking of every resident model (placement), what the device really has free (tier1_vram_real_free_bytes) and what the planner did not book (tier1_vram_unaccounted_bytes: driver, display, other processes). As a rule of thumb for the default lineup on a 24 GB GPU: the 26B chat model is 13.6 GiB of weights plus about 0.9 GiB of KV per slot at 16k context, so slots = 4 fills the card and leaves the embedder (0.3 GiB) and Kokoro (0.2 GiB) pinned; MOSS-Image (9.3 GiB) and the 26B cannot be resident together and evict each other.

Tier 2 (pinned RAM) holds the weight files of models staged there (initial_tier = "tier2_pinned_ram", or evicted with keep_warm_in_ram = true). Since PERF-6 the lock covers every file of the model (weights, projector, text encoder), so pinned_ram_budget_bytes must cover the sum of total_weight_bytes of the models you want warm at once, and that memory is gone for the OS and for the Hypnos process itself (CUDA staging buffers, docling, GLiNER). Keep the budget under half of physical RAM; the server warns at startup when it is not. On a 64 GB machine with the default lineup, 24 GiB for the two chat models (with projectors) is comfortable; 32 GiB with the image model kept warm as well was not (docs/bench/swap-cuda-2026-10-30.md).

Tier 3 (NVMe) is the file. Promotion from Tier 3 reads it through the OS page cache: 4.6 s for the 26B on this laptop when the cache is warm, disk speed when it is not.

The bookings are estimates, not limits the OS enforces. hypnos measure-memory hypnos.toml [--models a,b] --out report.md loads each model alone, runs one request of its kind and reports booked against real memory per model (docs/bench/memory-2026-11-01.md, Apple Silicon); run it on a new machine or after adding a model, and raise activation_workspace_bytes where real is above booked. At runtime /metrics has hypnos_memory_bytes{kind="booked"|"footprint"|"held"}: alert when held stays well above booked.

Three startup sanity checks: hypnos --check-config hypnos.toml validates the file without starting; the first lines of the log print the detected device, the budgets and every model's booking; /v1/memory/status right after /ready shows the pre-staged models in their tiers.

Choosing a lineup: hypnos advise --needs chat,voice,documents (this machine; or --vram 24G --ram 64G, --unified --ram 32G) recommends catalog models, variants, context, slots, tiers and priorities that fit, says what stays resident together and what will swap, and --config prints a config file for it (POST /v1/advisor from a client). It sizes with the same planner as the scheduler, so its numbers are bookings, not measurements.

Adding a model from the catalog (backend/models.catalog.toml, GET /v1/models/catalog): ./run.sh models <id>[:<quant>] downloads a variant's files into the models folder, then POST /v1/models {"catalog": "<id>[:<quant>]"} adds it to the running server (no restart; its first request loads it), PUT /v1/models/{id} swaps a model for another variant or another model under the same name (requests wait for the swap, running ones finish first) and DELETE takes one out. These changes are kept in <config name>.models.toml next to the config and applied at the next start; delete that file to go back to the config alone. With API keys set they need an admin key. Size the pool with the formula above (a block's total_weight_bytes is the variant's files); hypnos --catalog-block <id>[:<quant>] prints a block for the config file instead. The catalog only lists architectures Hypnos has been tested with: a GGUF that merely loads in llama.cpp can still get its prompt format, media markers or memory handling wrong. The route reports outdated for a file whose size differs from the catalog (an upload replaced upstream) and problem when downloaded weights report another architecture.

Workload profiles. [server] profile sets the knobs for an outcome instead of one by one. It fills in only what the config leaves out (a [[models]] block's priority, initial_tier, keep_warm_in_ram, slots, [models.arch] default_context_len, and [limits] max_queue_per_model), so explicit keys always win and a block that should follow the profile leaves them out; hypnos --check-config logs what the profile set. Models added at runtime get the same presets.

A model with several roles takes the most demanding one (the E4B speech model under voice_assistant is never evicted and gets one chat slot). /v1/memory/status reports the profile and its objective.

2. Which knob for which symptom#

The limits in [limits] (request_timeout_secs, max_body_bytes, max_prompt_chars, max_tokens_cap, max_images_per_request, allow_remote_image_urls) are safety rails, not tuning: raise them when a legitimate request hits one, and keep allow_remote_image_urls = false on a server that faces the internet (it makes the server fetch URLs that clients send).

3. Reading the telemetry#

Every response carries hypnos_telemetry:

/metrics (Prometheus text) has the same per model as histograms (hypnos_chat_ttft_ms, hypnos_chat_queue_ms, hypnos_chat_context_ms, hypnos_chat_prefill_tokens_per_s, hypnos_chat_decode_tokens_per_s, hypnos_tts_rtf, hypnos_tts_ttfa_ms, hypnos_stt_rtf, hypnos_image_jct_s, hypnos_document_ms_per_page, each with a _last gauge), the scheduler counters (hypnos_scheduler_promotions_total, _evictions_total, _load_failures_total, _admissions_rejected_total), the memory gauges (hypnos_vram_bytes{kind}, hypnos_pinned_ram_bytes{kind}, hypnos_model_tier, hypnos_model_gpu_layers, hypnos_model_active_leases), per-route HTTP counters and latency, the GPU health gauges (hypnos_gpu_utilization_percent, _temperature_celsius, _power_watts, _sm_clock_mhz, _throttled), the prefix-cache counters and the per-key counters.

Alerts worth having: hypnos_model_load_failures_total{model} increasing (an engine refused its weights: file gone or changed, CUDA out of memory; the per-model series says which), changes(hypnos_process_start_time_seconds[1h]) > 0 (the process restarted: an engine abort took it down and the supervisor brought it back), hypnos_gpu_throttled == 1 for minutes, /ready not 200 for longer than your pre-staging takes, hypnos_http_requests_total{class="5xx"} rate.

The request log (GET /v1/logs, the playground's "Request Logs" tab) keeps the last server.request_log_capacity calls with route, status, model, key, tokens, tier switch and timings; ?key=, ?model=, ?route=, ?status=error filter it.

4. Errors#

All errors come in the OpenAI shape, {"error": {"message", "type", ...}}, so OpenAI SDKs raise their usual exceptions.

5. Logs, levels and redaction#

• server.log_format = "json" (or HYPNOS_LOG_FORMAT=json) for log shippers; pretty for a terminal. Startup prints every config warning and error, the device and the budgets.
• The port is server.port (9999 by default), overridden by HYPNOS_PORT and, above that, by --port N on the command line, so one config file can serve several instances.
• Model files: relative weights_path, aux_weights_path, text_encoder_path, tokenizer_path and voices_path resolve against server.models_dir (itself relative to the working directory; the example configs set "models" with bare file names, the layout of models.manifest), overridden by HYPNOS_MODELS_DIR, e.g. to keep 31 GB on another drive. Without the key they resolve against the working directory, as configs written before it do. Startup logs the folder in use (Model files models_dir=…).
• Data folders (espeak-ng-data, frontend, models.catalog.toml) are looked for beside the binary, then in ../Resources of a macOS app bundle, then in HYPNOS_RESOURCE_DIR (the catalog in its server/ subfolder), which the desktop app sets for the server it carries on Windows and Linux; server.frontend_dir names the playground folder outright.
• server.request_log_level: info keeps metadata only and is the right level for anything exposed beyond a trusted group; debug stores prompts and completions; trace stores payloads with media replaced by placeholders. POST /v1/logs/level changes it at runtime.
• server.request_log_redact: regular expressions whose matches become [redacted] in stored text, or ["all"] to keep no text at any level.
• Rejections are logged with route, reason and the key's label, never the key.

6. Exposure#

Bind 127.0.0.1 unless keys and TLS (or a terminating proxy) are in place. With keys, every /v1/* route needs a bearer key; /health and /ready stay open, /metrics too unless server.metrics_require_auth = true. Per-key limits are the only multi-tenancy Hypnos has: for quotas per user or organisation, billing, or finer rules put an API gateway in front.

Remote image and document URLs (allow_remote_image_urls) never reach loopback, private, link-local or other non-public addresses, redirects included, unless limits.allow_private_urls = true: a client cannot make the server read the LAN or a cloud metadata endpoint. A server bound beyond loopback without API keys logs a warning at startup (anyone who reaches it can also change the model pool). See docs/security-review-2026-11-01.md.

7. Upgrading and rolling back#

• Release archives and the Docker image are versioned; hypnos --version prints the build.
• hypnos.toml carries config_version. A newer Hypnos migrates an older file in memory and logs each step; a file written for a newer Hypnos is refused with both versions named, so a rollback to an older binary needs the older config (keep the previous hypnos.toml next to the previous archive). Unknown keys are reported as warnings, not ignored.
• backend/models.manifest carries manifest-version; the fetch scripts refuse a newer layout.
• Run hypnos --check-config hypnos.toml with the new binary before switching traffic.
• Model files are addressed by path in the config: a new release that changes the lineup needs scripts/fetch-models for the new files; old files can stay.
• backend/crates/hypnos-core/tests/compat.rs loads every released example config (fixtures in tests/fixtures/, add the new release's when tagging) and checks that files written for a newer Hypnos are refused, not misread.
• models.catalog.toml carries catalog_version (the server ignores a newer one and its catalog route answers 404). The runtime overlay <config name>.models.toml is applied over the config at startup and checked with it by --check-config; when rolling back, or to return to the config alone, move it away. Back it up with the config: it holds the models added at runtime.

8. Running as a service#

Docker (CUDA):

docker run --gpus all -p 9999:9999 -v ./models:/opt/hypnos/models \
  -e HYPNOS_API_KEYS=... --restart unless-stopped ghcr.io/advancedlogic/hypnos:0.7.2

Mount your own hypnos.toml over /opt/hypnos/hypnos.toml or point HYPNOS_CONFIG at a mounted file. The models volume is /opt/hypnos/models (models_dir = "models" from the working directory); mount it elsewhere and set HYPNOS_MODELS_DIR to match. The image has a HEALTHCHECK on /health with a 10-minute start period for the pre-staging. --build-arg DOCLING_CUDA=1 adds the CUDA 13 runtime for docling on the GPU.

systemd (Linux archive unpacked in /opt/hypnos):

[Unit]
Description=Hypnos inference server
After=network-online.target

[Service]
WorkingDirectory=/opt/hypnos
ExecStart=/opt/hypnos/hypnos /opt/hypnos/hypnos.toml
Environment=HYPNOS_API_KEYS=...
Environment=HYPNOS_LOG_FORMAT=json
Restart=on-failure
RestartSec=5
LimitMEMLOCK=infinity
KillSignal=SIGTERM
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

LimitMEMLOCK matters: Tier 2 uses mlock, and the default limit is small (64 KiB to 8 MiB depending on the distribution). SIGTERM triggers the graceful shutdown (in-flight requests finish, engines are released).

Windows: the archive runs from its folder (hypnos.exe reads hypnos.toml); to run it as a service use a wrapper such as NSSM (nssm install hypnos C:\hypnos\hypnos.exe hypnos.toml, set the startup directory to the archive folder and HYPNOS_API_KEYS in its environment) or a Task Scheduler task that starts at boot. The process raises its own working-set limit for Tier 2.

Supervision: an engine abort inside llama.cpp or stable-diffusion.cpp (a C++ abort) takes the process down; the restart policy above is what brings it back, and /ready tells the load balancer when it is serving again. A load failure that the engine reports as an error (missing file, CUDA out of memory) does not: the booking is rolled back and the request answers 500.

9. Before going live#

1. hypnos --check-config is clean, and the startup log shows the budgets you expect.
2. backend/scripts/smoke_test.sh http://host:9999 passes once per modality.
3. backend/scripts/bench.sh against the box gives the numbers you will compare against later (docs/bench/ holds the reference runs).
4. Keys set, bound beyond loopback only behind TLS, request log at info, redaction on if the level is ever raised.
5. Alerts on /ready, load failures, 5xx rate and GPU throttling.