Hypnos HTTP API reference

Rendered from docs/api.md; edits go there.

This is the wire-level reference for anyone writing a client or an SDK against a Hypnos server. It describes every route, every request and response field, the streaming formats, the error shape and the behaviour a client has to handle (readiness, admission, quotas, cancellation). The operator's side (sizing, knobs, metrics) is in operating.md; the general introduction is the README.

Everything below is derived from the server source as of v0.1.0 (backend/crates/hypnos-server for the routes, backend/crates/hypnos-core/src/openai.rs for the schemas). Where the API is a Hypnos extension rather than OpenAI behaviour it is marked Hypnos extension.

1. Basics#

Base URL. http://<host>:<port>; the default port is 9999 (server.port, --port, HYPNOS_PORT). All API routes live under /v1. With [server.tls] configured the scheme is https. The playground, when present, is served at / and never interferes with /v1.

OpenAI compatibility. The chat, embeddings, transcription, speech, image and model-list routes accept the OpenAI request bodies and answer with the OpenAI response bodies, so the official OpenAI SDKs work unchanged with base_url = "http://host:9999/v1" and any non-empty API key (the key is checked only when the server has keys configured). Hypnos adds fields on top; it never removes an OpenAI field it supports. Unknown request fields are ignored, not rejected.

Content types. JSON bodies are application/json. Transcription and document conversion also accept multipart/form-data. Speech answers with audio bytes, streaming routes with text/event-stream (SSE) or a chunked audio body. Every JSON response is UTF-8.

Request id. Every response carries x-request-id: req-<32 hex>. Quote it when reporting a problem; the request log (section 15) is keyed by it.

Next stage. A request may carry x-hypnos-next-model: <id> (Hypnos extension): the model the client will call next, loaded in the background while this request runs, so the next call does not wait for it. A voice turn sends it on the transcription (naming the chat model) and on the chat (naming the speech model). Unknown ids are ignored; the header never changes the response.

Deadlines and estimated waits. x-hypnos-deadline-ms: <ms> (Hypnos extension) is the longest the request may wait before it starts: when the scheduler's estimate (requests ahead beyond the model's slots x its average service time, plus a load when the model is not resident) is longer, the request is refused at once with 503 deadline_unmeetable and a Retry-After, instead of being queued. The server can apply its workload profile's objective as an implicit deadline (limits.overload = "shed"). Every admitted request's response carries x-hypnos-estimated-wait-ms, the estimate made when it arrived (absent before the model has served anything while busy).

Workflows. x-hypnos-workflow: <name>/<id> marks the requests of one multi-step task (a voice turn: transcription, chat, speech); x-hypnos-workflow-end: true on the last one closes it (or 30 s without a request does). Their end-to-end time, first start to last byte, is reported per name as p50 / p95 / p99 in /v1/memory/status latency.workflows and as hypnos_workflow_e2e_ms on /metrics, next to the same per route (latency.routes, hypnos_request_e2e_ms).

CORS. Browsers may call the API from any loopback origin by default; the operator can list other origins (server.allowed_origins, http://host:* wildcards). The methods are GET, POST and OPTIONS (DELETE is not in the CORS allow-list, so release stored pictures from a non-browser client or same-origin). The telemetry headers of section 5 are exposed to browser scripts.

Body and time limits (defaults; the operator may change them in [limits]):

2. Authentication and per-key limits#

When the server has API keys configured (server.api_keys, [[server.keys]] or the HYPNOS_API_KEYS environment variable), every /v1/* request must carry

Authorization: Bearer <key>

The scheme is matched case-insensitively and surrounding whitespace is trimmed. /health, /ready and, unless the operator closes it, /metrics stay open. When no keys are configured the header is ignored and may be absent.

Changing the model pool (POST /v1/models, PUT / DELETE /v1/models/{id}, section 6) needs an admin key: every plain server.api_keys entry is one, a [[server.keys]] entry only with admin = true. Other keys get 403 permission_denied there.

401 (missing, malformed or unknown key) uses the OpenAI error shape, so the OpenAI SDKs raise their usual authentication error, and carries WWW-Authenticate: Bearer realm="hypnos":

{ "error": { "message": "Missing API key. Send it as `Authorization: Bearer <key>` (server.api_keys).",
             "type": "invalid_request_error", "param": null, "code": "invalid_api_key" } }

Per-key limits. Each key can have requests_per_minute, max_concurrent and tokens_per_day (UTC day, prompt plus completion tokens of chat, embeddings and the other token-counting routes; a chat stream is charged when it ends). A rejected call answers 429 with Retry-After: <seconds>:

{ "error": { "message": "...", "type": "rate_limit_error", "param": null,
             "code": "requests_per_minute" | "max_concurrent" | "tokens_per_day" } }

The concurrency permit is held for the whole response body, so a streaming chat or a long document conversion counts as one in-flight request until the client has read the last byte.

The key's label (from the config, or key-<8 hex> for unlabelled keys) is attached to the request log entry and to the hypnos_api_key_* metrics; the key itself is never logged.

3. Server lifecycle: health, readiness, 503#

The port opens before the startup models are staged, which can take minutes for a large model. During that window:

{ "error": { "message": "Hypnos is still pre-staging its startup models; retry shortly (GET /ready reports when it is done)",
             "type": "model_not_available" } }

A client that starts together with the server should poll /ready (or retry on this 503 with the Retry-After delay) instead of queueing requests. A 503 model_not_available can also come from a configuration problem with the requested model after startup; the message says which.

Once ready, the first request for a model that is not in GPU memory pays the promotion (seconds for a model warm in RAM, tens of seconds from disk); hypnos_telemetry.tier_switch_latency_ms (section 5) reports it. Clients should not time out below the operator's request_timeout_secs.

4. Errors#

All errors share one body:

{ "error": { "message": "<human readable>", "type": "<machine readable>" } }

Authentication and quota errors add "param": null and "code" (section 2).

On a streaming chat response that fails after the headers were sent, the error arrives as a final chunk with finish_reason: "error" (section 7.4); the HTTP status is already 200.

5. Telemetry on every response#

Hypnos extension. Every successful JSON inference response (and the last chunk of a chat stream) carries hypnos_telemetry:

"hypnos_telemetry": {
  "previous_tier": "tier2_pinned_ram",
  "placement": { "primary_tier": "tier1_gpu_vram", "gpu_layers": 30, "pinned_ram_layers": 0,
                 "ssd_layers": 0, "vram_weight_bytes": 15814641056,
                 "vram_kv_and_workspace_bytes": 5301600256, "pinned_ram_bytes": 0,
                 "allocated_context_len": 16384 },
  "tier_switch_latency_ms": 3662.1,
  "inference_latency_ms": 1240.5,
  "evicted_models": ["moss-image"],
  "engine": { "queue_ms": 0.1, "prefill_ms": 120.0, "decode_ms": 1100.0, "ttft_ms": 125.0,
              "prefill_tokens_per_s": 676.0, "decode_tokens_per_s": 163.7,
              "cached_tokens": 0, "concurrent_slots": 1 },
  "tier_switch_phases": { "load_mode": "read", "weights_ms": 2700.0, "projector_ms": 400.0,
                          "context_ms": 500.0, "total_ms": 3650.0 }
}

Speech responses are audio, so the same facts travel as headers: x-hypnos-previous-tier (spelled as in the JSON, tier2_pinned_ram), x-hypnos-gpu-layers, x-hypnos-switch-ms, x-hypnos-inference-ms, x-hypnos-audio-secs, x-hypnos-rtf. A streaming speech response carries the first three only (the rest is unknown when the headers go out).

A client may ignore all of this; an SDK should expose it as an optional typed object because it is the only way to tell a cold request (seconds of tier switch) from a slow one.

6. Models#

GET /v1/models#

Lists the models in service (disabled models are hidden). Open during startup staging.

{ "object": "list", "data": [ {
    "id": "gemma4:26b-q4", "object": "model", "created": 1793481600, "owned_by": "hypnos",
    "modality": "text_generation",
    "current_tier": "tier1_gpu_vram",
    "placement": { "...": "see appendix" },
    "active_leases": 0 } ] }

modality is the model's primary modality: text_generation, vision, speech_to_text, text_to_speech, embeddings, image_generation, document_conversion, entity_extraction. A model can serve more than its primary modality (the default gemma4:26b-q4 also does vision, gemma4:e4b-q4 does speech-to-text, text and vision); GET /v1/memory/status lists the full modalities array per model.

The default lineup (hypnos.example.toml):

Other profiles change the lineup: hypnos.lfm3.macos.example.toml serves Liquid AI's LFM3 models (lfm3:1.7b text, lfm3-vl:2b vision and text, lfm3-audio:2b speech-to-text and text) in place of the two Gemma chat models, and GET /v1/models/catalog lists every model Hypnos has been tested with. A client should not hard-code ids: operators rename and replace models in the config, and add, replace or remove them while the server runs (below).

GET /v1/models/catalog#

Hypnos extension. The model catalog (models.catalog.toml next to the server): models Hypnos has been tested with, configured here or not, each with its quantization variants, the files a variant needs, what of it is already in the server's models folder, and the [[models]] config block that runs it. A client picks a variant, downloads its files into models_dir (each file: repo + remote, see below) and adds config_block to the config. Open during startup staging. 404 not_found when the server has no catalog file. ?modality=vision keeps the entries serving that modality (primary or additional).

{ "object": "list", "catalog_version": 1,
  "models_dir": "/srv/hypnos/models",
  "tested_architectures": ["gemma4", "gemma-embedding2", "lfm3"],
  "data": [ {
    "id": "lfm3-vl:2b", "name": "LFM3-VL 2B", "publisher": "Liquid AI", "license": "lfm1.0",
    "architecture": "lfm3", "modality": "vision", "additional_modalities": ["text_generation"],
    "engine": "ggml_llama", "capabilities": ["thinking"],
    "summary": "Small vision-language model: ...", "lineup": false,
    "configured": true, "enabled": true,
    "variants": [ {
      "quant": "Q8_0", "default": true, "size_bytes": 2225234528,
      "files": [
        { "role": "weights", "repo": "LiquidAI/LFM3-VL-2B-GGUF", "remote": "LFM3-VL-2B-Q8_0.gguf",
          "local": "LFM3-VL-2B-Q8_0.gguf", "size": 1846238496, "platform": "all", "present": true },
        { "role": "aux", "...": "..." } ],
      "installed": true, "outdated": false,
      "config_block": "# LFM3-VL 2B (Q8_0), Liquid AI, license lfm1.0. ...\n[[models]]\nid = \"lfm3-vl:2b\"\n..." } ] } ] }

POST /v1/advisor#

Hypnos extension. Recommends a lineup from the catalog for a machine and a set of needs: which models and variants, their context and slots, starting tier and priority, what stays resident together, warnings, and a complete config file. Open during startup staging. 404 not_found without a catalog; 400 when no lineup fits (a needed model does not fit the memory even alone).

POST /v1/advisor
{ "needs": ["chat", "voice", "documents"], "vram_bytes": 25769803776, "ram_bytes": 68719476736 }

needs: chat, vision, voice (speech in, chat, speech out), transcription, speech, embeddings, images, documents, entities. vram_bytes, ram_bytes and unified_memory describe the machine; each one left out is this server's (unified_memory defaults to this machine's unless vram_bytes is given). On unified memory vram_bytes is ignored.

{ "machine": { "vram_bytes": 25769803776, "ram_bytes": 68719476736, "unified_memory": false },
  "budgets": { "tier1_bytes": 25232932864, "tier2_bytes": 34359738368, "unified_memory": false },
  "models": [ {
    "id": "gemma4:26b-q4", "name": "Gemma 4 26B-A4B", "quant": "Q4_0", "serves": ["chat", "voice"],
    "weights_bytes": 15812973984, "resident_bytes": 19005218816, "runs_on_cpu": false,
    "gpu_layers": 30, "num_layers": 30, "cpu_expert_layers": 0, "context_len": 8192, "slots": 1,
    "initial_tier": "tier1_gpu_vram", "priority": "high", "keep_warm_in_ram": true,
    "moe_offload": "off", "load_secs_cold": 5.3, "load_secs_warm": 5.3 }, "..." ],
  "interactive_coresident": true, "interactive_bytes": 22656532480,
  "warnings": [],
  "specs": ["gemma4:26b-q4:Q4_0", "lfm3-audio:2b:Q8_0", "kokoro-82m-q4:fp32", "docling:int8"],
  "config": "# Generated by the Hypnos advisor ...\nconfig_version = 1\n..." }

How it decides: the budgets are derived as the server derives them on such a machine (Metal's default working set and a 4 GiB OS reserve on unified memory; VRAM minus 512 MiB, warm memory at most half the RAM, on a discrete GPU), and every candidate model is placed with the scheduler's own planner (catalog estimate values stand in for what the GGUF reports). All combinations of catalog models and variants serving the needs are tried; a lineup scores for the size of the model doing each need (its parameter count, estimated from the weights file and the quantization; transcription a quarter and speech output a tenth, since their quality levels off early), for keeping the interactive models (chat, vision, speech, embeddings) resident together (trying one slot and 8k context on the chat models when that is what makes them fit, and Mixture-of-Experts offload on a discrete GPU), and against models that only partly fit the GPU. serves lists the needs a model is the lineup's choice for. load_secs_* are rough (3 GB/s). specs go to ./run.sh models <spec> and POST /v1/models {"catalog": <spec>}; hypnos advise --needs ... [--vram 24G --ram 64G | --unified --ram 32G] [--json | --config] prints the same from the command line.

POST /v1/models, PUT /v1/models/{id}, DELETE /v1/models/{id}#

Hypnos extension. Changes the model pool while the server runs: requests are routed by model name, so a model added here is served by its id right away (its first request loads it), a replaced one keeps its id, and a removed one answers 404 model_not_found. With API keys set (section 2) these routes need an admin key: a plain server.api_keys entry, or a [[server.keys]] entry with admin = true; any other key gets 403 permission_denied. Changes are kept in an overlay next to the config (<config name>.models.toml) and applied at the next start; the config file itself is never rewritten. Not open during startup staging.

Body of POST and PUT, one of:

{ "catalog": "lfm3-vl:2b:Q4_K_M" }
{ "config": "[[models]]\nid = \"my-model\"\nmodality = \"text_generation\"\nengine = \"ggml_llama\"\nweights_path = \"my-model-Q4_K_M.gguf\"\n[models.arch]\n" }

catalog is an entry of GET /v1/models/catalog, <id> or <id>:<quant> (its default variant without a quant); config is one [[models]] block as in the config file, paths relative to models_dir. The model goes through the startup checks: its files must exist inside models_dir, [models.arch] values it leaves out are read from the GGUF, the config validation must pass, and a ggml_llama model's GGUF must report one of the catalog's tested_architectures. A failed check is 400 invalid_request_error with the reasons. The model's files must already be on disk (./run.sh models <id>[:<quant>], or the desktop app's Models dialog).

• POST /v1/models adds the model; 400 when its id is taken. Answers 201.
• PUT /v1/models/{id} replaces the model {id} with the one in the body, under the same id (the block's own id is ignored). Requests that arrive meanwhile wait and are then served by the new model; running ones finish on the old one, which is then unloaded. drain_secs in the body (default 120) bounds that wait: past it the replacement is refused with 429 model_busy and the old model stays. 404 for an unknown {id}.
• DELETE /v1/models/{id}?drain_secs=120 removes the model the same way: new requests wait, running ones finish (429 model_busy past drain_secs), the engine is unloaded, and the waiting requests then get 404.

{ "model": "lfm3-vl:2b", "action": "added", "persisted": true,
  "placement": { "...": "see appendix" } }

action is added, replaced or removed; persisted is false when the overlay could not be written (the change is live until the next restart); placement is absent on a removal.

An added or replacing model starts cold in Tier 3 whatever its initial_tier (that setting applies at startup); its first request loads it, and POST /v1/memory/tier (section 14) can stage it ahead. Disabling a model (below) keeps its block; removing it takes it out of the pool.

POST /v1/models/{id}/enable, POST /v1/models/{id}/disable#

Hypnos extension. Takes a model out of service (it is evicted to disk and hidden from /v1/models; requests answer 404 model_disabled) or puts it back (it loads on its next request). Not persisted to the config file. Disable is refused with 429 model_busy while the model is leased. Response:

{ "model": "gemma4:e4b-q4", "enabled": false, "placement": { "...": "see appendix" } }

POST /v1/models/{id}/moe-offload#

Hypnos extension. Sets a Mixture-of-Experts model's expert offload: its expert tensors stay in host memory and run on the CPU while attention, the dense layers and the KV cache stay on the GPU, so the model needs far less VRAM (the 26B: about 2.8 GiB of weights instead of 14.7 GiB) at the cost of speed (on Apple Silicon: decode about half, prefill about 14x slower). Default from the model's moe_offload in the config; not persisted.

POST /v1/models/gemma4:26b-q4/moe-offload
{ "mode": "auto" }

Response:

{ "model": "gemma4:26b-q4", "moe_offload": "auto", "applies": "now",
  "placement": { "...": "see appendix" } }

applies is "now" when the model was idle or not loaded (an idle resident model is unloaded to Tier 2, and its next request loads it with the new placement; with KV parking its conversations come back too) and "next_load" when it was busy. A dense model accepts the mode and ignores it. An unknown mode is 422, an unknown model 404. /v1/memory/status shows each model's moe_offload, expert_bytes (0 for a dense model) and placement.cpu_expert_layers.

7. Chat completions (text, vision, audio in, tools)#

POST /v1/chat/completions#

7.1 Request#

{
  "model": "gemma4:26b-q4",
  "messages": [
    { "role": "system", "content": "You are terse." },
    { "role": "user", "content": "Explain sliding-window attention in two sentences." }
  ],
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0.95,
  "top_k": 40,
  "seed": 42,
  "stream": false,
  "num_ctx": 8192,
  "think": false
}

Not supported (ignored if sent): n, functions / function_call (the legacy form of tools), parallel_tool_calls, response_format, logprobs, stop, presence_penalty, frequency_penalty, logit_bias, user. There is no /v1/completions route.

Content parts. A message's content may be an array:

{ "role": "user", "content": [
  { "type": "text", "text": "What is in this picture, and what does the speaker say?" },
  { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0...", "detail": "auto" } },
  { "type": "input_audio", "input_audio": { "data": "<base64 wav>", "format": "wav" } }
] }

Media parts are placed in the prompt where they appear in the message. The server's context estimate budgets 576 tokens per image and 1024 per audio clip; pass num_ctx when sending many.

7.2 Response (non-streaming)#

{
  "id": "chatcmpl-6c1d...", "object": "chat.completion", "created": 1793481600,
  "model": "gemma4:26b-q4",
  "choices": [ { "index": 0,
                 "message": { "role": "assistant", "content": "Sliding-window attention ..." },
                 "finish_reason": "stop" } ],
  "usage": { "prompt_tokens": 31, "completion_tokens": 58, "total_tokens": 89,
             "prompt_tokens_details": { "cached_tokens": 24 } },
  "hypnos_telemetry": { "...": "section 5" }
}

With thinking on (reasoning_effort other than "none", or think) and a model that thinks, message.reasoning_content holds the thoughts that came before the answer; the field is omitted when there were none. content is the answer alone.

finish_reason is stop (end of turn), length (max_tokens reached) or tool_calls (the model called tools; message.tool_calls lists them and content is usually empty, see 7.5). choices always has one element. prompt_tokens_details.cached_tokens is present only when part of the prompt was served from the execution slot's KV cache (prefix reuse): a client that keeps the conversation prefix stable (same system prompt, append-only history) gets faster prefill.

7.3 Concurrency#

Each text model has slots execution slots (3 for the default 26B); requests beyond the slots queue, and beyond max_queue_per_model are refused with 429 model_busy. Requests that share a decode step show concurrent_slots > 1 in the telemetry.

7.4 Streaming ("stream": true)#

The response is text/event-stream. Every event is a data: line holding one JSON chunk, with the OpenAI shape, and the stream ends with the literal data: [DONE]. Keep-alive comments (: ...) may appear between events; ignore them.

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1793481600,"model":"gemma4:26b-q4","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":…,"model":"…","choices":[{"index":0,"delta":{"content":"Sliding"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":…,"model":"…","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":31,"completion_tokens":58,"total_tokens":89},"hypnos_telemetry":{…}}

data: [DONE]

Sequence:

1. One chunk with delta.role = "assistant" and empty content (the OpenAI convention).
2. With thinking on, zero or more chunks with delta.reasoning_content (pieces of the thoughts), all before the first delta.content.
3. Zero or more chunks with delta.content (a token or a few characters; special markers and the thinking block's delimiters are never emitted).
4. With tools, one chunk per call with delta.tool_calls (7.5).
5. One final chunk with an empty delta, finish_reason stop, length or tool_calls, plus usage and hypnos_telemetry (no stream_options.include_usage needed; usage always comes).
6. [DONE].

If generation fails after the stream started, step 5 is instead a chunk with finish_reason: "error" and an error object at the top level ({"message": "...", "type": "server_error"}), followed by [DONE]. A client must treat that chunk as a failure even though the HTTP status was 200.

Closing the connection cancels generation at the next token and releases the model. The request timeout applies to the headers only; a stream can run as long as max_tokens takes.

7.5 Tool calling#

OpenAI-compatible function calling. The client describes functions in tools; the model either answers or asks for one or more calls; the client runs them and sends the results back as tool messages; the model then answers with them. The server never runs a tool itself.

{ "model": "gemma4:26b-q4",
  "messages": [ { "role": "user", "content": "What's the weather in Paris?" } ],
  "tools": [ { "type": "function", "function": {
      "name": "get_weather", "description": "Current weather for a city",
      "parameters": { "type": "object",
                      "properties": { "city": { "type": "string" },
                                      "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } },
                      "required": ["city"] } } } ] }

The answer to that:

{ "choices": [ { "index": 0, "finish_reason": "tool_calls",
    "message": { "role": "assistant", "content": "",
      "tool_calls": [ { "id": "call_1f0c…_1", "type": "function",
        "function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" } } ] } } ], … }

arguments is a JSON object serialized as a string, as in OpenAI. To continue, append the assistant message (with its tool_calls; content may be null) and one tool message per call, then send the conversation again with the same tools:

{ "role": "assistant", "content": null, "tool_calls": [ { "id": "call_1f0c…_1", "type": "function",
    "function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" } } ] },
{ "role": "tool", "tool_call_id": "call_1f0c…_1", "content": "{\"temperature\": 18, \"condition\": \"sunny\"}" }

A tool message's content is a string (send JSON as a string; it reaches the model as text) or an array of text parts; name is optional, the function is found from tool_call_id.

Streaming. Each call arrives whole in one chunk, delta.tool_calls holding one element with its index (0, 1, … in the order of the calls), id, type and the complete function.name and function.arguments; OpenAI clients that concatenate argument fragments by index work unchanged. Text the model writes before a call arrives as delta.content as usual. The final chunk has finish_reason: "tool_calls".

Models. Tool calling uses the model's native format: Gemma 4 (all the default chat models, and Nemotron 3.5 Bolt), Qwen 3.5 / 3.6 / 3.8 (and MiniCPM-V 4.6), Qwen3-Coder, NVIDIA Nemotron 3.5 Lightning, GLM-4.7-Flash and the LFM3 models. LFM2.5 has none in Hypnos (the catalog lists each entry's capabilities). A request with tools (and tool_choice other than "none") for a model without one is refused with 400 invalid_request_error. "required" and a named function are enforced by starting the model's answer with a call, not by a grammar: the arguments follow the schema as well as the model manages. Validate them before running the tool.

8. Embeddings#

POST /v1/embeddings#

{ "model": "gemmaembedding2", "input": ["first text", "second text"], "encoding_format": "float" }

input is one string or an array of strings (max_prompt_chars in total). encoding_format is accepted for compatibility; vectors are always returned as JSON float arrays (base64 is not implemented).

{ "object": "list", "model": "gemmaembedding2",
  "data": [ { "object": "embedding", "index": 0, "embedding": [0.0123, -0.0456, "…"] },
            { "object": "embedding", "index": 1, "embedding": ["…"] } ],
  "usage": { "prompt_tokens": 14, "total_tokens": 14 },
  "hypnos_telemetry": { "…": "section 5" } }

data is in input order. The vector length is the model's (768 for the default embedding model); a client should read it from the first response rather than assume it.

9. Audio transcriptions#

POST /v1/audio/transcriptions#

Multipart (recommended; what the OpenAI SDKs send):

The multipart part names are the only ones read; response_format, temperature and timestamp_granularities are ignored (the answer is always JSON without timestamps).

JSON is also accepted, with the same field names and the audio as an array of byte values in audio_bytes ({"model": "…", "audio_bytes": [82, 73, 70, 70, …], "filename": "clip.wav"}). It exists for completeness; use multipart.

{ "text": "The quick brown fox …", "duration_secs": 4.21, "hypnos_telemetry": { "…": "" } }

Each model family gets the instruction it was trained with: Gemma 4 the audio followed by "Transcribe this audio exactly. Output only the transcript." (plus "The speech is in ''." with a language), LFM3-Audio the system prompt "Perform ASR in ." with the language named in English (de -> German; an unknown code is passed as given), or "Perform ASR." without one. A prompt replaces that instruction (Gemma) or system prompt (LFM3). Thinking is never used for a transcription.

language is echoed when given. duration_secs is read from the WAV header when the file is a WAV. The telemetry's engine.audio_secs and engine.rtf (processing seconds per audio second) say how fast the transcription ran.

The same model also accepts audio inside a chat message (input_audio part, section 7.1), which is the way to ask questions about a recording instead of transcribing it.

10. Speech synthesis#

POST /v1/audio/speech#

{ "model": "kokoro-82m-q4", "input": "Hello from Hypnos.", "voice": "alloy",
  "response_format": "wav", "speed": 1.0, "stream": false }

Speech models. Besides Kokoro, llama.cpp models with a speech decoder speak (24 kHz, mono): LFM3-Audio and LFM3-Omni (19 languages; voice picks female or male, and OpenAI's male names echo, onyx, ash, ballad, verse, fable map to male) and Qwen3-TTS (10 languages; voice cloning: when the model's voices_path is a folder, voice names a reference clip in it, <voice>.wav / .mp3 / .flac, and when it is one file that clip is used for every voice). With stream these models send the whole answer as one chunk, since their codec runs at the end of the generation.

Response. The audio bytes with Content-Type: audio/wav or audio/pcm, plus the x-hypnos-* headers of section 5. There is no JSON wrapper.

Streaming. With "stream": true the body is sent in chunks as each phrase is synthesized: for wav the first chunk begins with a WAV header whose length fields declare an open-ended stream (0xFFFFFFFF), so a player can start immediately; for pcm the chunks are raw 16-bit little-endian mono samples at 24 kHz. The telemetry headers present at the start are x-hypnos-previous-tier, x-hypnos-gpu-layers and x-hypnos-switch-ms; the rest of the timings are in the request log only. If synthesis fails mid-stream the body simply ends early; a client that needs certainty should compare the samples received with the input length or use the non-streaming form.

11. Image generation#

POST /v1/images/generations#

{ "model": "moss-image", "prompt": "a lighthouse at dawn, oil painting",
  "negative_prompt": "text, watermark", "n": 1, "size": "1024x1024",
  "response_format": "b64_json", "steps": 20, "seed": 7, "cfg_scale": 3.0 }

quality, style and user are ignored.

{ "created": 1793481600,
  "data": [ { "b64_json": "iVBORw0KGgo…", "seed": 2718281828 } ],
  "hypnos_telemetry": { "…": "section 5" } }

data[].seed (Hypnos extension) is the seed each image was made with: the request's seed (or the random one) plus the image's index in the batch. Sending it back with the same prompt, size, steps and model gives the same image, and changing only the prompt keeps much of its composition. POST /v1/images/edits returns it too.

Images are PNG. One 1024x1024 generation takes tens of seconds on a 24 GB GPU and more than two minutes when the generator runs partially offloaded; keep the client timeout at the server's request_timeout_secs for this route.

POST /v1/images/edits#

OpenAI's image edit route, as multipart/form-data: model, prompt, one or more image (or image[]) files (PNG, JPEG or WebP, each up to max_image_bytes), and the optional fields of the generation route (n, size, negative_prompt, steps, seed, cfg_scale). The images reach the model as references: it edits them, or composes from several, as the prompt says. The response is the one of /v1/images/generations.

curl localhost:9999/v1/images/edits -F model=flux2-klein:4b -F prompt="give the fox a red scarf" \
  -F image=@fox.png

Only models with [models.image] edit = true edit (in the catalog: FLUX.2 klein 4B); others answer 400. mask is accepted by the form but refused with 400: the current models edit from the prompt and the references, not by inpainting.

12. Document conversion and pictures#

Hypnos extension. Converts PDFs, images and the office and web formats docling reads into Markdown (or docling's JSON), one page at a time, with optional OCR, picture descriptions by a vision model and a retrievable picture store.

POST /v1/documents/convert#

Multipart parts, or the same names as JSON fields:

Booleans in multipart parts accept true/false, 1/0, yes/no, on/off.

Response (non-streaming):

{
  "object": "document",
  "id": "doc-5f2c…",                       // return_pictures only
  "pictures_expire_at": "2026-10-30T19:42:11Z",   // return_pictures only
  "model": "docling", "filename": "report.pdf",
  "page_count": 12,
  "pages": [
    { "page": 2, "markdown": "## Results\n\n<!-- image -->\n\n> **Picture:** A bar chart …",
      "json": { "…": "output_format = json only" },
      "pictures": 1, "tables": 2, "ms": 310.4,
      "picture_urls": ["/v1/documents/doc-5f2c…/pictures/1"] }
  ],
  "markdown": "…the requested pages joined, with tables, numbered lists and sentences mended across page breaks…",
  "usage": { "pages": 4, "pictures": 3, "tables": 5, "pictures_described": 3 },
  "hypnos_telemetry": { "…": "engine.pages and engine.ms_per_page" }
}

page_count is the source document's page count; pages holds only the converted ones. Formats without pages (DOCX, HTML, images) come back as one page numbered 1.

Streaming (stream = true): text/event-stream, one JSON object per data: line, distinguished by object:

The stream then ends with the literal data: [DONE], as chat streams do.

A request the engine refuses up front (bad file, unsupported format) gets its HTTP error status; the stream only starts once the first event exists. Closing the connection stops the conversion at the next page. The document.page events let a client show pages progressively; to store the result, take the final document event and put the collected document.page events into its pages.

Pictures of a converted document#

Available only for a conversion made with return_pictures = true, until pictures_expire_at (the operator's server.document_pictures_ttl_secs, 1 h by default) or an explicit delete, whichever comes first. The store also has a size cap; when it is full the oldest documents go first. These routes stay open during startup staging and need no model.

A picture record:

{ "object": "document.picture", "document_id": "doc-5f2c…", "index": 1, "page": 2,
  "url": "/v1/documents/doc-5f2c…/pictures/1",
  "mime": "image/png", "bytes": 48213, "width": 640, "height": 420, "dpi": 144,
  "bbox": { "l": 72.0, "t": 540.2, "r": 380.5, "b": 318.9, "coord_origin": "BOTTOMLEFT" },
  "page_size": { "width": 612.0, "height": 792.0 },
  "caption": "Figure 3: revenue by quarter", "ref": "#/pictures/0",
  "image": "iVBORw0KGgo…" }

index is 1-based and counts pictures across the document in Markdown order; bbox and page_size are in the page's units (PDF points for PDFs) with coord_origin saying whether t and b count from the top or the bottom. caption and ref are present only when docling found them. Unknown ids and indexes answer 404 not_found.

13. Entity extraction#

POST /v1/entities#

Hypnos extension. Zero-shot named-entity extraction: the labels are free text.

{ "model": "gliner2.5-multi",
  "input": ["Mario Rossi works in Milan for Ferrari.", "Tim Cook announced the iPhone 17 in Cupertino."],
  "labels": ["person", "city", "company", "product"],
  "threshold": 0.5, "flat": true }

{ "object": "list", "model": "gliner2.5-multi",
  "data": [
    { "object": "entities", "index": 0, "entities": [
        { "text": "Mario Rossi", "label": "person", "start": 0, "end": 11, "score": 0.97 },
        { "text": "Milan", "label": "city", "start": 21, "end": 26, "score": 0.93 },
        { "text": "Ferrari", "label": "company", "start": 31, "end": 38, "score": 0.91 } ] },
    { "object": "entities", "index": 1, "entities": ["…"] } ],
  "usage": { "texts": 2, "characters": 86 },
  "hypnos_telemetry": { "…": "section 5" } }

start and end are character offsets ([start, end), Unicode scalar values, not bytes and not UTF-16 units): JavaScript clients must convert before slicing strings with astral characters.

14. Memory status and placement control#

Hypnos extension. These are the routes the playground's tier visualizer uses.

GET /v1/memory/status#

Open during startup staging. The full snapshot of the three tiers (bytes unless stated):

{
  "capabilities": { "accelerator": "NvidiaCuda", "topology": "DiscretePcie",
                    "device_name": "NVIDIA GeForce RTX 5090 Laptop GPU",
                    "total_vram_bytes": 25648562176, "usable_vram_budget_bytes": 25648562176,
                    "pinned_ram_budget_bytes": 34359738368, "physical_memory_bytes": 68719476736,
                    "simulated": false, "installed_memory_bytes": 68719476736,
                    "tier2_to_tier1_bandwidth_gbps": 25.0, "tier3_to_tier2_bandwidth_gbps": 6.0 },
  "tier1_vram_used_bytes": 22145000000, "tier1_vram_free_bytes": 3503562176,
  "tier1_vram_real_free_bytes": 1240000000, "tier1_vram_real_total_bytes": 25648562176,
  "tier1_vram_unaccounted_bytes": 2200000000, "tier1_vram_effective_free_bytes": 1240000000,
  "tier2_pinned_ram_used_bytes": 17000000000, "tier2_pinned_ram_free_bytes": 17359738368,
  "physical_pool_free_bytes": null,
  "process_footprint_bytes": 31500000000, "process_memory_bytes": 31500000000,
  "process_memory_cap_bytes": null,
  "gpu": { "utilization_percent": 87, "temperature_celsius": 71, "power_watts": 148.2,
           "sm_clock_mhz": 2400, "throttle_reasons": 0, "throttled": false,
           "ecc_uncorrected_errors": 0, "retired_pages_pending": false, "sampled_unix_ms": 1793481600123 },
  "models": [ {
      "id": "gemma4:26b-q4", "enabled": true, "modality": "text_generation",
      "modalities": ["text_generation", "vision"], "engine": "ggml_llama",
      "total_weight_bytes": 15814641056, "priority": "high",
      "current_tier": "tier1_gpu_vram", "placement": { "…": "appendix" },
      "moe_offload": "off", "expert_bytes": 12846382080,
      "active_leases": 1, "idle_seconds": 0.0, "total_requests_served": 812,
      "requests_per_minute": 2.4, "reload_estimate_ms": 4600 } ],
  "recent_evictions": [ { "model": "moss-image", "for_model": "gemma4:26b-q4", "reason": "…",
                          "to_tier": "tier3_nvme_ssd", "unix_ms": 1793481500000 } ]
}

tier1_vram_used_bytes is what the planner has booked; tier1_vram_real_free_bytes is what the device reports; tier1_vram_unaccounted_bytes is the difference the planner subtracts from what it hands out; tier1_vram_effective_free_bytes is what a new promotion can use right now. kv_park is the state of KV parking (/v1/memory/kv-park below): {"enabled", "max_bytes", "used_bytes", "models": [{"model", "slots", "tokens", "bytes", "parked_secs_ago"}]}. requests_per_minute (decayed over about 10 minutes) and reload_estimate_ms (measured reload time from where an eviction would leave the model, else estimated from its size) are what eviction weighs within a priority: the model whose loss costs least (reload time x rate) goes first. recent_evictions lists the last automatic evictions with their reason (memory_budget, full_gpu_placement, process_memory_cap, device_out_of_memory). gpu is present on NVIDIA devices with nvidia-smi available and is sampled at most every few seconds. Additional fields may appear in later versions; clients should ignore unknown ones.

latency holds end-to-end p50 / p95 / p99 (ms, bucket upper bounds) with count, per route (routes) and per declared workflow (workflows, see section 1). Each model also has service_ms (average time a request holds it, once measured) and estimated_wait_ms (what a new request would wait now).

With [server] profile set, the status also carries "profile": {"name": "voice_assistant", "objective": {"metric": "first_audio", "target_ms": 1500}} (metric: first_token, first_audio or throughput, the last without target_ms).

POST /v1/memory/tier#

Moves a model by hand.

{ "model": "gemma4:26b-q4", "target_tier": "tier2_pinned_ram", "context_len": 16384 }

target_tier is tier1_gpu_vram, tier2_pinned_ram or tier3_nvme_ssd; context_len (optional) is the context to allocate when promoting to Tier 1. Refused with 429 model_busy while the model is leased, 507 when it does not fit, 404 when unknown or disabled. Response: {"model": "…", "placement": {…}}. A promotion blocks for as long as the load takes.

GET /v1/memory/kv-park, POST /v1/memory/kv-park#

Hypnos extension. KV parking keeps a llama.cpp model's conversations when it leaves Tier 1: each used slot's KV state is saved in host memory, and when the model comes back with the same context layout (same allocated_context_len, slot count, KV type and weights) the slots are restored before it takes requests. The next turn of a conversation then prefills only its new tokens instead of the whole history (on Apple Silicon, a 12.6k-token conversation: 4.7 s -> 1.1 s for the turn after a swap). Off by default (hardware.kv_park); max_bytes caps the host memory the parked states may hold, the least recently parked models going first.

POST /v1/memory/kv-park
{ "enabled": true, "max_bytes": 4294967296 }

Either field may be left out (an empty body is 400). Turning it off drops every parked state; a lower cap drops the oldest until the rest fits. The change is not written back to the config. Both routes answer the same object as kv_park in /v1/memory/status:

{ "enabled": true, "max_bytes": 4294967296, "used_bytes": 360710144,
  "models": [ { "model": "gemma4:26b-q4", "slots": 1, "tokens": 12612,
                "bytes": 360710144, "parked_secs_ago": 4.2 } ] }

A parked state that does not fit the new context (the model came back with another window or slot count) is dropped and the conversation prefills as before; parking never changes an answer. A disabled model's parked state is dropped. Slots emptied by server.kv_idle_timeout_secs have nothing to park.

15. Request log#

Hypnos extension. A ring buffer (200 entries by default) of recent inference calls, meant for the playground and for debugging. Open during startup staging. Polling it is not logged.

A record: id, route, method, status, key (label), started_at (RFC 3339), duration_ms, model, stream, tokens_in, tokens_out, cached_tokens, tokens_per_s, inference_ms, tier_switch_ms, previous_tier, error, level, truncated, and at debug or trace also input, output (debug) and request_payload, response_payload (trace). The operator's request_log_redact may blank or pattern-redact the captured text. Treat the captured text as sensitive; do not build product features on this route.

16. Metrics#

GET /metrics returns Prometheus text (text/plain; version=0.0.4). It is open unless the operator sets server.metrics_require_auth. The families a client dashboard may care about: hypnos_http_requests_total{route,class}, hypnos_http_request_latency_ms_{sum,max}, hypnos_chat_ttft_ms, hypnos_chat_decode_tokens_per_s, hypnos_chat_prefill_tokens_per_s, hypnos_prefix_cache_{hits,misses,tokens_reused}_total, hypnos_tts_rtf, hypnos_tts_ttfa_ms, hypnos_stt_rtf, hypnos_image_jct_s, hypnos_document_ms_per_page, hypnos_document_pages_total, hypnos_scheduler_{promotions,evictions,load_failures,admissions_rejected}_total, hypnos_model_load_failures_total{model}, hypnos_model_tier{model}, hypnos_model_gpu_layers{model}, hypnos_model_active_leases{model}, hypnos_model_requests_total{model}, hypnos_vram_bytes{kind}, hypnos_pinned_ram_bytes{kind}, hypnos_gpu_*, hypnos_api_key_{requests,rejections,tokens}_total{key}, hypnos_api_key_tokens_today{key}, hypnos_uptime_seconds, hypnos_process_start_time_seconds. The full list with help strings is in the endpoint's own output.

17. Writing a client: recommendations#

These are the behaviours a well-behaved client or SDK should implement; they follow from the sections above.

Configuration. Base URL (default http://127.0.0.1:9999), optional API key, a request timeout per route family, a User-Agent. Keep the timeout for chat, images and documents at or above the server's request_timeout_secs (900 s by default); a cold request pays a model load.

Startup. Poll GET /ready with a short delay until it answers 200, or treat 503 model_not_available with Retry-After as "wait and retry". GET /v1/models works before that and tells the client what it may ask for.

A changing model pool. Models can be added, replaced and removed while the server runs (section 6). A replacement is invisible to clients (requests wait for it and keep the id); a removal turns the id into 404 model_not_found. A client that caches GET /v1/models should refresh it on a 404 rather than retry, and pick models by modality (/v1/memory/status modalities) rather than by fixed ids. Do not retry POST, PUT or DELETE /v1/models automatically: a 429 there means running requests outlasted drain_secs, and nothing changed.

Thinking. reasoning_effort adds latency and tokens before the first answer token (the thoughts count in completion_tokens, and max_tokens must leave room for both). Show reasoning_content apart from the answer and never send it back in the history: only the answer belongs there.

Retries. Retry, with backoff, on: 429 model_busy (no Retry-After; start at 250 ms), 429 rate_limit_error (sleep Retry-After), 503 model_not_available (sleep Retry-After, default 5 s), 507 (another model holds the memory; sleep seconds, not milliseconds), 504 and network errors before any byte was received. Do not retry 400, 401, 404, 413, or a 500 more than once. Never retry a request whose stream already produced output; surface the partial result and the error.

Streaming. Parse SSE by lines: data: payload, blank line ends an event; ignore : comment lines; stop on [DONE]. Deliver delta.content pieces as they arrive (and the thoughts in delta.reasoning_content apart from them); collect usage and hypnos_telemetry from the final chunk; map finish_reason: "error" to an exception. Cancel by closing the connection, nothing else is needed. For document streams dispatch on object.

Cancellation. Closing the socket cancels any in-flight request at the next token, chunk, step or page and releases the model. SDKs should expose an abort signal that does exactly that.

Concurrency. A single client may keep several chat requests in flight; they batch on the server up to the model's slots. Beyond that they queue, and beyond max_queue_per_model they fail with 429. An SDK that fans out work should bound its parallelism per model.

Uploads. Use multipart for audio and documents; JSON with base64 inflates the body by a third and the server's limits apply to the decoded size. Remote URLs are fetched by the server only when the operator allows it.

Prefix reuse. Send conversations with a stable prefix (same system prompt, append-only history). usage.prompt_tokens_details.cached_tokens and engine.cached_tokens show when it paid off.

OpenAI SDK users. Point the SDK at http://host:9999/v1; chat.completions.create (with stream=True), embeddings.create, audio.transcriptions.create, audio.speech.create and images.generate work as-is. hypnos_telemetry is available through the SDK's raw-response or extra-fields mechanism. The Hypnos-only routes (documents, entities, memory, models enable/disable, logs) need a plain HTTP call.

Versioning. The response bodies are additive: new fields appear, existing ones keep their meaning. A client must ignore unknown fields. The server's version is in GET /health and is the release tag.

18. Appendix: shared types#

MemoryTier: "tier1_gpu_vram", "tier2_pinned_ram", "tier3_nvme_ssd".

Modality: "text_generation", "vision", "speech_to_text", "text_to_speech", "embeddings", "image_generation", "document_conversion", "entity_extraction".

EngineKind: "ggml_llama", "ggml_whisper", "ggml_diffusion", "onnx_runtime", "docling", "gliner".

ModelPriority: "low", "normal", "high", "realtime_pinned".

LayerPlacementPlan (in placement fields):

EngineTiming: see section 5. All durations are milliseconds; rates are per second; rtf is processing seconds per audio second.

Error body: section 4. Picture record: section 12.