OmlxGateway

Ktor-Client backed gateway to oMLX, an LLM server for Apple Silicon.

oMLX speaks the OpenAI chat-completions protocol. This gateway reuses the OpenAI gateway's message adapter and stream parsers, but not its model registry: every configured parameter reaches the server unchanged, for any model name.

Configuration

Each setting resolves as: the constructor argument, then the environment variable, then the default. Environment variables are read on the JVM only. On Android and iOS, pass explicit values.

The host has no /v1 suffix; the gateway adds it to every path. With an API key, requests carry Authorization: Bearer <key>. Without one, or with a blank key, they carry no authorization header.

One timeout applies to every request, including loadModel. It bounds connecting and each wait for data, so it also bounds a non-streaming reply, but not a whole streamed response that keeps sending. The default is longer than the other gateways use because local models are slow: a 16384-token reply at 16 tokens a second takes 17 minutes.

Chat requests

Requests send model, messages, temperature and max_tokens (never max_completion_tokens) from the config, and reasoning_effort, response_format and tools when they are set. CompletionConfig.numCtx and CompletionConfig.numPredict are not sent, because oMLX sets the context length per model.

reasoning_effort is sent as low, medium or high. It goes to the model's chat template, so its effect depends on the model. A null CompletionConfig.reasoningEffort leaves the model's default, and Qwen 3 models think by default. The model's reasoning arrives in LlmGatewayResponse.thinking.

Truncation

When max_tokens ends generation during thinking, a non-streaming response puts the partial reasoning in content, thinking is null, and the finish reason is length. A streaming response keeps it in the reasoning deltas. The gateway maps fields as the server sends them. content is not an answer unless LlmGatewayResponse.finishReason is stop.

Structured output

completeJson and completeJsonResponse send response_format with type json_schema and the schema named response. When oMLX cannot compile a grammar for a JSON format, it falls back to instructions in the prompt and says so in a Warning response header. On a structured request (those two methods, or CompletionConfig.responseFormat set to ResponseFormat.Json), the gateway puts the header value in LlmGatewayResponse.metadata under response_format_warning, joining several headers with , , and logs a warning. It ignores the header for text or absent formats. It does not retry or fail. Validate the content yourself.

Streaming

Both stream and streamEvents drop oMLX keep-alive frames (frames whose model is keepalive) before parsing. stream surfaces reasoning deltas as GatewayStreamEvent.Thinking; streamEvents yields no event for them.

Errors

A non-2xx response is an LlmGatewayException whose message carries the HTTP status and the response body unchanged, for example an unknown model (404) or unloading a model that is not loaded (400). In streamEvents it is a StreamErrorReason.ProviderError with the status.