pub struct OmlxGateway { /* private fields */ }Expand description
Gateway for a local oMLX server.
Supports chat, structured output, tool calls, both streaming APIs, embeddings, and loading and unloading models.
§Thinking
oMLX reports a model’s reasoning trace as reasoning_content. The gateway
maps it to LlmGatewayResponse::thinking and, in the legacy streaming
API, to StreamChunk::Thinking. The events API has no thinking event
and yields none.
reasoning_effort of low, medium or high is sent unchanged to the model’s
chat template as "low", "medium" or "high". Its effect depends on the
model. ReasoningEffort::Disabled sends enable_thinking: false and no
reasoning_effort, which turns thinking off. Leaving it unset keeps the
model’s default; Qwen 3 models think by default.
§Truncation
When max_tokens ends generation during thinking, a non-streaming response
carries the partial reasoning in content and no thinking, with a finish
reason of length. A streaming response keeps it as thinking. The gateway
maps the fields as oMLX reports them. When the finish reason in
LlmGatewayResponse::evidence is not stop, content is not an answer.
§Example
use mojentic::llm::gateways::OmlxGateway;
use mojentic::llm::{LlmBroker, LlmMessage};
use std::sync::Arc;
let broker = LlmBroker::new("Qwen3.8-27B-MLX-8bit", Arc::new(OmlxGateway::new()), None);
let reply = broker.generate(&[LlmMessage::user("Hello")], None, None, None).await?;Implementations§
Source§impl OmlxGateway
impl OmlxGateway
Sourcepub fn with_stream_clock(self, clock: Arc<dyn StreamClock>) -> Self
pub fn with_stream_clock(self, clock: Arc<dyn StreamClock>) -> Self
Select a monotonic measurement clock. The default uses std::time::Instant.
Sourcepub fn with_recovery(
self,
policy: RecoveryPolicy,
) -> Result<RecoveryGateway<Self>>
pub fn with_recovery( self, policy: RecoveryPolicy, ) -> Result<RecoveryGateway<Self>>
Opt into per-request recovery without changing legacy entrypoints.
Sourcepub fn new() -> Self
pub fn new() -> Self
Create a gateway configured from the environment. See OmlxConfig.
Sourcepub fn with_config(config: OmlxConfig) -> Self
pub fn with_config(config: OmlxConfig) -> Self
Create a gateway with explicit configuration.
Sourcepub fn with_host(host: impl Into<String>) -> Self
pub fn with_host(host: impl Into<String>) -> Self
Create a gateway for host, reading the key and timeout from the environment.
Sourcepub async fn load_model(&self, model: &str) -> Result<()>
pub async fn load_model(&self, model: &str) -> Result<()>
Load a model into memory ahead of its first request.
Blocks until the model is in memory. A chat request loads its model on its own, so this is only for warming up. oMLX downloads models only through its admin dashboard; there is no pull operation.
§Errors
MojenticError::GatewayError with the HTTP status and response body
when oMLX rejects the request, or MojenticError::HttpError when the
request fails.
Sourcepub async fn unload_model(&self, model: &str) -> Result<()>
pub async fn unload_model(&self, model: &str) -> Result<()>
Unload a model from memory.
§Errors
MojenticError::GatewayError with the HTTP status and response body
when oMLX rejects the request. Unloading a model that is not loaded is
a 400 invalid_request_error. MojenticError::HttpError when the
request fails.
Trait Implementations§
Source§impl Default for OmlxGateway
impl Default for OmlxGateway
Source§impl LlmGateway for OmlxGateway
impl LlmGateway for OmlxGateway
Source§fn calculate_embeddings<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
text: &'life1 str,
model: Option<&'life2 str>,
) -> Pin<Box<dyn Future<Output = Result<Vec<f32>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn calculate_embeddings<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
text: &'life1 str,
model: Option<&'life2 str>,
) -> Pin<Box<dyn Future<Output = Result<Vec<f32>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Embed text in one request.
oMLX has no standard embedding model, so model is required. There is
no client-side chunking; the server applies its model’s input limit.
§Errors
MojenticError::InvalidArgument without sending a request when
model is missing or blank. MojenticError::GatewayError with the
status and body when oMLX rejects the request, for example a 400 for a
model that is not an embedding model.
Source§fn stream_clock(&self) -> Arc<dyn StreamClock> ⓘ
fn stream_clock(&self) -> Arc<dyn StreamClock> ⓘ
Source§fn get_model_context_window<'life0, 'life1, 'async_trait>(
&'life0 self,
model: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Option<ModelContextWindow>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn get_model_context_window<'life0, 'life1, 'async_trait>(
&'life0 self,
model: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Option<ModelContextWindow>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Source§fn complete<'life0, 'life1, 'life2, 'life3, 'life4, 'async_trait>(
&'life0 self,
model: &'life1 str,
messages: &'life2 [LlmMessage],
tools: Option<&'life3 [Box<dyn LlmTool>]>,
config: &'life4 CompletionConfig,
) -> Pin<Box<dyn Future<Output = Result<LlmGatewayResponse>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
'life4: 'async_trait,
fn complete<'life0, 'life1, 'life2, 'life3, 'life4, 'async_trait>(
&'life0 self,
model: &'life1 str,
messages: &'life2 [LlmMessage],
tools: Option<&'life3 [Box<dyn LlmTool>]>,
config: &'life4 CompletionConfig,
) -> Pin<Box<dyn Future<Output = Result<LlmGatewayResponse>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
'life4: 'async_trait,
Source§fn complete_json<'life0, 'life1, 'life2, 'life3, 'async_trait>(
&'life0 self,
model: &'life1 str,
messages: &'life2 [LlmMessage],
schema: Value,
config: &'life3 CompletionConfig,
) -> Pin<Box<dyn Future<Output = Result<Value>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
fn complete_json<'life0, 'life1, 'life2, 'life3, 'async_trait>(
&'life0 self,
model: &'life1 str,
messages: &'life2 [LlmMessage],
schema: Value,
config: &'life3 CompletionConfig,
) -> Pin<Box<dyn Future<Output = Result<Value>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
Source§fn complete_json_response<'life0, 'life1, 'life2, 'life3, 'async_trait>(
&'life0 self,
model: &'life1 str,
messages: &'life2 [LlmMessage],
schema: Value,
config: &'life3 CompletionConfig,
) -> Pin<Box<dyn Future<Output = Result<LlmGatewayResponse<Value>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
fn complete_json_response<'life0, 'life1, 'life2, 'life3, 'async_trait>(
&'life0 self,
model: &'life1 str,
messages: &'life2 [LlmMessage],
schema: Value,
config: &'life3 CompletionConfig,
) -> Pin<Box<dyn Future<Output = Result<LlmGatewayResponse<Value>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
Source§fn get_available_models<'life0, 'async_trait>(
&'life0 self,
) -> Pin<Box<dyn Future<Output = Result<Vec<String>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
fn get_available_models<'life0, 'async_trait>(
&'life0 self,
) -> Pin<Box<dyn Future<Output = Result<Vec<String>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
Source§fn complete_stream<'a>(
&'a self,
model: &'a str,
messages: &'a [LlmMessage],
tools: Option<&'a [Box<dyn LlmTool>]>,
config: &'a CompletionConfig,
) -> Pin<Box<dyn Stream<Item = Result<StreamChunk>> + Send + 'a>>
fn complete_stream<'a>( &'a self, model: &'a str, messages: &'a [LlmMessage], tools: Option<&'a [Box<dyn LlmTool>]>, config: &'a CompletionConfig, ) -> Pin<Box<dyn Stream<Item = Result<StreamChunk>> + Send + 'a>>
Source§fn complete_stream_events<'a>(
&'a self,
model: &'a str,
messages: &'a [LlmMessage],
config: &'a CompletionConfig,
) -> Result<StreamEventStream<'a>, StreamEventError>
fn complete_stream_events<'a>( &'a self, model: &'a str, messages: &'a [LlmMessage], config: &'a CompletionConfig, ) -> Result<StreamEventStream<'a>, StreamEventError>
crate::llm::StreamEvents with terminal completion evidence. Read more