Skip to main content

OmlxGateway

Struct OmlxGateway 

Source
pub struct OmlxGateway { /* private fields */ }
Expand description

Gateway for a local oMLX server.

Supports chat, structured output, tool calls, both streaming APIs, embeddings, and loading and unloading models.

§Thinking

oMLX reports a model’s reasoning trace as reasoning_content. The gateway maps it to LlmGatewayResponse::thinking and, in the legacy streaming API, to StreamChunk::Thinking. The events API has no thinking event and yields none.

reasoning_effort of low, medium or high is sent unchanged to the model’s chat template as "low", "medium" or "high". Its effect depends on the model. ReasoningEffort::Disabled sends enable_thinking: false and no reasoning_effort, which turns thinking off. Leaving it unset keeps the model’s default; Qwen 3 models think by default.

§Truncation

When max_tokens ends generation during thinking, a non-streaming response carries the partial reasoning in content and no thinking, with a finish reason of length. A streaming response keeps it as thinking. The gateway maps the fields as oMLX reports them. When the finish reason in LlmGatewayResponse::evidence is not stop, content is not an answer.

§Example

use mojentic::llm::gateways::OmlxGateway;
use mojentic::llm::{LlmBroker, LlmMessage};
use std::sync::Arc;

let broker = LlmBroker::new("Qwen3.8-27B-MLX-8bit", Arc::new(OmlxGateway::new()), None);
let reply = broker.generate(&[LlmMessage::user("Hello")], None, None, None).await?;

Implementations§

Source§

impl OmlxGateway

Source

pub fn with_stream_clock(self, clock: Arc<dyn StreamClock>) -> Self

Select a monotonic measurement clock. The default uses std::time::Instant.

Source

pub fn with_recovery( self, policy: RecoveryPolicy, ) -> Result<RecoveryGateway<Self>>

Opt into per-request recovery without changing legacy entrypoints.

Source

pub fn new() -> Self

Create a gateway configured from the environment. See OmlxConfig.

Source

pub fn with_config(config: OmlxConfig) -> Self

Create a gateway with explicit configuration.

Source

pub fn with_host(host: impl Into<String>) -> Self

Create a gateway for host, reading the key and timeout from the environment.

Source

pub async fn load_model(&self, model: &str) -> Result<()>

Load a model into memory ahead of its first request.

Blocks until the model is in memory. A chat request loads its model on its own, so this is only for warming up. oMLX downloads models only through its admin dashboard; there is no pull operation.

§Errors

MojenticError::GatewayError with the HTTP status and response body when oMLX rejects the request, or MojenticError::HttpError when the request fails.

Source

pub async fn unload_model(&self, model: &str) -> Result<()>

Unload a model from memory.

§Errors

MojenticError::GatewayError with the HTTP status and response body when oMLX rejects the request. Unloading a model that is not loaded is a 400 invalid_request_error. MojenticError::HttpError when the request fails.

Trait Implementations§

Source§

impl Default for OmlxGateway

Source§

fn default() -> Self

Returns the “default value” for a type. Read more
Source§

impl LlmGateway for OmlxGateway

Source§

fn calculate_embeddings<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, text: &'life1 str, model: Option<&'life2 str>, ) -> Pin<Box<dyn Future<Output = Result<Vec<f32>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Embed text in one request.

oMLX has no standard embedding model, so model is required. There is no client-side chunking; the server applies its model’s input limit.

§Errors

MojenticError::InvalidArgument without sending a request when model is missing or blank. MojenticError::GatewayError with the status and body when oMLX rejects the request, for example a 400 for a model that is not an embedding model.

Source§

fn stream_clock(&self) -> Arc<dyn StreamClock> ⓘ

Monotonic clock used by maintained streaming adapters.
Source§

fn get_model_context_window<'life0, 'life1, 'async_trait>( &'life0 self, model: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<Option<ModelContextWindow>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Query advertised context capacity without changing subsequent requests. Read more
Source§

fn complete<'life0, 'life1, 'life2, 'life3, 'life4, 'async_trait>( &'life0 self, model: &'life1 str, messages: &'life2 [LlmMessage], tools: Option<&'life3 [Box<dyn LlmTool>]>, config: &'life4 CompletionConfig, ) -> Pin<Box<dyn Future<Output = Result<LlmGatewayResponse>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait, 'life4: 'async_trait,

Complete an LLM request with text response
Source§

fn complete_json<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, model: &'life1 str, messages: &'life2 [LlmMessage], schema: Value, config: &'life3 CompletionConfig, ) -> Pin<Box<dyn Future<Output = Result<Value>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Complete an LLM request with structured JSON response
Source§

fn complete_json_response<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, model: &'life1 str, messages: &'life2 [LlmMessage], schema: Value, config: &'life3 CompletionConfig, ) -> Pin<Box<dyn Future<Output = Result<LlmGatewayResponse<Value>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Complete a structured JSON request and keep the provider’s evidence. Read more
Source§

fn get_available_models<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Vec<String>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Get list of available models
Source§

fn complete_stream<'a>( &'a self, model: &'a str, messages: &'a [LlmMessage], tools: Option<&'a [Box<dyn LlmTool>]>, config: &'a CompletionConfig, ) -> Pin<Box<dyn Stream<Item = Result<StreamChunk>> + Send + 'a>>

Stream LLM responses chunk by chunk Read more
Source§

fn complete_stream_events<'a>( &'a self, model: &'a str, messages: &'a [LlmMessage], config: &'a CompletionConfig, ) -> Result<StreamEventStream<'a>, StreamEventError>

Stream one turn as crate::llm::StreamEvents with terminal completion evidence. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more