Skip to main content

LlmBroker

Struct LlmBroker 

Source
pub struct LlmBroker { /* private fields */ }
Expand description

Main interface for LLM interactions

Implementations§

Source§

impl LlmBroker

Source

pub fn new( model: impl Into<String>, gateway: Arc<dyn LlmGateway>, tracer: Option<Arc<TracerSystem>>, ) -> Self

Create a new LLM broker

Uses SerialToolRunner as the tool execution strategy. Callers that want parallel fan-out should use LlmBroker::with_tool_runner.

§Arguments
  • model - The name of the LLM model to use
  • gateway - The gateway to use for LLM communication
  • tracer - Optional tracer system for observability
Source

pub fn with_tool_runner( model: impl Into<String>, gateway: Arc<dyn LlmGateway>, tracer: Option<Arc<TracerSystem>>, tool_runner: Arc<dyn ToolRunner>, ) -> Self

Create a new LLM broker with an explicit tool runner.

Use this when you want crate::llm::tools::ParallelToolRunner (or a custom strategy) instead of the default serial runner.

Source

pub async fn generate_response( &self, messages: &[LlmMessage], tools: Option<&[Box<dyn LlmTool>]>, config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Result<LlmGatewayResponse>

Return one native provider response without executing tools or extending history. The caller supplies the complete context for each request.

§Errors

Returns gateway errors unchanged.

Source

pub async fn generate( &self, messages: &[LlmMessage], tools: Option<&[Box<dyn LlmTool>]>, config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Result<String>

Generate text response from LLM

§Arguments
  • messages - The messages to send to the LLM
  • tools - Optional tools available to the LLM
  • config - Optional completion configuration
  • correlation_id - Optional correlation ID for tracing (generates UUID if None)
Source

pub async fn generate_object<T>( &self, messages: &[LlmMessage], config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Result<T>
where T: for<'de> Deserialize<'de> + Serialize + JsonSchema + Send,

Generate structured object response from LLM

§Arguments
  • messages - The messages to send to the LLM
  • config - Optional completion configuration
  • correlation_id - Optional correlation ID for tracing (generates UUID if None)
Source

pub fn generate_stream<'a>( &'a self, messages: &'a [LlmMessage], tools: Option<&'a [Box<dyn LlmTool>]>, config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Pin<Box<dyn Stream<Item = Result<String>> + 'a>>

Generate streaming text response from LLM

Returns a stream that yields content chunks as they arrive. When tool calls are detected, the broker executes them and recursively streams the LLM’s follow-up response.

§Arguments
  • messages - The messages to send to the LLM
  • tools - Optional tools available to the LLM
  • config - Optional completion configuration
  • correlation_id - Optional correlation ID for tracing (generates UUID if None)
§Example
ⓘ
use futures::stream::StreamExt;

let broker = LlmBroker::new("qwen3:32b", gateway, None);
let messages = vec![LlmMessage::user("Tell me a story")];

let mut stream = broker.generate_stream(&messages, None, None, None);
while let Some(result) = stream.next().await {
    match result {
        Ok(chunk) => print!("{}", chunk),
        Err(e) => eprintln!("Error: {}", e),
    }
}
Source

pub fn generate_stream_events<'a>( &'a self, messages: &'a [LlmMessage], config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> StreamEventStream<'a>

Stream one turn as StreamEvents, ending with terminal completion evidence.

The stream yields StreamEvent::Content in order, then exactly one terminal event: StreamEvent::Completed when the provider finished with stop and sent its terminal marker, or StreamEvent::Error otherwise. Nothing follows the terminal event. Content that arrived before an error is evidence, not a complete result.

This sends one request with no tools. Tool iterations are forced to zero, and there is no retry, continuation or recursion. Dropping the stream (or breaking out of the loop that reads it) cancels the request.

The tracer records the call when the request starts and the response, with its evidence, when the terminal event arrives. A consumer that stops early gets a traced call and no traced response. A gateway that does not support this API yields a single StreamEventError::StreamEventsUnsupported error, sends no request, and records nothing.

§Example
use futures::StreamExt;
use mojentic::llm::gateways::OllamaGateway;
use mojentic::llm::{LlmBroker, LlmMessage, StreamEvent};
use std::sync::Arc;

let broker = LlmBroker::new("qwen3:32b", Arc::new(OllamaGateway::new()), None);
let messages = vec![LlmMessage::user("Write a haiku about rivers.")];

let mut events = broker.generate_stream_events(&messages, None, None);
while let Some(event) = events.next().await {
    match event {
        StreamEvent::Content(text) => print!("{text}"),
        StreamEvent::Completed(evidence) => println!("\nusage: {:?}", evidence.usage),
        StreamEvent::Error(error) => eprintln!("\nincomplete: {error}"),
        _ => {}
    }
}

Trait Implementations§

Source§

impl Clone for LlmBroker

Source§

fn clone(&self) -> LlmBroker

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> DynClone for T
where T: Clone,

Source§

fn __clone_box(&self, _: Private) -> *mut ()

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more