pub struct LlmBroker { /* private fields */ }Expand description
Main interface for LLM interactions
Implementations§
Source§impl LlmBroker
impl LlmBroker
Sourcepub fn new(
model: impl Into<String>,
gateway: Arc<dyn LlmGateway>,
tracer: Option<Arc<TracerSystem>>,
) -> Self
pub fn new( model: impl Into<String>, gateway: Arc<dyn LlmGateway>, tracer: Option<Arc<TracerSystem>>, ) -> Self
Create a new LLM broker
Uses SerialToolRunner as the tool execution strategy. Callers that
want parallel fan-out should use LlmBroker::with_tool_runner.
§Arguments
model- The name of the LLM model to usegateway- The gateway to use for LLM communicationtracer- Optional tracer system for observability
Sourcepub fn with_tool_runner(
model: impl Into<String>,
gateway: Arc<dyn LlmGateway>,
tracer: Option<Arc<TracerSystem>>,
tool_runner: Arc<dyn ToolRunner>,
) -> Self
pub fn with_tool_runner( model: impl Into<String>, gateway: Arc<dyn LlmGateway>, tracer: Option<Arc<TracerSystem>>, tool_runner: Arc<dyn ToolRunner>, ) -> Self
Create a new LLM broker with an explicit tool runner.
Use this when you want crate::llm::tools::ParallelToolRunner (or a
custom strategy) instead of the default serial runner.
Sourcepub async fn generate_response(
&self,
messages: &[LlmMessage],
tools: Option<&[Box<dyn LlmTool>]>,
config: Option<CompletionConfig>,
correlation_id: Option<String>,
) -> Result<LlmGatewayResponse>
pub async fn generate_response( &self, messages: &[LlmMessage], tools: Option<&[Box<dyn LlmTool>]>, config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Result<LlmGatewayResponse>
Return one native provider response without executing tools or extending history. The caller supplies the complete context for each request.
§Errors
Returns gateway errors unchanged.
Sourcepub async fn generate(
&self,
messages: &[LlmMessage],
tools: Option<&[Box<dyn LlmTool>]>,
config: Option<CompletionConfig>,
correlation_id: Option<String>,
) -> Result<String>
pub async fn generate( &self, messages: &[LlmMessage], tools: Option<&[Box<dyn LlmTool>]>, config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Result<String>
Generate text response from LLM
§Arguments
messages- The messages to send to the LLMtools- Optional tools available to the LLMconfig- Optional completion configurationcorrelation_id- Optional correlation ID for tracing (generates UUID if None)
Sourcepub async fn generate_object<T>(
&self,
messages: &[LlmMessage],
config: Option<CompletionConfig>,
correlation_id: Option<String>,
) -> Result<T>
pub async fn generate_object<T>( &self, messages: &[LlmMessage], config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Result<T>
Generate structured object response from LLM
§Arguments
messages- The messages to send to the LLMconfig- Optional completion configurationcorrelation_id- Optional correlation ID for tracing (generates UUID if None)
Sourcepub fn generate_stream<'a>(
&'a self,
messages: &'a [LlmMessage],
tools: Option<&'a [Box<dyn LlmTool>]>,
config: Option<CompletionConfig>,
correlation_id: Option<String>,
) -> Pin<Box<dyn Stream<Item = Result<String>> + 'a>>
pub fn generate_stream<'a>( &'a self, messages: &'a [LlmMessage], tools: Option<&'a [Box<dyn LlmTool>]>, config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Pin<Box<dyn Stream<Item = Result<String>> + 'a>>
Generate streaming text response from LLM
Returns a stream that yields content chunks as they arrive. When tool calls are detected, the broker executes them and recursively streams the LLM’s follow-up response.
§Arguments
messages- The messages to send to the LLMtools- Optional tools available to the LLMconfig- Optional completion configurationcorrelation_id- Optional correlation ID for tracing (generates UUID if None)
§Example
use futures::stream::StreamExt;
let broker = LlmBroker::new("qwen3:32b", gateway, None);
let messages = vec![LlmMessage::user("Tell me a story")];
let mut stream = broker.generate_stream(&messages, None, None, None);
while let Some(result) = stream.next().await {
match result {
Ok(chunk) => print!("{}", chunk),
Err(e) => eprintln!("Error: {}", e),
}
}Sourcepub fn generate_stream_events<'a>(
&'a self,
messages: &'a [LlmMessage],
config: Option<CompletionConfig>,
correlation_id: Option<String>,
) -> StreamEventStream<'a>
pub fn generate_stream_events<'a>( &'a self, messages: &'a [LlmMessage], config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> StreamEventStream<'a>
Stream one turn as StreamEvents, ending with terminal completion evidence.
The stream yields StreamEvent::Content in order, then exactly one
terminal event: StreamEvent::Completed when the provider finished
with stop and sent its terminal marker, or StreamEvent::Error
otherwise. Nothing follows the terminal event. Content that arrived
before an error is evidence, not a complete result.
This sends one request with no tools. Tool iterations are forced to zero, and there is no retry, continuation or recursion. Dropping the stream (or breaking out of the loop that reads it) cancels the request.
The tracer records the call when the request starts and the response,
with its evidence, when the terminal event arrives. A consumer that
stops early gets a traced call and no traced response. A gateway that
does not support this API yields a single
StreamEventError::StreamEventsUnsupported error, sends no request,
and records nothing.
§Example
use futures::StreamExt;
use mojentic::llm::gateways::OllamaGateway;
use mojentic::llm::{LlmBroker, LlmMessage, StreamEvent};
use std::sync::Arc;
let broker = LlmBroker::new("qwen3:32b", Arc::new(OllamaGateway::new()), None);
let messages = vec![LlmMessage::user("Write a haiku about rivers.")];
let mut events = broker.generate_stream_events(&messages, None, None);
while let Some(event) = events.next().await {
match event {
StreamEvent::Content(text) => print!("{text}"),
StreamEvent::Completed(evidence) => println!("\nusage: {:?}", evidence.usage),
StreamEvent::Error(error) => eprintln!("\nincomplete: {error}"),
_ => {}
}
}