Mojentic for Rust

Mojentic is an agentic framework with an asynchronous pub/sub architecture designed to orchestrate LLM-powered agents. This Rust crate focuses on high performance, strong typing, and reliability.

• Fast, async-first event processing
• Agents compose into complex systems
• Clear separation between Core (LLM primitives) and Agents (behaviors)

If you’re familiar with the Python project, this book mirrors its structure and concepts, adapted to Rust idioms.

Quick links

• Getting Started → Get set up
• LLM usage → Broker and gateways
• Core concepts → Layer 1
• Agents → Layer 2
• Observability → Tracer
• API Docs → Rust API reference

Getting Started

This section helps you install and begin using Mojentic in Rust.

Install

Add the dependency (placeholder until published):

[dependencies]
mojentic = { path = "../" }

Minimal Example

use mojentic::{Broker, Message};

fn main() {
    let mut broker = Broker::default();
    broker.emit(Message::text("hello world"));
}

Next Steps

• Learn the LLM broker → Broker
• Dive into Core concepts → Layer 1 overview

Using LLMs (Broker)

Mojentic’s LLM broker routes completion requests to pluggable gateways (e.g., Ollama). It provides a unified API for chat and text completions.

Quick chat example

use mojentic::llm::gateways::OllamaGateway;
use mojentic::llm::{LlmBroker, LlmMessage};
use std::sync::Arc;

#[tokio::main]
async fn main() -> mojentic::Result<()> {
    let broker = LlmBroker::new("qwen3:32b", Arc::new(OllamaGateway::new()), None);
    let messages = vec![
        LlmMessage::system("You are a helpful assistant"),
        LlmMessage::user("Say hi in one sentence"),
    ];

    let text = broker.generate(&messages, None, None, None).await?;
    println!("{text}");
    Ok(())
}

Gateways

• Ollama: local models for fast iteration.
• oMLX: local models on Apple Silicon. See oMLX Gateway.
• HTTP-based gateways: add your own by implementing the Gateway trait.

Structured output

Use typed schemas to parse the model output into structs. See Structured Output.

Structured output in streaming requests

CompletionConfig.response_format asks the provider for a response format. The OpenAI and Ollama gateways send it in streaming and non-streaming requests alike.

#![allow(unused)]
fn main() {
use mojentic::llm::gateway::{CompletionConfig, ResponseFormat};

let config = CompletionConfig {
    response_format: Some(ResponseFormat::JsonObject {
        schema: Some(serde_json::json!({
            "type": "object",
            "properties": {"answer": {"type": "string"}},
            "required": ["answer"]
        })),
    }),
    ..Default::default()
};
let mut stream = broker.generate_stream(&messages, None, Some(config), None);
}

The format records what you asked for. It does not prove that the provider enforced it. Parse and validate the streamed content before you use it.

Configuration

Use CompletionConfig to control generation parameters:

#![allow(unused)]
fn main() {
use mojentic::llm::gateway::{CompletionConfig, ReasoningEffort};

let config = CompletionConfig {
    temperature: Some(0.3),
    reasoning_effort: Some(ReasoningEffort::High),
    ..Default::default()
};
}

Available Parameters

• temperature (Option<f32>): Sampling temperature; default None uses provider settings.
• num_ctx (Option<usize>): Ollama context size; default None uses model/server settings.
• max_tokens (Option<usize>): The only output limit; default None sends no limit.
• reasoning_effort (Option<ReasoningEffort>): Extended thinking level — Low, Medium, High, or None. Default: None

Reasoning Effort

Control how much the model thinks before responding:

#![allow(unused)]
fn main() {
use mojentic::llm::gateway::{CompletionConfig, ReasoningEffort};

// Deep reasoning for complex problems
let config = CompletionConfig {
    reasoning_effort: Some(ReasoningEffort::High),
    temperature: Some(0.1),
    ..Default::default()
};

// Quick responses
let config = CompletionConfig {
    reasoning_effort: Some(ReasoningEffort::Low),
    ..Default::default()
};
}

• Ollama: Maps to think: true parameter for extended thinking. The model’s reasoning trace is available in LlmGatewayResponse.thinking.
• OpenAI: Maps to reasoning_effort API parameter for reasoning models (o1, o3 series). Ignored with a warning for non-reasoning models.

For full details, see Reasoning Effort Control.

Single-turn streaming with terminal completion evidence

broker.generate_stream_events(&messages, config, correlation_id) streams one turn as StreamEvent::Content values and ends with exactly one terminal event: StreamEvent::Completed(evidence) or StreamEvent::Error(error). A response cut off at the token limit ends with StreamEventError::IncompleteCompletion, never with Completed. It sends one request with no tools, and dropping the stream cancels it. See Streaming for the full rules.

Caller-owned context and native responses

Use broker.generate_response(&messages, Some(&tools), Some(config), None).await to receive one native gateway response without executing tools, extending history or making a follow-up request. Assemble the complete message array before each call. The broker traces the supplied request and returned response; it does not read repository guidance or apply a context policy.

The existing convenience completion method still executes tools and follows up. Choose a serial or parallel runner according to the tools’ effects. Parallel execution does not make dependent edits safe.

Set CompletionConfig::default().with_unlimited_tool_iterations() to disable the tool-round limit. Existing finite defaults remain unchanged. Concurrency controls simultaneous work; it is not a task budget or a loop detector.

The unlimited builder uses usize::MAX as a symbolic value and bypasses the iteration check; existing numeric configuration remains compatible.

Native responses preserve the fields supplied by the gateway. Missing provider usage or termination evidence must remain unknown; configured model names and text length are not substitutes for reported metadata.

LlmGatewayResponse.evidence holds that evidence: usage, provider_model, finish_reason and metadata. The broker copies it into the response trace. See Tracer System.

Quick Reference

Installation

[dependencies]
mojentic = { path = "./mojentic-rust" }
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
schemars = "0.8"  # For structured output

Basic Usage

Import the prelude

#![allow(unused)]
fn main() {
use mojentic::prelude::*;
use std::sync::Arc;
}

Create a broker

#![allow(unused)]
fn main() {
let gateway = OllamaGateway::new();
let broker = LlmBroker::new("qwen3:32b", Arc::new(gateway));
}

Simple text generation

#![allow(unused)]
fn main() {
let messages = vec![LlmMessage::user("Hello, how are you?")];
let response = broker.generate(&messages, None, None).await?;
}

Message Types

User message

#![allow(unused)]
fn main() {
LlmMessage::user("Your message here")
}

System message

#![allow(unused)]
fn main() {
LlmMessage::system("You are a helpful assistant")
}

Assistant message

#![allow(unused)]
fn main() {
LlmMessage::assistant("Response from assistant")
}

Message with images

#![allow(unused)]
fn main() {
LlmMessage::user("Describe this image")
    .with_images(vec!["path/to/image.jpg".to_string()])
}

Structured Output

Define your type

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};
use schemars::JsonSchema;

#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct Person {
    name: String,
    age: u32,
    occupation: String,
}
}

Generate typed response

#![allow(unused)]
fn main() {
let messages = vec![
    LlmMessage::user("Extract person info: John Doe, 30, software engineer")
];

let person: Person = broker.generate_object(&messages, None).await?;
}

Configuration

Default config

#![allow(unused)]
fn main() {
let config = CompletionConfig::default();
// temperature: None
// num_ctx: None
// max_tokens: None
// top_p: None
// top_k: None
// response_format: None
}

Custom config

#![allow(unused)]
fn main() {
let config = CompletionConfig {
    temperature: Some(0.7),
    num_ctx: Some(16384),
    max_tokens: Some(8192),
    top_p: Some(0.9),
    top_k: Some(40),
    response_format: None,
    ..Default::default()
};

let response = broker.generate(&messages, None, Some(config)).await?;
}

JSON response format

#![allow(unused)]
fn main() {
use mojentic::llm::gateway::ResponseFormat;

// Request JSON without schema
let config = CompletionConfig {
    temperature: Some(0.7),
    num_ctx: Some(16384),
    max_tokens: Some(8192),
    top_p: None,
    top_k: None,
    response_format: Some(ResponseFormat::JsonObject { schema: None }),
    ..Default::default()
};

// Request JSON with schema
let schema = serde_json::json!({
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "number"}
    }
});

let config = CompletionConfig {
    temperature: Some(0.7),
    num_ctx: Some(16384),
    max_tokens: Some(8192),
    top_p: None,
    top_k: None,
    response_format: Some(ResponseFormat::JsonObject {
        schema: Some(schema)
    }),
    ..Default::default()
};
}

Gateway Configuration

Ollama with custom host

#![allow(unused)]
fn main() {
let gateway = OllamaGateway::with_host("http://localhost:11434");
}

Ollama with environment variable

#![allow(unused)]
fn main() {
// Set OLLAMA_HOST environment variable
std::env::set_var("OLLAMA_HOST", "http://custom-host:11434");
let gateway = OllamaGateway::new();
}

Error Handling

#![allow(unused)]
fn main() {
use mojentic::Result;

async fn my_function() -> Result<String> {
    let gateway = OllamaGateway::new();
    let broker = LlmBroker::new("qwen3:32b", Arc::new(gateway));

    let messages = vec![LlmMessage::user("Hello")];

    match broker.generate(&messages, None, None).await {
        Ok(response) => Ok(response),
        Err(e) => {
            eprintln!("Error: {}", e);
            Err(e)
        }
    }
}
}

Common Patterns

Multi-turn conversation

#![allow(unused)]
fn main() {
let mut messages = vec![
    LlmMessage::system("You are a helpful assistant"),
];

// First turn
messages.push(LlmMessage::user("What is Rust?"));
let response1 = broker.generate(&messages, None, None).await?;
messages.push(LlmMessage::assistant(response1));

// Second turn
messages.push(LlmMessage::user("What are its benefits?"));
let response2 = broker.generate(&messages, None, None).await?;
}

Async parallel requests

#![allow(unused)]
fn main() {
use tokio::try_join;

let broker = Arc::new(broker);
let broker1 = broker.clone();
let broker2 = broker.clone();

let (response1, response2) = try_join!(
    async move {
        broker1.generate(
            &[LlmMessage::user("Question 1")],
            None,
            None
        ).await
    },
    async move {
        broker2.generate(
            &[LlmMessage::user("Question 2")],
            None,
            None
        ).await
    }
)?;
}

Model management (Ollama)

#![allow(unused)]
fn main() {
let gateway = OllamaGateway::new();

// List available models
let models = gateway.get_available_models().await?;

// Pull a model
gateway.pull_model("qwen3:32b").await?;

// Calculate embeddings
let embedding = gateway.calculate_embeddings(
    "Some text to embed",
    Some("mxbai-embed-large")
).await?;
}

Logging

Enable logging with tracing:

// In main.rs
use tracing_subscriber;

#[tokio::main]
async fn main() {
    // Initialize logging
    tracing_subscriber::fmt::init();

    // Your code here
}

Set log level via environment variable:

RUST_LOG=debug cargo run
RUST_LOG=info cargo run
RUST_LOG=mojentic=debug cargo run

Use Cases

Tutorial: Building Chatbots

Why Use Chat Sessions?

When working with Large Language Models (LLMs), simple text generation is useful for one-off interactions, but many applications require ongoing conversations where the model remembers previous exchanges. This is where chat sessions come in.

Chat sessions are essential when you need to:

• Build conversational agents or chatbots
• Maintain context across multiple user interactions
• Create applications where the LLM needs to remember previous information
• Develop more natural and coherent conversational experiences

When to Apply This Approach

Use chat sessions when:

• Your application requires multi-turn conversations
• You need the LLM to reference information from earlier in the conversation
• You want to create a more interactive and engaging user experience

The Key Difference: Expanding Context

The fundamental difference between simple text generation and chat sessions is the expanding context. With each new message in a chat session:

1. The message is added to the conversation history
2. All previous messages (within token limits) are sent to the LLM with each new query
3. The LLM can reference and build upon earlier parts of the conversation

Getting Started

Let’s walk through a simple example of building a chatbot using Mojentic’s ChatSession.

Basic Implementation

Here’s the simplest way to create a chat session with Mojentic:

use mojentic::prelude::*;
use std::io::{self, Write};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<()> {
    // 1. Create an LLM broker
    let gateway = Arc::new(OllamaGateway::new());
    let broker = LlmBroker::new("qwen3:32b", gateway, None);

    // 2. Initialize a chat session
    let mut session = ChatSession::new(broker);

    // 3. Simple interactive loop
    println!("Chatbot started. Type 'exit' to quit.");

    loop {
        print!("Query: ");
        io::stdout().flush()?;

        let mut query = String::new();
        io::stdin().read_line(&mut query)?;
        let query = query.trim();

        if query == "exit" {
            break;
        }

        let response = session.send_message(query).await?;
        println!("{}", response);
    }
    
    Ok(())
}

This code creates an interactive chatbot that maintains context across multiple exchanges.

Step-by-Step Explanation

1. Initialize the Broker

#![allow(unused)]
fn main() {
let gateway = Arc::new(OllamaGateway::new());
let broker = LlmBroker::new("qwen3:32b", gateway, None);
}

The LlmBroker is the central component that handles communication with the LLM provider (in this case, Ollama).

2. Start the Session

#![allow(unused)]
fn main() {
let mut session = ChatSession::new(broker);
}

ChatSession holds the state of the conversation. By default, it manages the message history and ensures it fits within the model’s context window.

3. Send Messages

#![allow(unused)]
fn main() {
let response = session.send_message(query).await?;
}

When you send a message:

1. It’s added to the history.
2. The full history is sent to the LLM.
3. The LLM’s response is added to the history.
4. The response text is returned.

Customizing Your Chat Session

You can customize the session with a system prompt or tools.

System Prompt

The system prompt sets the behavior of the assistant.

#![allow(unused)]
fn main() {
let mut session = ChatSession::new(broker)
    .with_system_prompt("You are a helpful AI assistant specialized in Rust programming.");
}

Adding Tools

You can enhance your chatbot by providing tools that the LLM can use.

#![allow(unused)]
fn main() {
use mojentic::tools::DateResolver;

let tools: Vec<Arc<dyn LlmTool>> = vec![Arc::new(DateResolver::new())];

let mut session = ChatSession::new(broker)
    .with_tools(tools);

// The LLM can now use the date tool in conversations
let response = session.send_message("What day of the week is July 4th, 2025?").await?;
println!("{}", response);
}

Streaming Responses

For a better user experience with longer responses, you can stream the LLM’s reply as it’s generated:

#![allow(unused)]
fn main() {
use futures::stream::StreamExt;

let mut stream = session.send_stream("Tell me a story");
while let Some(result) = stream.next().await {
    print!("{}", result?);
}
println!(); // Newline after streaming completes
}

The send_stream() method works just like send() for conversation management — it adds the user message to history before streaming, and records the full assembled response after the stream is consumed. Tools are handled transparently through the broker’s recursive streaming.

Summary

In this tutorial, we’ve learned how to:

1. Initialize a ChatSession with an LlmBroker.
2. Create an interactive loop to chat with the model.
3. Customize the session with system prompts and tools.

By leveraging chat sessions, you can create engaging conversational experiences that maintain context across multiple interactions.

Tutorial: Extracting Structured Data

Why Use Structured Output?

LLMs are great at generating text, but sometimes you need data in a machine-readable format like JSON. Structured output allows you to define a schema (using Rust structs with Serde) and force the LLM to return data that matches that schema.

This is essential for:

• Data extraction from unstructured text
• Building API integrations
• Populating databases
• ensuring reliable downstream processing

Getting Started

Let’s build an example that extracts user information from a natural language description.

1. Define Your Data Schema

We use serde to define the structure we want.

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};
use schemars::JsonSchema;

#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct UserInfo {
    name: String,
    age: u32,
    interests: Vec<String>,
}
}

2. Initialize the Broker

#![allow(unused)]
fn main() {
use mojentic::prelude::*;
use std::sync::Arc;

let gateway = Arc::new(OllamaGateway::new());
let broker = LlmBroker::new("qwen3:32b", gateway, None);
}

3. Generate Structured Data

Use broker.generate_structured to request the data.

#![allow(unused)]
fn main() {
let text = "John Doe is a 30-year-old software engineer who loves hiking and reading.";

let user_info: UserInfo = broker.generate_structured(text).await?;

println!("{:?}", user_info);
// UserInfo {
//   name: "John Doe",
//   age: 30,
//   interests: ["hiking", "reading"]
// }
}

How It Works

1. Schema Definition: Mojentic uses schemars to convert your Rust struct into a JSON schema that the LLM can understand.
2. Prompt Engineering: The broker automatically appends instructions to the prompt, telling the LLM to output JSON matching the schema.
3. Validation: When the response comes back, Mojentic parses the JSON and deserializes it into your struct, performing validation (e.g., ensuring age is a number).

Advanced: Nested Schemas

You can also use nested structs for more complex data.

#![allow(unused)]
fn main() {
#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct Address {
    street: String,
    city: String,
}

#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct UserProfile {
    name: String,
    address: Address,
}
}

Summary

Structured output turns unstructured text into reliable data structures. By defining Rust structs, you can integrate LLM outputs directly into your application’s logic with type safety and validation.

Tutorial: Building Autonomous Agents

What is an Autonomous Agent?

An autonomous agent is a system that can perceive its environment, reason about how to achieve a goal, and take actions (use tools) to accomplish that goal. Unlike a simple chatbot, an agent has a loop of “Thought -> Action -> Observation”.

The Simple Recursive Agent (SRA)

Mojentic provides a SimpleRecursiveAgent pattern. This agent:

1. Receives a goal.
2. Thinks about the next step.
3. Selects a tool to use.
4. Executes the tool.
5. Observes the result.
6. Repeats until the goal is met.

Building an Agent

Let’s build an agent that can answer questions using a web search tool.

1. Setup

You’ll need the WebSearchTool (or any other tool) and a Broker.

use mojentic::prelude::*;
use mojentic::tools::WebSearchTool;
use mojentic::agents::SimpleRecursiveAgent;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize broker
    let gateway = Arc::new(OllamaGateway::new());
    let broker = LlmBroker::new("qwen3:32b", gateway, None);

    // Configure tools
    let tools: Vec<Arc<dyn LlmTool>> = vec![
        Arc::new(WebSearchTool::new(SearchProvider::Tavily)),
    ];
    
    // ...
    Ok(())
}

2. Run the Agent

#![allow(unused)]
fn main() {
let goal = "Find out who won the latest Super Bowl and tell me the score.";

let result = SimpleRecursiveAgent::run(&broker, goal, tools, None).await?;

println!("Final Answer: {}", result);
}

Step-by-Step Execution

When you run this, the agent enters a loop:

1. Thought: “I need to search for the latest Super Bowl winner.”
2. Action: Calls WebSearchTool with query “latest Super Bowl winner score”.
3. Observation: Receives search results (e.g., “Kansas City Chiefs defeated San Francisco 49ers 25-22…”).
4. Thought: “I have the information. I can now answer the user.”
5. Final Answer: “The Kansas City Chiefs won the latest Super Bowl with a score of 25-22.”

Customizing the Agent

You can customize the agent’s behavior by:

• Adding more tools: Give it file access, calculation abilities, etc.
• System Prompt: Adjust its personality or constraints.
• Max Iterations: Limit how many steps it can take to prevent infinite loops.

#![allow(unused)]
fn main() {
let config = AgentConfig {
    max_iterations: 10,
    ..Default::default()
};

SimpleRecursiveAgent::run(&broker, goal, tools, Some(config)).await?;
}

Summary

Autonomous agents allow you to solve complex, multi-step problems. By combining a reasoning loop with a set of tools, you can build systems that can interact with the world to achieve user goals.

Image Analysis

Vision-capable models can analyze images by passing image file paths along with text prompts. The framework automatically handles reading image files and encoding them as Base64 for transmission to the LLM.

Basic Usage

Attach images to a message using the with_images() method:

#![allow(unused)]
fn main() {
use mojentic::prelude::*;

let message = LlmMessage::user("Describe this image")
    .with_images(vec!["/path/to/image.jpg".to_string()]);
}

You can attach multiple images to a single message:

#![allow(unused)]
fn main() {
let message = LlmMessage::user("Compare these images")
    .with_images(vec![
        "/path/to/image1.jpg".to_string(),
        "/path/to/image2.jpg".to_string(),
    ]);
}

For the OpenAI gateway, with_images() also accepts HTTP(S) URLs and data URIs. These sources pass through unchanged in image_url.url; local files are encoded as Base64. This also applies to oMLX, which shares the adapter. Ollama image inputs remain local file paths.

Complete Example

use mojentic::prelude::*;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<()> {
    // Create gateway and broker with a vision-capable model
    let gateway = OllamaGateway::new();
    let broker = LlmBroker::new("llava:latest", Arc::new(gateway), None);

    // Create a message with image
    let message = LlmMessage::user("What's in this image?")
        .with_images(vec!["path/to/image.jpg".to_string()]);

    // Generate response
    let response = broker.generate(&[message], None, None, None).await?;
    println!("{}", response);

    Ok(())
}

Vision-Capable Models

Common vision-capable models available through Ollama:

• llava:latest - General-purpose vision model
• bakllava:latest - BakLLaVA vision model
• qwen3-vl:30b - Qwen3 vision-language model
• gemma3:27b - Gemma 3 with vision support

Pull a model before using:

ollama pull llava:latest

How It Works

When you attach images to a message:

1. File Reading: The gateway reads the image file from the specified path
2. Base64 Encoding: The image bytes are encoded as Base64 using the base64 crate
3. API Transmission: The encoded image is included in the images field of the Ollama API request
4. Model Processing: The vision-capable model analyzes both the text prompt and image(s)

Error Handling

Image processing can fail if:

• The image file doesn’t exist or isn’t readable
• The file path is invalid
• The model doesn’t support vision

Always handle errors when working with images:

#![allow(unused)]
fn main() {
match broker.generate(&[message], None, None, None).await {
    Ok(response) => println!("Response: {}", response),
    Err(e) => eprintln!("Error analyzing image: {}", e),
}
}

Supported Image Formats

The framework reads raw image bytes and passes them to the model. Supported formats depend on the specific model being used. Most vision models support common formats like JPEG and PNG.

See Also

• Examples - See image_analysis.rs for a working example
• LlmMessage API - Full message construction API
• Ollama Gateway - Gateway-specific details

Examples

Example: File Tools

This chapter demonstrates how to build tools for interacting with the file system using the Mojentic framework. The included FileTool and CodingFileTool serve as reference implementations that you can use directly or adapt for your specific needs.

Available Tools

FileTool

The FileTool provides basic file operations:

• read_file: Read content of a file
• write_file: Write content to a file
• list_dir: List directory contents
• file_exists: Check if a file exists

CodingFileTool

The CodingFileTool extends FileTool with features specifically for coding tasks:

• apply_patch: Apply a unified diff patch to a file
• replace_text: Replace specific text in a file
• search_files: Search for patterns in files

Usage

use mojentic::prelude::*;
use mojentic::tools::FileTool;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize broker
    let gateway = Arc::new(OllamaGateway::new());
    let broker = LlmBroker::new("qwen3:32b", gateway, None);

    // Register tools
    let tools: Vec<Arc<dyn LlmTool>> = vec![
        Arc::new(FileTool::default()),
    ];

    // Ask the agent to perform file operations
    let messages = vec![
        LlmMessage::user("Create a file named 'hello.txt' with the content 'Hello, World!'"),
    ];

    let response = broker.generate(&messages, Some(tools), None).await?;
    println!("{}", response);
    
    Ok(())
}

Security

By default, file tools are restricted to the current working directory. You can configure allowed paths to restrict access further.

Example: Task Management

The TaskManager is an example of how to build stateful tools that allow agents to manage ephemeral tasks. This reference implementation shows how to maintain state across tool calls.

Features

• Create Tasks: Add new tasks to the list
• List Tasks: View all current tasks and their status
• Complete Tasks: Mark tasks as done
• Prioritize: Agents can determine the order of execution

Usage

use mojentic::prelude::*;
use mojentic::tools::TaskManager;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize broker
    let gateway = Arc::new(OllamaGateway::new());
    let broker = LlmBroker::new("qwen3:32b", gateway, None);

    // Register the tool
    let tools: Vec<Arc<dyn LlmTool>> = vec![
        Arc::new(TaskManager::new()),
    ];

    // The agent can now manage its own tasks
    let messages = vec![
        LlmMessage::system("You are a helpful assistant. Use the task manager to track your work."),
        LlmMessage::user("Plan a party for 10 people."),
    ];

    let response = broker.generate(&messages, Some(tools), None).await?;
    println!("{}", response);
    
    Ok(())
}

Integration with Agents

The Task Manager is particularly powerful when combined with the IterativeProblemSolver agent, allowing it to maintain state across multiple reasoning steps.

Example: Web Search

The WebSearchTool demonstrates how to integrate external APIs into your agent’s toolset. This example implementation supports multiple providers like Tavily and Serper.

Configuration

The web search tool typically requires an API key for a search provider (e.g., Tavily, Serper).

#![allow(unused)]
fn main() {
use std::env;

env::set_var("TAVILY_API_KEY", "your-api-key");
}

Usage

use mojentic::prelude::*;
use mojentic::tools::WebSearchTool;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize broker
    let gateway = Arc::new(OllamaGateway::new());
    let broker = LlmBroker::new("qwen3:32b", gateway, None);

    // Register the tool
    let tools: Vec<Arc<dyn LlmTool>> = vec![
        Arc::new(WebSearchTool::new(SearchProvider::Tavily)),
    ];

    // Ask a question requiring up-to-date info
    let messages = vec![
        LlmMessage::user("What is the current stock price of Apple?"),
    ];

    let response = broker.generate(&messages, Some(tools), None).await?;
    println!("{}", response);
    
    Ok(())
}

Supported Providers

• Tavily: Optimized for LLM agents
• Serper: Google Search API
• DuckDuckGo: Privacy-focused search (no API key required for basic usage)

Streaming Responses

Streaming gives you a model’s response in pieces as the provider generates it. Mojentic has two streaming APIs:

• generate_stream streams text and runs tools between rounds. Use it for chat-style output where tools can take part.
• generate_stream_events streams one turn with no tools and ends with evidence of how the turn finished. Use it when truncated output must not be mistaken for a complete answer.

Streaming with tools

broker.generate_stream returns a stream of Result<String> chunks:

use futures::StreamExt;
use mojentic::llm::gateways::OllamaGateway;
use mojentic::llm::{LlmBroker, LlmMessage};
use std::sync::Arc;

#[tokio::main]
async fn main() {
    let broker = LlmBroker::new("qwen3:32b", Arc::new(OllamaGateway::new()), None);
    let messages = vec![LlmMessage::user("Tell me a story.")];

    let mut stream = broker.generate_stream(&messages, None, None, None);
    while let Some(result) = stream.next().await {
        match result {
            Ok(chunk) => print!("{}", chunk),
            Err(e) => eprintln!("Error: {}", e),
        }
    }
}

Pass tools to let the model call them. The broker stops streaming to run the tools, then streams the model’s follow-up response:

#![allow(unused)]
fn main() {
let tools: Vec<Box<dyn LlmTool>> = vec![Box::new(SimpleDateTool)];
let mut stream = broker.generate_stream(&messages, Some(&tools), None, None);
}

generate_stream does not tell you why the provider stopped. A response cut off at the token limit looks the same as a finished one.

Single-turn streaming with terminal completion evidence

broker.generate_stream_events(&messages, config, correlation_id) returns a stream of StreamEvent values:

Every stream ends with exactly one terminal event, and nothing follows it. Completed carries a ResponseEvidence with the finish reason, usage (None when the provider did not report it), provider model and provider metadata (for example Ollama’s total_duration and eval_duration).

#![allow(unused)]
fn main() {
use futures::StreamExt;
use mojentic::llm::{StreamEvent, StreamEventError};

let mut events = broker.generate_stream_events(&messages, None, None);
while let Some(event) = events.next().await {
    match event {
        StreamEvent::Content(text) => print!("{text}"),
        StreamEvent::Completed(evidence) => println!("\nusage: {:?}", evidence.usage),
        StreamEvent::Error(StreamEventError::IncompleteCompletion(evidence)) => {
            eprintln!("\ncut off: {:?}", evidence.finish_reason)
        }
        StreamEvent::Error(error) => eprintln!("\nfailed: {error}"),
        _ => {}
    }
}
}

Content that arrived before an error is evidence of what the provider sent. It is not a result. Mojentic does not continue, retry or estimate anything. What to do with incomplete output is your decision.

When a turn is complete

• OpenAI-compatible: the stream needs a finish_reason of stop and then the data: [DONE] marker.
• Ollama: the final frame needs done: true and a done_reason of stop. Ollama servers too old to send done_reason always end with IncompleteCompletion, so they cannot use this API.

Errors

Behaviour

• One HTTP request. Tool iterations are forced to zero. No retry, no continuation, no recursion.
• Dropping the stream, or breaking out of the loop that reads it, cancels the request.
• The OpenAI and Ollama gateways support this API. Other gateways yield a single StreamEventsUnsupported error. A custom gateway opts in by implementing LlmGateway::complete_stream_events.
• CompletionConfig.response_format is forwarded as described in Using LLMs.
• OpenAI requests set stream_options: {"include_usage": true} so the provider reports usage.

Tracing

With a tracer, the broker records the LLM call when the request starts. When the terminal event arrives, success or failure, it records the response: the content received so far and the evidence (usage, provider_model, finish_reason, metadata). If you stop reading before the terminal event, the call is traced and no response is. That is not an error. An unsupported gateway records nothing.

Provider-specific stream detail

The gateways’ lower-level complete_stream also yields StreamChunk::Thinking, StreamChunk::Progress and StreamChunk::Metrics for Ollama. The events API exposes client timing through StreamEvent::Metrics, with provider facts on terminal evidence; it has no thinking or frame-progress event. Thinking text is not assistant content.

The oMLX gateway’s complete_stream yields StreamChunk::Thinking for reasoning deltas and terminal client Metrics chunks, with original provider usage retained. It does not emit Ollama frame-progress counters. See oMLX Gateway.

Client timing

Every dispatched stream reports client milliseconds in StreamMetrics: time_to_first_token_ms, first_token_kind and request_duration_ms. The first kind is thinking, content or tool_call. A single frame uses that precedence; argument fragments count before a complete tool call can be decoded. Empty and keepalive-only streams have no first-token value or kind. Duration is available on both success and failure. Dropping a stream cancels it; no terminal metric is promised to a consumer that stops polling.

Measurements use a monotonic clock and start when the request is dispatched, so constructing an unpolled stream adds no latency. Admitted retries share that clock, including admission and backoff. Sending tool results is a new completion with a fresh clock. No measurement estimates provider inference time.

Raw complete_stream emits StreamChunk::Metrics before a dispatched error or at completion. complete_stream_events emits nonterminal StreamEvent::Metrics immediately before Completed or Error; keep polling for the terminal outcome. Completed and incomplete-completion evidence also carry client fields in metadata. Preserve the metrics event when collecting errors that have no provider evidence. LlmBroker::generate_stream and ChatSession::send_stream expose terminal timing through LlmResponseTracerEvent.evidence.metadata when tracing is enabled, including when the HTTP body is interrupted after a semantic token. The original error still terminates the text stream. A failed session retains the user message and existing history without adding a partial assistant response. Provider usage and durations keep their original fields and units; client timing does not replace oMLX’s usage timings or Ollama’s nanosecond durations.

#![allow(unused)]
fn main() {
use futures::StreamExt;
use mojentic::llm::{CompletionConfig, LlmGateway, LlmMessage, StreamEvent};
use mojentic::llm::gateways::OmlxGateway;
async fn example() -> Result<(), Box<dyn std::error::Error>> {
let gateway = OmlxGateway::new();
let messages = [LlmMessage::user("Hello")];
let config = CompletionConfig::default();
let mut stream = gateway.complete_stream_events("my-model", &messages, &config)?;
while let Some(event) = stream.next().await {
    match event {
        StreamEvent::Metrics(metrics) => println!("{:?} ms", metrics.time_to_first_token_ms),
        StreamEvent::Content(text) => print!("{text}"),
        StreamEvent::Completed(_) => break,
        StreamEvent::Error(error) => return Err(error.into()),
        _ => {},
    }
}
Ok(())
}
}

Recovery’s native stream keeps its original provider Metrics events. Inspect Completed(result).report.stream_metrics or Failed(error).stream_metrics for final client measurements, including interrupted output. Their raw provider usage requires explicit inspection and is omitted from default safe recovery summaries. The compatibility complete_stream maps final metrics to StreamChunk::Metrics. Broker/session text APIs continue yielding text; enable tracing to read each completion’s client fields and stream_metrics from response evidence, including separate completed tool turns. A custom with_stream_clock(Arc<dyn StreamClock>) on a maintained gateway supports deterministic measurements; its clock must be monotonic. Recovery retains the inner gateway’s measurement clock.

Embeddings

Embeddings allow you to convert text into vector representations, which are useful for semantic search, clustering, and similarity comparisons.

Setup

You need an embedding model. Ollama supports models like mxbai-embed-large or nomic-embed-text.

use mojentic::prelude::*;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize gateway
    let gateway = Arc::new(EmbeddingsGateway::new("mxbai-embed-large"));
    
    Ok(())
}

Generating Embeddings

#![allow(unused)]
fn main() {
let text = "The quick brown fox jumps over the lazy dog.";
let vector = gateway.embed(text).await?;

println!("{:?}", &vector[0..5]);
// => [0.123, -0.456, ...]
}

Batch Processing

You can embed multiple texts at once:

#![allow(unused)]
fn main() {
let texts = vec!["Hello", "World"];
let vectors = gateway.embed_batch(texts).await?;
}

Cosine Similarity

Mojentic provides utilities to calculate similarity between vectors:

#![allow(unused)]
fn main() {
use mojentic::utils::math::cosine_similarity;

let similarity = cosine_similarity(&vector1, &vector2);
}

OpenAI long text

The OpenAI gateway splits long text into parts of at most 8191 tokens with the cl100k_base tokenizer. It weights each part vector by the number of tokens in that part, then normalizes the result to length 1. A short trailing part has less weight than a full part. For a single part, the gateway returns the provider vector unchanged.

Layer 1 - Core

Layer 1 provides primitives for interacting with LLMs and composing messages, tools, and structured outputs.

Components

• Messages: typed containers for role + content.
• Completion Config: tuning parameters (temperature, max tokens, etc.).
• Tools: callable functions surfaced to the LLM.
• Gateway: abstraction over model providers.

Proceed through the subsections for practical usage.

The Basics

A quick orientation to the core types and flows.

• Create messages (system, user, assistant)
• Configure a completion
• Send to a gateway via the broker

use mojentic::llm::{Broker, CompletionConfig, Message};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let broker = Broker::new()?;
    let cfg = CompletionConfig::default();
    let msgs = vec![
        Message::system("You are concise"),
        Message::user("Explain Rust lifetimes in one line"),
    ];
    let out = broker.chat(cfg, msgs).await?;
    println!("{}", out.text());
    Ok(())
}

Message Composers

Helpers for building prompts and role-structured messages.

• System prompts set behavior
• User messages provide tasks
• Assistant messages chain context

Tip: keep messages small and focused; prefer function/tool calls for larger payloads.

Simple Text Generation

The minimal path from prompt to text.

use mojentic::llm::{Broker, CompletionConfig, Message};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let resp = Broker::new()?
        .chat(CompletionConfig::default(), [Message::user("Haiku about Rust")])
        .await?;
    println!("{}", resp.text());
    Ok(())
}

Reasoning controls and native history

Reasoning has three independent controls: the requested effort, the selected model capability profile, and retention of native assistant reasoning. Output budgets and application stop policies remain separate.

Configure a known model

#![allow(unused)]
fn main() {
use mojentic::llm::{CompletionConfig, ReasoningProfile};
use mojentic::llm::gateway::ReasoningEffort;

let config = CompletionConfig {
    reasoning_profile: Some(ReasoningProfile::qwen3_8()),
    reasoning_effort: Some(ReasoningEffort::Medium),
    preserve_reasoning: true,
    ..Default::default()
};
config.validate_reasoning().expect("supported model controls");
}

Profiles are explicit capabilities, not model-name matching. The Qwen3.8 profile accepts Disabled, Low, Medium, and XHigh. High is not an alias for XHigh and is rejected before HTTP. Applications can construct another ReasoningProfile with its exact supported controls and native-history capability.

An omitted profile delegates model validation to the provider. An omitted effort leaves the provider/model default unspecified, even when a profile is selected. For comparable benchmarks, specify effort explicitly. Profiles do not claim to know server overrides or the final rendered prompt.

Provider mappings

Enabled requests boolean enablement without choosing a depth. Use it for models that accept boolean controls; the Qwen3.8 profile deliberately requires a named level or omission instead. Named values are never collapsed to true. Provider or model validation may reject an unsupported name when no profile is supplied. Ordinary, streaming, and structured output share the same mapping.

The OpenAI gateway retains its existing model registry for named effort support. It can send XHigh to reasoning models; individual models may reject it. Boolean Enabled and automatic native-history preservation fail explicitly in that gateway. Its legacy Disabled behavior is unchanged. The OpenAI adapter does not send the oMLX-specific reasoning_content extension to OpenAI.

Preserve completed reasoning

LlmMessage now carries optional reasoning_content separately from content. The Ollama gateway sends it as assistant thinking; oMLX sends assistant reasoning_content. Existing JSON without this field remains readable, and an incoming thinking field is accepted as an alias. Constructors initialize it to None; with_reasoning attaches it explicitly.

#![allow(unused)]
fn main() {
use mojentic::llm::LlmMessage;
let message = LlmMessage::assistant("I will inspect the file.")
    .with_reasoning("Inspect before editing.");
assert!(message.reasoning_content.is_some());
}

preserve_reasoning defaults to false. When enabled, the broker retains reasoning from completed ordinary or streaming responses that request tools, paired with the correct assistant message and tool results. It does not emit hidden reasoning as visible response text. oMLX also requests chat_template_kwargs.preserve_thinking=true; forced server settings can still take precedence. Disabling retention does not disable reasoning generation.

Manually supplied message history belongs to the caller: the two local gateways serialize native assistant reasoning already present even when automatic broker retention is off. Callers that manage their own loop must retain responses explicitly. ChatSession’s convenience text-only history API is unchanged.

A failed or cancelled stream is not a completed assistant turn. This feature does not fabricate a closing think block, continue a partial response, or convert a partial reasoning tail to user instructions. Applications must define and measure that separate policy. Retention can increase context size and preserve wrong plans; compatibility tests do not establish better task performance.

Prototype migration

This unreleased Rust prototype adds fields to LlmMessage and CompletionConfig, and variants to ReasoningEffort. Rust struct literals and exhaustive matches need updating; message constructors and CompletionConfig defaults ease migration. Other language ports are unchanged. Consume the audited Git revision during validation; a coordinated release and final version decision are separate work.

Building Tools

Creating Custom Tools

Tools extend LLM capabilities by providing functions they can call. All tools must implement the LlmTool trait.

Basic Tool Structure

#![allow(unused)]
fn main() {
use mojentic::prelude::*;
use serde_json::{json, Value};
use std::collections::HashMap;

struct CalculatorTool;

impl LlmTool for CalculatorTool {
    fn run(&self, args: &HashMap<String, Value>) -> mojentic::Result<Value> {
        let operation = args.get("operation")
            .and_then(|v| v.as_str())
            .ok_or_else(|| mojentic::MojenticError::ToolError(
                "Missing operation parameter".to_string()
            ))?;

        let a = args.get("a")
            .and_then(|v| v.as_f64())
            .ok_or_else(|| mojentic::MojenticError::ToolError(
                "Missing 'a' parameter".to_string()
            ))?;

        let b = args.get("b")
            .and_then(|v| v.as_f64())
            .ok_or_else(|| mojentic::MojenticError::ToolError(
                "Missing 'b' parameter".to_string()
            ))?;

        let result = match operation {
            "add" => a + b,
            "subtract" => a - b,
            "multiply" => a * b,
            "divide" => {
                if b == 0.0 {
                    return Err(mojentic::MojenticError::ToolError(
                        "Division by zero".to_string()
                    ));
                }
                a / b
            }
            _ => return Err(mojentic::MojenticError::ToolError(
                format!("Unknown operation: {}", operation)
            )),
        };

        Ok(json!({
            "result": result,
            "operation": operation,
            "a": a,
            "b": b
        }))
    }

    fn descriptor(&self) -> ToolDescriptor {
        ToolDescriptor {
            r#type: "function".to_string(),
            function: FunctionDescriptor {
                name: "calculator".to_string(),
                description: "Perform basic arithmetic operations".to_string(),
                parameters: json!({
                    "type": "object",
                    "properties": {
                        "operation": {
                            "type": "string",
                            "enum": ["add", "subtract", "multiply", "divide"],
                            "description": "The operation to perform"
                        },
                        "a": {
                            "type": "number",
                            "description": "First operand"
                        },
                        "b": {
                            "type": "number",
                            "description": "Second operand"
                        }
                    },
                    "required": ["operation", "a", "b"]
                }),
            },
        }
    }
}
}

Using Custom Tools

#[tokio::main]
async fn main() -> mojentic::Result<()> {
    let gateway = OllamaGateway::new();
    let broker = LlmBroker::new("qwen3:32b", Arc::new(gateway));

    let tools: Vec<Box<dyn LlmTool>> = vec![
        Box::new(CalculatorTool),
    ];

    let messages = vec![
        LlmMessage::user("What is 42 multiplied by 17?")
    ];

    let response = broker.generate(&messages, Some(&tools), None).await?;
    println!("{}", response);

    Ok(())
}

Testing Custom Tools

Create tests for your tools:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_calculator_add() {
        let tool = CalculatorTool;
        let mut args = HashMap::new();
        args.insert("operation".to_string(), json!("add"));
        args.insert("a".to_string(), json!(5.0));
        args.insert("b".to_string(), json!(3.0));

        let result = tool.run(&args).unwrap();
        assert_eq!(result["result"], 8.0);
    }

    #[test]
    fn test_calculator_divide_by_zero() {
        let tool = CalculatorTool;
        let mut args = HashMap::new();
        args.insert("operation".to_string(), json!("divide"));
        args.insert("a".to_string(), json!(10.0));
        args.insert("b".to_string(), json!(0.0));

        assert!(tool.run(&args).is_err());
    }
}
}

Best Practices

1. Error Handling: Always use proper error types, never panic in library code
2. Documentation: Add rustdoc comments to your tool implementations
3. Testing: Write comprehensive tests for your tools
4. Type Safety: Validate all input parameters
5. JSON Schema: Provide clear, detailed parameter descriptions

Tool Usage

Let models invoke your tools to retrieve data or perform actions.

Patterns:

• Stateless RPC-like tools
• Stateful tools with access to context
• Safety: validate inputs, timeouts, retries

Parallel Tool Execution

Mojentic factors tool-batch execution behind a ToolRunner trait so brokers don’t grow their own concurrency code:

LlmBroker::new wires SerialToolRunner by default; opt in to parallel fan-out with LlmBroker::with_tool_runner:

use mojentic::llm::{LlmBroker, tools::ParallelToolRunner};
use std::sync::Arc;

let broker = LlmBroker::with_tool_runner(
    "qwen3:32b",
    gateway,
    None,
    Arc::new(ParallelToolRunner::default()),
);

Cancellation

LlmTool::run accepts a &ToolRunCtx carrying a tokio_util::sync::CancellationToken. Long-running async tools can observe it between work units and abort early:

async fn run(&self, args: &HashMap<String, Value>, ctx: &ToolRunCtx) -> Result<Value> {
    for chunk in big_input(args) {
        if ctx.cancel.is_cancelled() {
            return Err(MojenticError::Cancelled);
        }
        process(chunk).await?;
    }
    Ok(json!({ "done": true }))
}

Tools that ignore the context continue to work unchanged. The SerialToolRunner and ParallelToolRunner themselves surface a cancelled batch as ToolCallOutcome { ok: false, error: Some("Cancelled"), .. } without invoking the tool.

Batch tracer event

LlmBroker and RealtimeVoiceBroker both emit a ToolBatchTracerEvent alongside the per-call ToolCallTracerEvents whenever a batch executes more than one tool, so observers can measure parallelism gains.

Realtime Voice

mojentic::realtime::RealtimeVoiceBroker opens duplex voice + tool sessions against realtime-capable providers (currently OpenAI’s Realtime API over WebSocket). It mirrors the Python, Elixir, and TypeScript ports — a long-lived broker, a per-session driver that owns a WebSocket, and a vendor-neutral event enum you consume.

30-second example (text mode)

use mojentic::realtime::{
    RealtimeVoiceBroker, RealtimeVoiceConfig, RealtimeEvent,
    RealtimeModality, OpenAIRealtimeGateway, OpenAIRealtimeGatewayOptions,
};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let gateway = Arc::new(OpenAIRealtimeGateway::new(
        OpenAIRealtimeGatewayOptions::default(),
    )?);

    let config = RealtimeVoiceConfig {
        modalities: Some(vec![RealtimeModality::Text]),
        instructions: Some("You are a concise assistant.".to_string()),
        ..Default::default()
    };

    let broker = RealtimeVoiceBroker::new("gpt-realtime", gateway, config, None, None);
    let mut session = broker.connect().await?;
    session.send_text("What's the capital of Canada?")?;

    while let Some(event) = session.next_event().await {
        match event {
            RealtimeEvent::AssistantTextDelta { delta, .. } => print!("{delta}"),
            RealtimeEvent::AssistantTurnCompleted { .. } => {
                println!();
                break;
            }
            RealtimeEvent::Error { error, .. } => {
                eprintln!("[error] {error}");
                break;
            }
            _ => {}
        }
    }

    session.close()?;
    Ok(())
}

Events

RealtimeEvent is an enum with 23 variants spanning session lifecycle, user speech, assistant output, tool calls, and control. Pattern match on it in match arms.

Tools

Pass tools (any Box<dyn LlmTool>) via RealtimeVoiceConfig::tools. The session dispatches them through ParallelToolRunner by default — multiple function_call items in one assistant turn execute concurrently via FuturesUnordered + Semaphore (capped at 4) and the results are batched back as function_call_output items before the next response.create.

Audio I/O

The library is hardware-free. Use session.send_audio_frame(Vec<i16>) to push samples and consume AssistantAudioDelta events on the way back. For live device I/O integrate a platform library (e.g. cpal) at the boundary; the session API stays the same.

Interruption

session.interrupt() cancels the in-flight turn and aborts running tools by signalling their CancellationToken. Server-driven barge-in (input_audio_buffer.speech_started) does the same automatically.

RealtimeVoiceConfig::on_interrupt controls what happens to tool outputs already in flight when the cancel lands: Drop (default), SubmitCompletedOnly, or Submit.

Agent Delegation with ToolWrapper

The ToolWrapper enables agent delegation patterns by wrapping an agent (broker + tools + behavior) as a tool that can be used by other agents. This allows you to build hierarchical agent systems where coordinator agents can delegate specialized tasks to expert agents.

Overview

ToolWrapper wraps three core components:

• Broker: The LLM interface for the agent
• Tools: The tools available to the agent
• Behavior: The system message defining the agent’s personality and capabilities

The wrapped agent appears as a standard tool with a single input parameter, making it easy for other agents to delegate work.

Basic Usage

#![allow(unused)]
fn main() {
use mojentic::llm::{LlmBroker, LlmTool, ToolWrapper};
use mojentic::llm::gateways::OllamaGateway;
use std::sync::Arc;

// Create a specialist agent
let gateway = Arc::new(OllamaGateway::default());
let specialist_broker = Arc::new(LlmBroker::new("qwen2.5:7b", gateway.clone()));

let specialist_tools: Vec<Box<dyn LlmTool>> = vec![
    // Add specialist's tools here
];

let specialist_behaviour =
    "You are a specialist in temporal reasoning and date calculations.";

// Wrap the specialist as a tool
let specialist_tool = ToolWrapper::new(
    specialist_broker,
    specialist_tools,
    specialist_behaviour,
    "temporal_specialist",
    "A specialist that handles date and time-related queries."
);

// Use the specialist in a coordinator agent
let coordinator_broker = LlmBroker::new("qwen2.5:14b", gateway);
let coordinator_tools: Vec<Box<dyn LlmTool>> = vec![
    Box::new(specialist_tool)
];

// The coordinator can now delegate to the specialist
}

How It Works

1. Tool Descriptor: The ToolWrapper generates a tool descriptor with:

   • Function name (e.g., “temporal_specialist”)
   • Description (what the agent does)
   • Single input parameter for instructions

2. Execution Flow: When called:

   • Extracts the input parameter
   • Creates initial messages with the agent’s behavior (system message)
   • Appends the input as a user message
   • Calls the agent’s broker with its tools
   • Returns the agent’s response

3. Delegation Pattern: The coordinator agent sees specialist agents as tools and decides when to delegate based on the task requirements.

Example: Multi-Agent System

See examples/broker_as_tool.rs for a complete example demonstrating:

• Creating specialist agents with specific tools and behaviors
• Wrapping specialists as tools
• Building a coordinator that delegates appropriately
• Testing different query types

cargo run --example broker_as_tool

Design Considerations

Agent Ownership

ToolWrapper uses Arc<LlmBroker> to handle shared ownership of the broker. This allows multiple references to the same broker if needed.

Async Execution

The run method is synchronous (required by the LlmTool trait) but internally handles async operations using tokio::task::block_in_place. This requires tests to use the multi-threaded runtime:

#![allow(unused)]
fn main() {
#[tokio::test(flavor = "multi_thread")]
async fn test_tool_wrapper() {
    // Test code here
}
}

Tool Isolation

Each wrapped agent maintains its own:

• Tool set
• Behavior/personality
• Conversation context (per invocation)

This ensures clean separation between agents and prevents context bleeding.

Common Patterns

Specialist Agent Pattern

Create focused agents with specific expertise:

#![allow(unused)]
fn main() {
// Data analysis specialist
let data_analyst = ToolWrapper::new(
    analyst_broker,
    vec![Box::new(DataQueryTool), Box::new(VisualizationTool)],
    "You are a data analyst specializing in statistical analysis.",
    "data_analyst",
    "Analyzes data and provides statistical insights."
);

// Writing specialist
let writer = ToolWrapper::new(
    writer_broker,
    vec![Box::new(GrammarCheckTool), Box::new(StyleGuideTool)],
    "You are a professional writer and editor.",
    "writer",
    "Improves and edits written content."
);
}

Coordinator Pattern

Build a coordinator that orchestrates multiple specialists:

#![allow(unused)]
fn main() {
let coordinator = LlmBroker::new("qwen2.5:32b", gateway);
let tools: Vec<Box<dyn LlmTool>> = vec![
    Box::new(data_analyst),
    Box::new(writer),
    Box::new(researcher),
];

// Coordinator decides which specialist to use based on the task
}

Hierarchical Agents

Create multi-level hierarchies:

#![allow(unused)]
fn main() {
// Level 1: Base specialists
let math_specialist = ToolWrapper::new(...);
let physics_specialist = ToolWrapper::new(...);

// Level 2: Domain coordinator
let science_coordinator = ToolWrapper::new(
    coordinator_broker,
    vec![Box::new(math_specialist), Box::new(physics_specialist)],
    "You coordinate math and physics specialists.",
    "science_coordinator",
    "Handles scientific queries."
);

// Level 3: Top-level coordinator
let main_coordinator = LlmBroker::new("qwen2.5:32b", gateway);
let main_tools: Vec<Box<dyn LlmTool>> = vec![
    Box::new(science_coordinator),
    // Other domain coordinators...
];
}

Best Practices

1. Clear Specialization: Give each agent a well-defined area of expertise
2. Descriptive Names: Use clear, descriptive names for tools (e.g., “temporal_specialist”, not “agent1”)
3. Comprehensive Descriptions: Write detailed descriptions so coordinators understand when to delegate
4. Model Selection: Use appropriate model sizes for each role:
   • Specialists: Smaller, faster models (7B-14B)
   • Coordinators: Larger, more capable models (32B+)
5. Tool Composition: Specialists should have tools relevant to their domain
6. Testing: Test both individual specialists and the full delegation chain

Limitations

• Synchronous Interface: The LlmTool trait requires synchronous run methods, though async operations happen internally
• No Streaming: Tool calls don’t support streaming responses (limitation of current tool architecture)
• Context: Each tool invocation is independent; no automatic context sharing between calls
• Error Handling: Tool errors propagate up to the coordinator; implement appropriate error handling

See Also

• Tool Usage - General tool usage patterns
• Building Tools - Creating custom tools
• Chat Sessions with Tools - Using tools in chat sessions

Chat Sessions with Tools

Combine conversational context with tool calling for grounded, actionable responses. ChatSession seamlessly integrates with Mojentic’s tool system to enable function calling within conversations.

Overview

When tools are provided to a ChatSession, the LLM can call them during the conversation. The broker handles:

1. Detecting tool call requests
2. Executing the appropriate tools
3. Adding tool results to the conversation
4. Continuing the conversation with the LLM

This happens transparently - you just call send() as normal.

Basic Tool Integration

Add tools when building the session:

use mojentic::llm::{ChatSession, LlmBroker};
use mojentic::llm::gateways::OllamaGateway;
use mojentic::llm::tools::simple_date_tool::SimpleDateTool;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let gateway = Arc::new(OllamaGateway::default());
    let broker = LlmBroker::new("qwen3:32b", gateway);

    // Create tools
    let tools: Vec<Box<dyn mojentic::llm::LlmTool>> = vec![
        Box::new(SimpleDateTool),
    ];

    // Build session with tools
    let mut session = ChatSession::builder(broker)
        .system_prompt(
            "You are a helpful assistant. When users ask about dates, \
             use the resolve_date tool to convert relative dates to absolute dates."
        )
        .tools(tools)
        .build();

    // The LLM will automatically use the tool when needed
    let response = session.send("What is tomorrow's date?").await?;
    println!("{}", response);

    Ok(())
}

How It Works

1. User sends a message: session.send("What is tomorrow's date?")
2. LLM recognizes need for tool: Generates a tool call request
3. Broker executes tool: Calls SimpleDateTool with appropriate arguments
4. Tool result added to history: As a tool response message
5. LLM generates final response: Using the tool’s output
6. Response returned to user: “Tomorrow’s date is 2025-11-14”

All of this happens in a single send() call.

Multiple Tools

Provide multiple tools for different capabilities:

#![allow(unused)]
fn main() {
use mojentic::llm::tools::simple_date_tool::SimpleDateTool;
// Import other tools...

let tools: Vec<Box<dyn mojentic::llm::LlmTool>> = vec![
    Box::new(SimpleDateTool),
    Box::new(CalculatorTool),
    Box::new(WeatherTool),
];

let mut session = ChatSession::builder(broker)
    .system_prompt(
        "You are a helpful assistant with access to date resolution, \
         calculation, and weather information tools. Use them when appropriate."
    )
    .tools(tools)
    .build();
}

The LLM will choose which tool to use based on the user’s question.

Tool Call History

Tool calls and results are part of the conversation history:

#![allow(unused)]
fn main() {
// After a tool call
for msg in session.messages() {
    match msg.role() {
        MessageRole::User => println!("User: {}", msg.content().unwrap_or("")),
        MessageRole::Assistant => {
            if let Some(tool_calls) = &msg.message.tool_calls {
                println!("Assistant called tool: {:?}", tool_calls);
            } else {
                println!("Assistant: {}", msg.content().unwrap_or(""));
            }
        }
        MessageRole::Tool => println!("Tool result: {}", msg.content().unwrap_or("")),
        _ => {}
    }
}
}

Context Management with Tools

Tool calls and results consume tokens. The session manages this automatically:

#![allow(unused)]
fn main() {
let mut session = ChatSession::builder(broker)
    .max_context(8192)
    .tools(tools)
    .build();

// Even with tool calls, context trimming works
session.send("What's tomorrow?").await?;  // May trigger tool call
session.send("And the day after?").await?;  // May trigger another tool call

// Old messages (including old tool calls) are trimmed when needed
println!("Total tokens: {}", session.total_tokens());
}

Creating Custom Tools

Build your own tools for specific needs:

#![allow(unused)]
fn main() {
use mojentic::llm::tools::{LlmTool, ToolDescriptor, FunctionDescriptor};
use serde_json::{json, Value};
use std::collections::HashMap;

struct WeatherTool;

impl LlmTool for WeatherTool {
    fn run(&self, args: &HashMap<String, Value>) -> mojentic::error::Result<Value> {
        let city = args.get("city")
            .and_then(|v| v.as_str())
            .ok_or_else(|| mojentic::error::MojenticError::ToolError(
                "Missing city argument".to_string()
            ))?;

        // In reality, you'd call a weather API here
        Ok(json!({
            "city": city,
            "temperature": "72°F",
            "conditions": "Sunny"
        }))
    }

    fn descriptor(&self) -> ToolDescriptor {
        ToolDescriptor {
            r#type: "function".to_string(),
            function: FunctionDescriptor {
                name: "get_weather".to_string(),
                description: "Get current weather for a city".to_string(),
                parameters: json!({
                    "type": "object",
                    "properties": {
                        "city": {
                            "type": "string",
                            "description": "The city name"
                        }
                    },
                    "required": ["city"]
                }),
            },
        }
    }
}
}

Then use it:

#![allow(unused)]
fn main() {
let tools: Vec<Box<dyn mojentic::llm::LlmTool>> = vec![
    Box::new(WeatherTool),
];

let mut session = ChatSession::builder(broker)
    .system_prompt("You can check weather using the get_weather tool.")
    .tools(tools)
    .build();

let response = session.send("What's the weather in San Francisco?").await?;
}

Interactive Example

Run the complete example:

cargo run --example chat_session_with_tool

This demonstrates:

• Tool integration in conversation
• Automatic tool invocation
• Natural multi-turn dialogue with tool results

Best Practices

1. Clear tool descriptions: Help the LLM understand when to use each tool
2. Descriptive system prompts: Explain what tools are available
3. Error handling in tools: Return meaningful errors for invalid inputs
4. Tool result formatting: Return structured JSON that’s easy for the LLM to interpret
5. Monitor token usage: Tools can add significant tokens to the conversation

Debugging Tool Calls

Enable tracing to see tool execution:

#![allow(unused)]
fn main() {
// Initialize tracing
tracing_subscriber::fmt::init();

// Tool calls will be logged
let response = session.send("What's tomorrow?").await?;
}

You’ll see logs like:

INFO Tool calls requested: 1
INFO Executing tool: resolve_date

Limitations

• Tool calls count toward the context window
• Very long tool results may need truncation
• Not all models support tool calling equally well
• Tool execution is synchronous (async tools not yet supported)

Next Steps

• Learn about Building Tools
• Explore Tool Usage patterns
• Read about Simple Text Generation

Model context windows

Use provider metadata to learn the advertised capacity before deciding your application’s token budget. Lookup does not configure a completion or load a model. Unknown capacities stay unknown rather than acquiring a guessed limit.

#![allow(unused)]
fn main() {
use mojentic::llm::{CompletionConfig, LlmGateway, gateways::OllamaGateway};

async fn example() -> mojentic::error::Result<()> {
let gateway = OllamaGateway::new();
let window = gateway.get_model_context_window("qwen3.8:27b-mlx").await?;
if let Some(window) = &window {
    println!("{} tokens from {}", window.max_context_tokens, window.field);
}
let config = CompletionConfig { num_ctx: Some(8192), ..Default::default() };
let effective = OllamaGateway::effective_context_window(window.as_ref(), &config);
if let Some(tokens) = effective {
    assert!(tokens <= 8192);
}
Ok(())
}
}

Ok(Some(ModelContextWindow)) contains a positive max_context_tokens, its field provenance, and optional Ollama model_num_ctx. Ok(None) means the provider did not advertise a window for that model. Transport failures, non-success statuses (including an unknown-model Ollama 404), malformed JSON and invalid advertised windows return the existing gateway errors.

OpenAI’s completion capability inference for unknown model names is unchanged; metadata never uses that inference to guess a context window. Third-party LlmGateway implementations inherit Ok(None) without needing a new method. RecoveryGateway delegates metadata lookup to its wrapped adapter; lookup does not opt into completion retries or change admission behavior.

An Ollama model whose parameters carry a num_ctx line but whose model_info has no context_length still has a usable window: Ollama runs it at that num_ctx when a request sends none. Since 3.0.1 the lookup returns that value as max_context_tokens with field parameters.num_ctx and the same value in model_num_ctx, instead of None. When both are present, max_context_tokens stays the context_length and model_num_ctx carries the parameter, as before.

Ollama’s effective-window helper first requires an advertised window. It clamps an explicit config.num_ctx to that maximum, otherwise clamps the model’s num_ctx parameter, otherwise returns the maximum. If no window is known it returns None, even when configuration sets num_ctx. The helper is pure and does not modify the configuration or later HTTP payloads.

Without either num_ctx, the result is an upper bound: the client cannot see the server’s OLLAMA_CONTEXT_LENGTH or built-in default. Applications needing an exact loaded context can inspect GET /api/ps after their first request. Budgeting, pressure bands and eviction remain application policy.

This implements section 1 of MODEL-METADATA-2026-10.md. Provider evidence is the coordinating Ollama 0.40.5 captures and oMLX models capture. The local executable contract is tests/contracts/model_metadata_section1_v1.json; cargo test --all-features --test model_metadata verifies the public adapters against it. The same public tests run in a fresh consumer of the packaged crate. External dependent adoption follows coordinated publication under Stacey’s library self-validation rule; no live provider verification is claimed.

Client first-token and request-duration measurements are described in Streaming. Lookup itself does not start a stream clock.

OpenAI Model Registry

Overview

The OpenAI Model Registry is a centralized system for managing model-specific configurations, capabilities, and parameter requirements for OpenAI models. It provides a compile-time type-safe registry that informs the gateway about which parameters and features each model supports.

This registry is essential for:

• Determining which models support tools, streaming, or vision capabilities
• Selecting the correct token limit parameter (max_tokens vs max_completion_tokens)
• Understanding temperature support for reasoning models
• Tracking which API endpoints each model supports
• Providing sensible defaults when working with unknown models

Model Types

The registry categorizes models into four distinct types:

Reasoning Models

Models that use internal reasoning steps before generating responses. These models use max_completion_tokens instead of max_tokens.

#![allow(unused)]
fn main() {
ModelType::Reasoning
}

Examples: o1, o1-mini, o3, o3-mini, gpt-5, gpt-5.1, gpt-5.4, gpt-5.5

Chat Models

Standard chat completion models that use max_tokens for output control.

#![allow(unused)]
fn main() {
ModelType::Chat
}

Examples: gpt-4o, gpt-4-turbo, gpt-3.5-turbo

Embedding Models

Models designed for generating text embeddings.

#![allow(unused)]
fn main() {
ModelType::Embedding
}

Examples: text-embedding-3-small, text-embedding-3-large

Moderation Models

Models for content moderation and safety classification.

#![allow(unused)]
fn main() {
ModelType::Moderation
}

Examples: omni-moderation-latest, text-moderation-latest

Model Capabilities

The ModelCapabilities struct defines what each model can do. Here are all the fields:

Core Capabilities

• model_type: ModelType - The type of model (Reasoning, Chat, Embedding, or Moderation)
• supports_tools: bool - Whether the model supports function/tool calling
• supports_streaming: bool - Whether the model supports streaming responses
• supports_vision: bool - Whether the model can process image inputs

Token Limits

• max_context_tokens: Option<u32> - Maximum input context window size in tokens
• max_output_tokens: Option<u32> - Maximum number of output tokens the model can generate

Temperature Support

• supported_temperatures: Option<Vec<f32>> - Temperature constraints:
  • None - Accepts any temperature value (most chat models)
  • Some(vec![]) - Does not accept temperature parameter (not exposed to users)
  • Some(vec![1.0]) - Only accepts temperature 1.0 (reasoning models like o1, o3)

API Endpoint Support

• supports_chat_api: bool - Supports /v1/chat/completions endpoint
• supports_completions_api: bool - Supports /v1/completions endpoint
• supports_responses_api: bool - Supports /v1/responses endpoint

Note: The OpenAI gateway currently only calls the Chat API. These flags are informational and available for future gateway enhancements.

API Endpoint Categories

OpenAI models support different API endpoints based on their design and capabilities:

Chat API (Most Common)

The /v1/chat/completions endpoint is supported by most modern models:

• All GPT-4 variants (except legacy completions models)
• Reasoning models (o1, o3, o4-mini)
• Base GPT-5 models

Completions API (Legacy)

The /v1/completions endpoint is supported by older models:

• babbage-002
• davinci-002
• gpt-3.5-turbo-instruct

These models do not support the chat API format.

Both APIs

Some models support both chat and completions:

• gpt-4o-mini
• gpt-4.1-nano
• gpt-5.1

Responses API (Newer Models)

The /v1/responses endpoint is supported by specialized models:

• gpt-5-pro
• codex-mini-latest

Using the Global Registry

The registry provides a global instance for convenience:

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::openai_model_registry::{
    get_model_registry, ModelType
};

// Access the global registry
let registry = get_model_registry();

// Look up model capabilities
let caps = registry.get_model_capabilities("gpt-4o");
assert_eq!(caps.model_type, ModelType::Chat);
assert!(caps.supports_tools);
assert!(caps.supports_streaming);
assert!(caps.supports_vision);
}

Token Limit Parameters

Different model types use different parameter names for controlling output length:

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::openai_model_registry::get_model_registry;

let registry = get_model_registry();

// Chat models use "max_tokens"
let gpt4_caps = registry.get_model_capabilities("gpt-4o");
assert_eq!(gpt4_caps.get_token_limit_param(), "max_tokens");

// Reasoning models use "max_completion_tokens"
let o1_caps = registry.get_model_capabilities("o1");
assert_eq!(o1_caps.get_token_limit_param(), "max_completion_tokens");
}

Temperature Support

Reasoning models have restricted temperature support:

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::openai_model_registry::get_model_registry;

let registry = get_model_registry();

// Chat models support arbitrary temperatures
let gpt4_caps = registry.get_model_capabilities("gpt-4o");
assert!(gpt4_caps.supports_temperature(0.7));
assert!(gpt4_caps.supports_temperature(1.5));

// Reasoning models only support temperature 1.0
let o1_caps = registry.get_model_capabilities("o1");
assert!(o1_caps.supports_temperature(1.0));
assert!(!o1_caps.supports_temperature(0.7));
}

Checking Endpoint Support

You can check which API endpoints a model supports:

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::openai_model_registry::get_model_registry;

let registry = get_model_registry();

// Standard chat model - chat API only
let caps = registry.get_model_capabilities("gpt-4o");
assert!(caps.supports_chat_api);
assert!(!caps.supports_completions_api);
assert!(!caps.supports_responses_api);

// Dual-endpoint model
let mini_caps = registry.get_model_capabilities("gpt-4o-mini");
assert!(mini_caps.supports_chat_api);
assert!(mini_caps.supports_completions_api);
assert!(!mini_caps.supports_responses_api);

// Completions-only model
let instruct_caps = registry.get_model_capabilities("gpt-3.5-turbo-instruct");
assert!(!instruct_caps.supports_chat_api);
assert!(instruct_caps.supports_completions_api);
assert!(!instruct_caps.supports_responses_api);

// Responses-only model
let pro_caps = registry.get_model_capabilities("gpt-5-pro");
assert!(!pro_caps.supports_chat_api);
assert!(!pro_caps.supports_completions_api);
assert!(pro_caps.supports_responses_api);
}

Creating Custom Registries

While the global registry is convenient, you can create custom registries for testing or specialized configurations:

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::openai_model_registry::{
    OpenAIModelRegistry, ModelCapabilities, ModelType
};

// Create a new empty registry
let mut custom_registry = OpenAIModelRegistry::new();

// Register a custom model
custom_registry.register_model("custom-model", ModelCapabilities {
    model_type: ModelType::Chat,
    supports_tools: true,
    supports_streaming: true,
    supports_vision: false,
    max_context_tokens: Some(8192),
    max_output_tokens: Some(4096),
    supported_temperatures: None, // Accepts any temperature
    supports_chat_api: true,
    supports_completions_api: false,
    supports_responses_api: false,
});

// Use the custom registry
let caps = custom_registry.get_model_capabilities("custom-model");
assert_eq!(caps.model_type, ModelType::Chat);
assert!(caps.supports_tools);
}

Pattern Matching for Unknown Models

When a model is not explicitly registered, the registry infers capabilities from the model name:

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::openai_model_registry::get_model_registry;

let registry = get_model_registry();

// Unknown model with "gpt-4" prefix inherits GPT-4 capabilities
let unknown_caps = registry.get_model_capabilities("gpt-4-future-model");
assert!(unknown_caps.supports_tools);
assert!(unknown_caps.supports_streaming);

// Unknown model with "o" prefix is treated as a reasoning model
let unknown_o_caps = registry.get_model_capabilities("o5-experimental");
assert_eq!(unknown_o_caps.get_token_limit_param(), "max_completion_tokens");
assert!(unknown_o_caps.supports_temperature(1.0));
assert!(!unknown_o_caps.supports_temperature(0.7));
}

Common Model Categories

Chat-Only Models

Most GPT-4, reasoning, and base GPT-5 models:

• gpt-4o, gpt-4-turbo, gpt-4
• o1, o1-mini, o3, o3-mini, o4-mini
• gpt-5, gpt-5.1

Chat + Responses API Models

The GPT-5.4 and GPT-5.5 reasoning families support both the Chat Completions and Responses APIs. They expose a 1.05M-token context window (400K for mini/nano variants) with a 128K output cap, support tools, streaming, and vision, and accept only temperature 1.0:

• gpt-5.4, gpt-5.4-mini, gpt-5.4-nano
• gpt-5.5, gpt-5.5-pro

Completions-Only Models

Legacy models that don’t support chat format:

• babbage-002
• davinci-002
• gpt-3.5-turbo-instruct

Dual-Endpoint Models

Models supporting both chat and completions:

• gpt-4o-mini
• gpt-4.1-nano
• gpt-5.1

Responses-Only Models

Specialized newer endpoint models:

• gpt-5-pro
• codex-mini-latest

Summary

The OpenAI Model Registry provides:

• Type-safe model capability definitions
• Automatic parameter selection based on model type
• Temperature validation for reasoning models
• API endpoint support tracking
• Pattern matching for unknown models
• A global registry for convenience
• Custom registries for specialized use cases

By consulting the registry, the OpenAI gateway ensures it uses the correct parameters and features for each model, providing a robust and maintainable integration layer.

Caller intent in 3.0

The registry only adapts controls the caller selected. An unset temperature or output limit stays absent. For a model with one allowed temperature, an explicit selection is rewritten to that value; for a model accepting none, it is omitted. Reasoning effort on a non-reasoning model is dropped with a warning. Native named effort mappings and explicit reasoning profile validation remain unchanged.

oMLX Gateway

oMLX is an LLM server for Apple Silicon. OmlxGateway connects Mojentic to it.

oMLX uses the OpenAI chat completions protocol, but do not use OpenAIGateway for it. The OpenAI gateway changes requests for model names that it does not know, it drops the reasoning trace, and it splits embedding input with the OpenAI tokenizer. OmlxGateway sends your settings as you set them.

Quick start

use mojentic::llm::gateways::OmlxGateway;
use mojentic::llm::{LlmBroker, LlmMessage};
use std::sync::Arc;

#[tokio::main]
async fn main() -> mojentic::Result<()> {
    let gateway = Arc::new(OmlxGateway::new());
    let broker = LlmBroker::new("Qwen3.8-27B-MLX-8bit", gateway, None);

    let messages = vec![LlmMessage::user("Name three rivers in Canada.")];
    let reply = broker.generate(&messages, None, None, None).await?;
    println!("{reply}");
    Ok(())
}

The omlx example runs one turn and shows the thinking and the evidence:

cargo run --example omlx -- Qwen3.8-27B-MLX-8bit

Configuration

• Give the host without /v1. The gateway adds /v1 to each path.
• When you set an API key, the gateway sends Authorization: Bearer <key>. When you do not, it sends no authorization header. An empty OMLX_API_KEY counts as no key. An explicit empty or whitespace-only key also sends no authorization header and does not fall back to the environment.
• The timeout bounds non-streaming requests, including model load. Streaming requests omit this whole-response timeout so long replies can finish. Drop the stream to cancel its request. When OMLX_TIMEOUT is not a number, the gateway logs a warning and uses the default. Set timeout: None for no timeout.

OmlxConfig::default() reads the environment. A field that you set yourself has priority. The environment has priority above the defaults:

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::{OmlxConfig, OmlxGateway};
use std::time::Duration;

let gateway = OmlxGateway::with_config(OmlxConfig {
    host: "http://studio.local:8000".to_string(),
    timeout: Some(Duration::from_secs(1800)),
    ..Default::default()
});
}

Request settings

The gateway sends these CompletionConfig fields to oMLX:

The gateway does not send num_ctx or num_predict. oMLX sets the context length for each model.

Reasoning effort

oMLX gives reasoning_effort to the model’s chat template, so the effect of low, medium and high depends on the model. enable_thinking: false turns thinking off.

The explicit Qwen3.8 profile rejects high and accepts xhigh. See reasoning controls and native history for model validation, opt-in retention, and prototype migration requirements.

Thinking

oMLX reports the model’s reasoning trace as reasoning_content. The gateway puts it in LlmGatewayResponse::thinking. When the model does not think, thinking is None.

In the streaming APIs:

• complete_stream yields reasoning deltas as StreamChunk::Thinking.
• generate_stream_events yields no event for reasoning. It has no thinking event.

Truncation during thinking

When max_tokens stops the model while it thinks, oMLX puts the partial reasoning in content, sets no reasoning_content, and reports the finish reason length. A streamed response keeps the partial reasoning as thinking. The gateway does not move text between the fields.

When the finish reason in response.evidence is not stop, content is not an answer. Read the finish reason before you use content.

Structured output

generate_object, complete_json and complete_json_response send response_format: {"type": "json_schema", "json_schema": {"name": "response", "schema": ...}}. The gateway parses the content as JSON. You must still validate the object.

When oMLX cannot compile a grammar for the schema, it does not enforce the format. It adds instructions to the prompt and sends a Warning response header. When you asked for structured output (the object APIs, or a ResponseFormat::JsonObject with or without a schema), the gateway:

• puts the header value in response.evidence.metadata with the key response_format_warning (the constant omlx::RESPONSE_FORMAT_WARNING). It joins several Warning headers with ,.
• logs a warning.

The gateway does not retry and does not fail. The warning is evidence. For a text format or no format, the gateway ignores the header.

Streaming

oMLX starts each stream with a keep-alive frame whose model is keepalive, and sends more of them during a long prefill. The gateway drops these frames before it parses the stream, so keepalive never shows as the provider model.

generate_stream_events uses the OpenAI completion rules. It asks for usage with stream_options.include_usage. A clean finish needs the finish reason stop and the data: [DONE] marker. See Streaming.

complete_stream yields content, thinking and tool calls. oMLX sends all of a tool call in one delta. The gateway yields it as one StreamChunk::ToolCalls.

Models

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::OmlxGateway;
use mojentic::llm::LlmGateway;

async fn run() -> mojentic::Result<()> {
let gateway = OmlxGateway::new();
let models = gateway.get_available_models().await?; // sorted ids
gateway.load_model(&models[0]).await?;
gateway.unload_model(&models[0]).await?;
Ok(())
}
}

• Blank model ids for load or unload fail before a request is sent.
• load_model blocks until the model is in memory. A chat request loads its model when necessary, so use load_model only to warm up a model early.
• unload_model for a model that is not loaded fails with the 400 error from oMLX.
• oMLX downloads models only through its admin dashboard. The gateway has no pull operation.

Embeddings

#![allow(unused)]
fn main() {
use mojentic::llm::gateways::OmlxGateway;
use mojentic::llm::LlmGateway;
async fn run() -> mojentic::Result<()> {
let gateway = OmlxGateway::new();
let embedding = gateway.calculate_embeddings("A river delta", Some("my-embedding-model")).await?;
Ok(())
}
}

• You must give a model. oMLX has no default embedding model. Without a model, the gateway returns MojenticError::InvalidArgument and sends no request.
• The gateway sends the text in one request. It does not split the text.
• A chat model fails with a 400 error from oMLX.

Errors

oMLX errors use the OpenAI shape. The gateway reports each one as MojenticError::GatewayError, with the HTTP status and the response body. In generate_stream_events, the error is StreamEventError::ProviderError, with the status and the body’s error value.

Request controls and migration to 3.0

Mojentic sends sampling, context and output controls only when you choose them. CompletionConfig::default() leaves temperature, num_ctx, and max_tokens unset (None), so your model and server settings apply. Setting zero temperature requests zero; it does not mean “use the default”. top_p and top_k remain optional. Provider support determines which controls can be sent.

Every supported field in this table is omitted when unset. There is no separate num_predict config field and no negative sentinel. Use max_tokens: None to leave the output limit to the provider. oMLX does not require max_tokens. Explicit zero limits are forwarded; the provider decides whether to accept them.

OpenAI models with a single allowed temperature rewrite an explicit temperature to that allowed value. Models that accept no temperature omit it. An unset value remains absent in both cases. Reasoning effort on a non-reasoning model is omitted with a warning. Named native reasoning mappings, explicit profile validation, and restrictions on boolean enablement and reasoning history are unchanged. See reasoning controls.

Migrating from 2.1

Wrap numeric values in Some(...) and handle None when reading config fields. To retain the former library defaults explicitly:

#![allow(unused)]
fn main() {
use mojentic::llm::gateway::CompletionConfig;

let config = CompletionConfig {
    temperature: Some(1.0),
    num_ctx: Some(32768),
    max_tokens: Some(16384),
    ..Default::default()
};
}

Without these settings, Ollama uses model/server precedence, which may produce a smaller context and unlimited output. OpenAI no longer receives the library’s 16,384 output cap; reasoning completions may run longer and cost more. Each provider retains its own limits.

Use a full config in a session to carry the same intent through ordinary turns, streaming and automatic tool continuations:

#![allow(unused)]
fn main() {
use mojentic::llm::{ChatSession, LlmBroker, gateway::CompletionConfig, gateways::OllamaGateway};
use std::sync::Arc;

let broker = LlmBroker::new("qwen3", Arc::new(OllamaGateway::new()), None);
let session = ChatSession::builder(broker)
    .max_context(32768) // local history budget only
    .completion_config(CompletionConfig {
        temperature: Some(0.0),
        num_ctx: Some(131072),
        max_tokens: Some(8192),
        ..Default::default()
    })
    .build();
}

.temperature(value) remains available and changes only that config field. Builder calls apply in order: .completion_config(...) replaces the config. A session with no explicit config sends no provider controls. max_context trims local history and never sets num_ctx.

Recovery freezes and replays the encoded request, including absent controls. Raw event streams retain their single-send behavior even on a recovery wrapper; see completion recovery. Tracer call events retain caller intent as Option<f64>: an unset temperature is absent from serialized events. Readers of older numeric trace events remain supported.

Provider references and verification limits

Ollama chat accepts runtime options; its parameter reference describes num_ctx, num_predict and sampling controls. OpenAI Chat Completions documents the optional token-limit and sampling fields, including reasoning-token accounting for max_completion_tokens. oMLX’s API implementation is the pinned provider source reference; omission behavior follows the coordinated October 2026 model-metadata contract and its recorded no-limit server capture.

The shared request-fidelity-v1 tests exercise the production HTTP adapters and broker/session paths using artifact-derived responses. Local verification does not establish external provider or microgentic agreement. Those counterpart commands and source identities remain required initiative coordination; see CONFORMANCE-3.0.0.md in the repository.

Recovering one completion request

Transient completion recovery is opt-in for Ollama, oMLX, and OpenAI. It preserves one encoded HTTP request across attempts. A retry never starts the broker tool loop again. Embeddings, realtime voice, model management, and the original generate_stream_events API keep their existing behavior.

Wrap the gateway before giving it to a broker or chat session:

#![allow(unused)]
fn main() {
use mojentic::llm::{LlmBroker, LlmMessage};
use mojentic::llm::gateways::OllamaGateway;
use mojentic::llm::recovery::{RecoveryPolicy, RecoveryCall, Outcome};
use mojentic::llm::gateway::CompletionConfig;
use std::{num::NonZeroU32, sync::Arc, time::Duration};

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let policy = RecoveryPolicy {
    max_attempts: NonZeroU32::new(3).expect("positive attempt limit"),
    budget: Some(Duration::from_secs(20)),
    ..Default::default()
};
let gateway = OllamaGateway::new().with_recovery(policy)?;
let call = RecoveryCall::default();
let messages = [LlmMessage::user("Explain ownership")];
let result = gateway.complete_with_recovery(
    "qwen3", &messages, None, &CompletionConfig::default(), &call,
).await;
match result {
    Ok(result) => {
        println!("wire attempts: {}", result.report.identity.wire_attempt);
        // Consume result.value as the existing LlmGatewayResponse.
    }
    Err(error) if error.outcome == Outcome::AdmissionRequired => {
        // The harness must establish ownership/termination and explicitly admit
        // recovery. No second request was dispatched.
    }
    Err(error) => return Err(error.into()),
}
let broker = LlmBroker::new("qwen3", Arc::new(gateway), None);
// broker.generate(...) and ChatSession propagate MojenticError::Recovery.
Ok(())
}
}

The default policy allows only one attempt. Setting a larger maximum selects eligibility; it does not authorize an ambiguous local inference resend. Implement Admission::decide(AdmissionContext) to await your harness’s checks and return AdmissionDecision::Allow, Reject, or Unresolved. Only Allow permits a resend. Return Unresolved when the hook completes without a decision (for example, an external approval source has no answer). Recovery then terminates with Outcome::AdmissionUnresolved and Transition::AdmissionUnresolved (serialized as admission_unresolved), distinct from caller rejection. It retains the failed attempt, history, and private evidence without executing tools or emitting AdmissionAllowed, DelayScheduled, or RetryStarted. The context includes the previous failure, logical and attempt identity, next wire number, and observed/delivered progress. A pending future stays pending and is cancelled when the call is cancelled or its recovery budget expires. Every retry invokes the hook when configured, including OpenAI.

complete_json_with_recovery returns a parsed object and the same report. complete_stream_with_recovery returns content, reasoning, tool fragments, nonsemantic progress/metrics when supported, and exactly one Completed or Failed event. Fragments are inspection data; execute only complete calls after terminal success. Wrapping the gateway also enables recovery on LlmGateway::complete_stream, including broker/session streaming. Native thinking remains available to broker-managed tool rounds when preserve_reasoning is enabled. Raw complete_stream_events and generate_stream_events deliberately remain the original single-send APIs.

Any observed reasoning, content, or tool fragment prevents replay, including output buffered before delivery. A partial successful HTTP JSON body also prevents replay conservatively. Incomplete stream frames fail closed. Complete keepalive frames affect raw bytes and decoded frame progress only. Cancellation yields Cancelled, retains received evidence, and closes owned response resources even if a streaming consumer pauses. Dropping a recovery stream cancels its owned worker. Closing a socket proves local cleanup; it does not prove remote inference terminated. Cancellation from a body wire observer also wins over terminal-only stream completion. An active cancelled send emits AttemptStarted, AttemptFailed, then one Cancelled; it emits neither AttemptSucceeded nor Completed. The failure retains both observed and delivered progress with attempt history.

Recovery budgets start at the first failure. Absolute deadlines and budgets limit admission, waiting, and the start of a retry; they never impose a timeout on active generation. Provider configuration whole-response timeouts are omitted on recovery completion requests. Cancel with call.cancellation.cancel(). Retries use exponential full jitter; numeric and HTTP-date Retry-After values set a minimum. An excessive minimum is refused rather than shortened. Inject Timing to control monotonic/wall clocks, sleeping, and jitter in harness tests.

Safe errors and lifecycle events omit response text, tool arguments, credentials, and original cause formatting. inspect_cause, inspect_response_bytes, and inspect_response_headers explicitly expose potentially sensitive evidence. LifecycleObserver receives typed safe metadata. Request IDs are validated and omitted when their spelling echoes outbound credentials or prompt text. WireObserver is a separate opt-in HTTP-boundary hook receiving raw requests, response headers, and each received body chunk, including partial/failing responses. Its request callback runs before dispatch; cancellation or capture failure can still prevent that send. Use reports/history for actual attempt accounting. Capture errors retain semantic progress already parsed from received bytes, stop recovery before delivery, and never cause another inference request. Callers own capture storage and must apply their credential/payload policy.

Model unload/list operations are aggregate model operations, not termination proof for this request. Logical/attempt UUIDs are correlation IDs only.

Existing gateways are unchanged when used without this wrapper. The wrapper’s MojenticError::Recovery retains the structured failure for broker/session callers; update exhaustive matches on MojenticError for the new variant. Direct recovery methods return RecoveryError and Recovered<T> without flattening metadata into a legacy error string. See RECOVERY-CONFORMANCE.md in the repository for assertions, limitations, and validation evidence.

Known HTTP 400/401/403 responses remain permanent even when their bodies are truncated. A body transport error preserves the HTTP status and private original cause; selecting those statuses or allowing admission cannot authorize a resend.

Decoded stream progress and metrics

Recovery streams now expose RecoveryStreamEvent::Progress { identity, progress } and RecoveryStreamEvent::Metrics { identity, metrics }. Add these arms to exhaustive event matches when migrating. identity correlates each event to the logical request, attempt ID, and one-based wire attempt. Ollama frame indices start at one again on each attempt, including after a safely admitted retry.

The values are the same StreamProgress and StreamMetrics used by the original OllamaGateway::complete_stream. Progress precedes semantic output from each valid NDJSON frame, including empty keepalives and the terminal frame. Its accumulated tool count describes calls accumulated before that frame. The existing content_chars and thinking_chars fields measure UTF-8 byte lengths; their names and values are preserved for compatibility. Terminal metrics follow terminal progress; completed tool calls are delivered only after provider terminal proof. Recovery’s LlmGateway::complete_stream maps these events to StreamChunk::Progress and StreamChunk::Metrics, preserving the values.

A structurally valid Ollama terminal frame with an unsuccessful finish reason (for example done_reason="length") still emits exact Progress, then Metrics, then Failed. Compatibility streams emit the same metadata followed by the recovery error. Telemetry describes what the provider reported, rather than certifying a successful reply. Semantic fields in that rejected frame count as observed evidence, but are withheld from delivery; no completed tools or retry follow. Earlier delivered semantic output remains accounted for. Malformed fields do not produce telemetry from unvalidated data. Cancellation and capture failure can stop delivery before metadata reaches the consumer while retaining received semantic evidence in the failure report.

Ollama durations/counts come directly from provider fields; absent fields and undefined throughput remain None. OpenAI and oMLX do not expose these Ollama frame counters or final duration metrics through their existing compatibility streams, so recovery emits neither event for them. Provider response evidence remains available on completion. Do not infer missing telemetry or fabricate local/cloud equivalents.

Progress and metrics are nonsemantic metadata. They do not increment delivered content, reasoning, tool fragments, completed tools, or model/tool interactions. Keepalives can inform a harness’s progress/stall classification, but never prove remote termination or authorize another attempt. Wire body observation remains raw transport evidence and is not a substitute for decoded frame progress. Cancellation and capture failures retain the existing observed/delivered semantic counters; metadata never changes replay safety or admission requirements.

For migration, update exhaustive matches on AdmissionDecision, Outcome, and Transition for the unresolved variants. Existing hook implementations still return the same future result type; use Unresolved for a completed absence of permission. Continue awaiting when the decision is still pending. Cancellation and budget expiry continue to terminate pending hooks, and absent-hook local safety checks and remote retry rules are unchanged. The scoped verification map and consumer coordination gap are in CONFORMANCE-3.0.0.md.

Endpoint capability declarations

gateway.capabilities() reports five Support values for the adapter’s exact completion endpoint. gateway.capability_evidence() returns the corresponding CapabilityEvidenceSet: every field contains value, endpoint, documentation, inspected_on, and basis. Canonical URLs identify provider endpoints; a configured proxy or custom host may offer different guarantees. These declarations do not change requests, retry eligibility, cancellation or admission policy, and do not issue status/cancel calls or idempotency headers.

The shared recovery-capabilities-v1 declarations retain their original 2026-10-16 inspection date and were rechecked against provider documentation on 2026-10-17. Ollama’s SDK abort documentation explains local streaming abort; the chat API has no remote cancellation, active-request status or deduplicated resend facility. OpenAI’s streaming documentation describes interrupted/cancelled streams, while its Chat Completions operations provide no remote cancel or idempotent replay facility for this POST. Stored completion retrieval is not an active-request status facility. The pinned oMLX endpoint inventory does not document local cancellation or deduplication semantics sufficiently; OpenAI compatibility alone does not establish them. Its inventory has no chat request cancel or status operation. Unsupported declarations describe these inspected endpoint documents; they do not assert a product-wide impossibility.

Local HTTP cancellation never proves that one exact remote attempt terminated. All three adapters therefore retain Unknown for exact-attempt termination. Local ambiguous retries still need explicit admission under the existing rules. The anthropic feature flag does not provide a recovery adapter.

#![allow(unused)]
fn main() {
use mojentic::llm::{gateways::OllamaGateway, recovery::{RecoveryPolicy, Support}};
fn example() -> Result<(), Box<dyn std::error::Error>> {
let gateway = OllamaGateway::new().with_recovery(RecoveryPolicy::default())?;
let capabilities = gateway.capabilities();
assert_eq!(capabilities.local_request_cancellation, Support::Supported);
assert_eq!(capabilities.exact_attempt_termination, Support::Unknown);
let evidence = gateway.capability_evidence();
println!("{}: {} ({})", evidence.idempotency.endpoint,
         evidence.idempotency.documentation, evidence.idempotency.inspected_on);
Ok(())
}
}

For migration, replace the serialized request_cancel key with remote_request_cancellation. The Rust request_cancel field remains a legacy alias for remote cancellation and is omitted from serialization. Add remote_request_cancellation, local_request_cancellation and exact_attempt_termination to capability literals. Provider declarations come from the adapter, not Capabilities::default(), which is unbound and carries no provider evidence. Ollama/OpenAI idempotency changes from Unknown to Unsupported; oMLX remains Unknown. Consumers must verify the versioned executable set in their own gates before claiming boundary compatibility; see CONFORMANCE-3.0.0.md for the coordination gap.

Policy and typed terminal errors

with_recovery and RecoveryGateway::new validate the policy before creating any ordinary, structured, streaming, broker or session recovery entrypoint. An invalid selection returns MojenticError::InvalidRecoveryPolicy with a concrete RecoveryPolicyError::InvalidStatus or UnsupportedCategory. Use HTTP error statuses (400–599) and Transport, Http or ClientTimeout categories. Explicitly selecting 400/401/403 remains valid but never makes them retryable. Protocol, provider-response and cancellation categories have no supported retry policy. Attempt limits and delays use NonZeroU32 and Duration, so zero attempts, negative delays and nonfinite delays cannot be constructed. Zero budgets remain valid; base delays above the ceiling are clamped as before.

Length termination means the answer and any tool calls are incomplete. Recovery returns a protocol failure for ordinary, structured and streaming requests, without retry or tool execution. In structured output, the length cause takes precedence over parsing truncated JSON. Received semantic observations and wire bytes remain available; they do not imply successful delivery. A failed session retains the user’s turn without committing an incomplete assistant turn.

#![allow(unused)]
fn main() {
use mojentic::llm::recovery::{CompletionDecodeError, RecoveryError};

fn reached_output_limit(error: &RecoveryError) -> bool {
    matches!(
        error.failure.inspect_cause()
            .and_then(|cause| cause.downcast_ref::<CompletionDecodeError>()),
        Some(CompletionDecodeError::Length)
    )
}
}

Structural decoding failures expose CompletionDecodeError::InvalidFrame; JSON failures expose the original serde_json::Error directly from inspect_cause(), without the former private wrapper’s source() hop. A wire capture callback’s error can be downcast to the callback’s own concrete type. Inspect causes explicitly: they may contain sensitive information, and are omitted from normal error formatting, serialization and lifecycle events. The final failure and its retained history entry refer to the same cause. Broker and session calls preserve these through MojenticError::Recovery; stream failures emit one terminal error.

This changes recovery-enabled length handling and adds an error variant for exhaustive matches. Gateways without recovery and the raw legacy event-stream APIs retain their earlier behavior. No alternate supported typed-errors contract version predates typed-errors-v1; existing successful optional-field behavior is covered alongside legacy length behavior. Local executable cases live in tests/contracts/typed_errors_v1.json and run via cargo test --all-features --test recovery typed_errors_v1. Scripted HTTP fixtures verify Rust’s boundary behavior and run again from the packaged artifact. External counterpart verification is not a publication prerequisite for this library; see CONFORMANCE-3.0.0.md.

HTTP 500 and 502 are retryable under the default policy when no semantic output has been observed or delivered. Both the HTTP status and Category::Http must be selected. Removing either from retryable_statuses or retryable_categories makes that failure ineligible (Reason::Permanent) and prevents resend, even if admission would allow it. Eligibility still does not bypass admission or budgets.

The local recovery-status-selection-v1 executable set verifies those paired choices through actual loopback HTTP adapters and public broker/session APIs: cargo test --all-features --test recovery recovery_status_selection_v1. It checks successful output, typed failures, immutable wire payloads, attempt identity and safe lifecycle fields. Existing partial reasoning, content and tool interruption cases continue to prohibit replay. The public suites and a fresh packaged consumer verify this identical executable set. Dependent adoption and post-publication installation remain coordinated release work.

Final client stream measurements are available through RecoveryReport.stream_metrics and RecoveryError.stream_metrics. They span admitted retries, preserve the first semantic token and retain duration on failure. Raw provider telemetry is unchanged; see client timing for the event and compatibility APIs.

Layer 2 - Agents

Agents are the smallest unit of computation in Mojentic. They process events and emit new events, composing into larger systems.

This section mirrors the Python project’s agent layer but uses Rust’s async and type system.

Asynchronous Capabilities

Mojentic (Rust) embraces async end-to-end.

• Concurrency for overlapping LLM calls
• Channels/streams for event propagation
• Timeouts and cancellation for robustness

Iterative Problem Solver

The IterativeProblemSolver is an agent that iteratively attempts to solve problems using available tools. It employs a chat-based approach and continues working until it succeeds, fails explicitly, or reaches the maximum number of iterations.

Overview

The Iterative Problem Solver follows a simple but powerful pattern:

1. Plan - Analyze the problem and identify what needs to be done
2. Act - Execute actions using available tools
3. Observe - Review the results
4. Refine - Adjust the approach based on observations
5. Terminate - Stop when the goal is met or the iteration budget is exhausted

Key Features

• Tool Integration: Seamlessly integrates with any LlmTool implementations
• Automatic Termination: Stops when the LLM responds with “DONE” or “FAIL”
• Iteration Control: Configurable maximum iteration count prevents infinite loops
• Chat-Based Context: Maintains conversation history for context-aware problem solving
• Summary Generation: Provides a clean summary of the final result

Usage

Basic Example

use mojentic::agents::IterativeProblemSolver;
use mojentic::llm::{LlmBroker, LlmTool};
use mojentic::llm::gateways::OllamaGateway;
use mojentic::llm::tools::simple_date_tool::SimpleDateTool;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize the LLM broker
    let gateway = Arc::new(OllamaGateway::default());
    let broker = LlmBroker::new("qwen3:32b", gateway, None);

    // Define available tools
    let tools: Vec<Box<dyn LlmTool>> = vec![
        Box::new(SimpleDateTool),
    ];

    // Create the solver
    let mut solver = IterativeProblemSolver::builder(broker)
        .tools(tools)
        .max_iterations(5)
        .build();

    // Solve a problem
    let result = solver.solve("What's the date next Friday?").await?;
    println!("Result: {}", result);

    Ok(())
}

Custom System Prompt

You can customize the system prompt to guide the solver’s behavior:

#![allow(unused)]
fn main() {
let mut solver = IterativeProblemSolver::builder(broker)
    .tools(tools)
    .max_iterations(10)
    .system_prompt(
        "You are a specialized data analysis assistant. \
         Break down complex queries into clear steps and use tools methodically."
    )
    .build();
}

With Multiple Tools

The solver works best when given appropriate tools for the problem domain:

#![allow(unused)]
fn main() {
use mojentic::llm::tools::ask_user_tool::AskUserTool;
use mojentic::llm::tools::simple_date_tool::SimpleDateTool;

let tools: Vec<Box<dyn LlmTool>> = vec![
    Box::new(AskUserTool::new()),
    Box::new(SimpleDateTool),
];

let mut solver = IterativeProblemSolver::builder(broker)
    .tools(tools)
    .max_iterations(5)
    .build();
}

How It Works

Step-by-Step Process

1. Initialization: The solver creates a ChatSession with the provided system prompt and tools
2. Iteration Loop: For each iteration:
   • Sends the problem description with instructions to use tools
   • Checks the response for “DONE” (success) or “FAIL” (failure)
   • Continues if neither keyword is present and iterations remain
3. Summary: After termination, requests a concise summary of the result
4. Return: Returns the summary as the final result

Termination Conditions

The solver terminates when one of these conditions is met:

• Success: The LLM’s response contains “DONE” (case-insensitive)
• Failure: The LLM’s response contains “FAIL” (case-insensitive)
• Exhaustion: The maximum number of iterations is reached

Logging

The solver uses the tracing crate to log important events:

• info: Logged when a task completes successfully or fails
• warn: Logged when maximum iterations are reached

Configuration Options

Builder Pattern

The IterativeProblemSolver uses the builder pattern for configuration:

#![allow(unused)]
fn main() {
IterativeProblemSolver::builder(broker)
    .tools(tools)              // Set available tools
    .max_iterations(10)        // Set max iterations (default: 3)
    .system_prompt("...")      // Set custom system prompt
    .build()
}

Default Values

• max_iterations: 3
• system_prompt:

  “You are a problem-solving assistant that can solve complex problems step by step. You analyze problems, break them down into smaller parts, and solve them systematically. If you cannot solve a problem completely in one step, you make progress and identify what to do next.”

Best Practices

1. Choose Appropriate Tools

Select tools that are relevant to your problem domain:

#![allow(unused)]
fn main() {
// For date/time problems
let tools = vec![Box::new(SimpleDateTool)];

// For user interaction
let tools = vec![Box::new(AskUserTool::new())];

// For data analysis
let tools = vec![
    Box::new(CalculatorTool),
    Box::new(DataRetrievalTool),
];
}

2. Set Reasonable Iteration Limits

Balance between giving the solver enough attempts and preventing excessive computation:

• Simple queries: 3-5 iterations
• Complex analyses: 10-15 iterations
• Open-ended exploration: 20+ iterations

3. Provide Context in Problem Description

The more context you provide, the better the solver can work:

#![allow(unused)]
fn main() {
// Less effective
solver.solve("Analyze the data").await?;

// More effective
solver.solve(
    "Analyze the sales data from Q1 2024. \
     Focus on trends in the technology sector. \
     Provide insights on growth patterns."
).await?;
}

4. Monitor Logs

Enable tracing to understand solver behavior:

#![allow(unused)]
fn main() {
tracing_subscriber::fmt()
    .with_max_level(tracing::Level::INFO)
    .init();
}

Common Patterns

Retry Logic

For operations that might fail transiently:

#![allow(unused)]
fn main() {
let mut attempts = 0;
let max_attempts = 3;

let result = loop {
    attempts += 1;
    match solver.solve(problem).await {
        Ok(result) if !result.contains("FAIL") => break result,
        Ok(_) if attempts < max_attempts => continue,
        Ok(result) => break result,
        Err(e) => return Err(e),
    }
};
}

Multi-Stage Problems

For problems that require multiple phases:

#![allow(unused)]
fn main() {
// Phase 1: Data gathering
let mut solver = IterativeProblemSolver::builder(broker.clone())
    .tools(data_tools)
    .max_iterations(5)
    .build();
let data = solver.solve("Gather all relevant data").await?;

// Phase 2: Analysis
let mut solver = IterativeProblemSolver::builder(broker)
    .tools(analysis_tools)
    .max_iterations(10)
    .build();
let analysis = solver.solve(&format!("Analyze: {}", data)).await?;
}

Examples

See the complete examples at:

• examples/iterative_solver.rs - Basic usage with date and user interaction tools
• examples/solver_chat_session.rs - Interactive chat session with solver delegation pattern

Error Handling

The solver returns Result<String, MojenticError>:

#![allow(unused)]
fn main() {
match solver.solve(problem).await {
    Ok(result) => println!("Solution: {}", result),
    Err(MojenticError::GatewayError(msg)) => {
        eprintln!("Gateway error: {}", msg);
    }
    Err(MojenticError::ToolError(msg)) => {
        eprintln!("Tool error: {}", msg);
    }
    Err(e) => {
        eprintln!("Unexpected error: {}", e);
    }
}
}

Advanced: Solver as a Tool

The IterativeProblemSolver can be wrapped as a tool and used within a ChatSession, enabling powerful delegation patterns where a chat assistant can offload complex problems to a specialized solver agent.

Creating a Solver Tool

#![allow(unused)]
fn main() {
use mojentic::llm::tools::{FunctionDescriptor, LlmTool, ToolDescriptor};
use mojentic::agents::IterativeProblemSolver;
use serde_json::{json, Value};
use std::collections::HashMap;
use std::sync::Arc;

struct IterativeProblemSolverTool {
    broker: Arc<LlmBroker>,
    tools: Vec<Box<dyn LlmTool>>,
}

impl IterativeProblemSolverTool {
    fn new(broker: Arc<LlmBroker>, tools: Vec<Box<dyn LlmTool>>) -> Self {
        Self { broker, tools }
    }
}

impl Clone for IterativeProblemSolverTool {
    fn clone(&self) -> Self {
        Self {
            broker: self.broker.clone(),
            tools: self.tools.iter().map(|t| t.clone_box()).collect(),
        }
    }
}

impl LlmTool for IterativeProblemSolverTool {
    fn run(&self, args: &HashMap<String, Value>) -> mojentic::error::Result<Value> {
        let problem_to_solve = args
            .get("problem_to_solve")
            .and_then(|v| v.as_str())
            .ok_or_else(|| {
                mojentic::error::MojenticError::ToolError(
                    "Missing required argument: problem_to_solve".to_string(),
                )
            })?;

        let solver_tools: Vec<Box<dyn LlmTool>> =
            self.tools.iter().map(|t| t.clone_box()).collect();

        let runtime = tokio::runtime::Handle::current();
        let broker_clone = (*self.broker).clone();

        let result = runtime.block_on(async move {
            let mut solver = IterativeProblemSolver::builder(broker_clone)
                .tools(solver_tools)
                .max_iterations(5)
                .build();

            solver.solve(problem_to_solve).await
        })?;

        Ok(json!({"solution": result}))
    }

    fn descriptor(&self) -> ToolDescriptor {
        ToolDescriptor {
            r#type: "function".to_string(),
            function: FunctionDescriptor {
                name: "iterative_problem_solver".to_string(),
                description: "Iteratively solve a complex multi-step problem using available tools.".to_string(),
                parameters: json!({
                    "type": "object",
                    "properties": {
                        "problem_to_solve": {
                            "type": "string",
                            "description": "The problem or request to be solved."
                        }
                    },
                    "required": ["problem_to_solve"],
                    "additionalProperties": false
                }),
            },
        }
    }

    fn clone_box(&self) -> Box<dyn LlmTool> {
        Box::new(self.clone())
    }
}
}

Using the Solver Tool in a Chat Session

use mojentic::llm::{ChatSession, LlmBroker};
use mojentic::llm::tools::simple_date_tool::SimpleDateTool;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let gateway = Arc::new(OllamaGateway::default());
    let broker = Arc::new(LlmBroker::new("qwq", gateway, None));

    // Create the solver tool with SimpleDateTool as the inner tool
    let solver_tools: Vec<Box<dyn LlmTool>> = vec![Box::new(SimpleDateTool)];
    let solver_tool = IterativeProblemSolverTool::new(broker.clone(), solver_tools);

    // Create chat session with the solver tool
    let mut session = ChatSession::builder((*broker).clone())
        .system_prompt(
            "You are a helpful assistant with access to an iterative problem solver. \
             When faced with complex multi-step problems or questions that require \
             reasoning and tool usage, use the iterative_problem_solver tool."
        )
        .tools(vec![Box::new(solver_tool)])
        .build();

    // Interactive loop
    loop {
        let mut query = String::new();
        print!("Query: ");
        io::stdout().flush()?;
        io::stdin().read_line(&mut query)?;

        if query.trim().is_empty() {
            break;
        }

        let response = session.send(query.trim()).await?;
        println!("{}\n", response);
    }

    Ok(())
}

Benefits of This Pattern

1. Delegation: The chat assistant can offload complex problems to a specialized solver
2. Composability: Mix solver capabilities with other tools in the same session
3. Context Preservation: The chat session maintains conversation history
4. Flexible Interaction: Users can ask simple questions directly or complex problems that trigger the solver

See the complete example at:

• examples/solver_chat_session.rs - Interactive chat session with solver delegation

Limitations

• LLM Dependency: Quality of results depends on the underlying LLM’s capabilities
• Tool Design: Effectiveness relies on well-designed tools with clear descriptions
• Token Limits: Long iterations may hit context window limits
• Cost: Multiple LLM calls per problem can increase API costs

See Also

• Chat Sessions - Understanding the underlying chat mechanism
• Building Tools - Creating custom tools for the solver
• Simple Recursive Agent - Alternative problem-solving pattern

Working Memory Pattern

The working memory pattern enables agents to maintain and share context across multiple interactions. This guide shows how to use SharedWorkingMemory and build memory-aware agents in Rust.

Overview

Working memory provides:

• Shared Context: Multiple agents can read from and write to the same memory
• Continuous Learning: Agents automatically learn and remember new information
• State Persistence: Knowledge is maintained across interactions
• Thread Safety: Safe concurrent access using Arc<Mutex<T>>

Quick Start

Basic Usage

#![allow(unused)]
fn main() {
use mojentic::context::SharedWorkingMemory;
use serde_json::json;

// Create memory with initial data
let memory = SharedWorkingMemory::new(json!({
    "User": {
        "name": "Alice",
        "age": 30
    }
}));

// Retrieve current state
let current = memory.get_working_memory();

// Update memory (deep merge)
memory.merge_to_working_memory(&json!({
    "User": {
        "city": "NYC",
        "preferences": {
            "theme": "dark"
        }
    }
}));

// Result: {"User": {"name": "Alice", "age": 30, "city": "NYC", "preferences": {...}}}
}

Memory-Aware Agent Pattern

use mojentic::llm::{LlmBroker, LlmMessage};
use mojentic::llm::gateways::OllamaGateway;
use mojentic::context::SharedWorkingMemory;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize
    let gateway = OllamaGateway::default();
    let broker = LlmBroker::new("qwen2.5:7b", gateway);
    let memory = SharedWorkingMemory::new(json!({
        "User": {"name": "Alice"}
    }));

    // Create prompt with memory context
    let memory_context = memory.get_working_memory();
    let prompt = format!(
        "This is what you remember: {}\n\nRemember anything new you learn.\n\nUser: I love pizza",
        serde_json::to_string_pretty(&memory_context)?
    );

    // Generate response with schema that includes memory field
    let schema = json!({
        "type": "object",
        "required": ["answer"],
        "properties": {
            "answer": {"type": "string"},
            "memory": {
                "type": "object",
                "description": "Add anything new you learned here."
            }
        }
    });

    let response = broker.generate(
        vec![LlmMessage::user(&prompt)],
        Some(schema),
        None,
        None,
        None,
        None
    ).await?;

    // Parse response and update memory
    let response_json: serde_json::Value = serde_json::from_str(&response.content)?;
    if let Some(learned) = response_json.get("memory") {
        memory.merge_to_working_memory(learned);
    }

    Ok(())
}

Core Concepts

SharedWorkingMemory

A thread-safe, mutable key-value store that agents use to share context:

#![allow(unused)]
fn main() {
pub struct SharedWorkingMemory {
    memory: Arc<Mutex<HashMap<String, Value>>>,
}

impl SharedWorkingMemory {
    pub fn new(initial: Value) -> Self
    pub fn get_working_memory(&self) -> Value
    pub fn merge_to_working_memory(&self, updates: &Value)
}
}

Key features:

• Thread-Safe: Uses Arc<Mutex<T>> for concurrent access
• Deep Merge: Nested JSON objects are recursively merged
• Simple API: Just 3 methods to learn

Deep Merge Behavior

Memory updates use deep merge to preserve existing data:

#![allow(unused)]
fn main() {
use serde_json::json;

// Initial memory
let memory = SharedWorkingMemory::new(json!({
    "User": {
        "name": "Alice",
        "age": 30,
        "address": {
            "city": "NYC",
            "state": "NY"
        }
    }
}));

// Update with nested data
memory.merge_to_working_memory(&json!({
    "User": {
        "age": 31,
        "address": {
            "zip": "10001"
        }
    }
}));

// Result: All fields preserved, nested objects merged
// {
//   "User": {
//     "name": "Alice",      // Preserved
//     "age": 31,            // Updated
//     "address": {
//       "city": "NYC",      // Preserved
//       "state": "NY",      // Preserved
//       "zip": "10001"      // Added
//     }
//   }
// }
}

Building Memory-Aware Agents

Here’s a complete pattern for memory-aware agents:

#![allow(unused)]
fn main() {
use mojentic::llm::{LlmBroker, LlmMessage};
use mojentic::context::SharedWorkingMemory;
use mojentic::agents::{Event, BaseAsyncAgent};
use serde::{Deserialize, Serialize};
use serde_json::json;
use async_trait::async_trait;

#[derive(Serialize, Deserialize)]
struct ResponseModel {
    text: String,
    #[serde(skip_serializing_if = "Option::is_none")]
    memory: Option<serde_json::Value>,
}

struct MemoryAgent {
    broker: Arc<LlmBroker>,
    memory: SharedWorkingMemory,
    behaviour: String,
    instructions: String,
}

impl MemoryAgent {
    fn new(
        broker: Arc<LlmBroker>,
        memory: SharedWorkingMemory,
        behaviour: String,
        instructions: String,
    ) -> Self {
        Self { broker, memory, behaviour, instructions }
    }

    async fn generate_with_memory(
        &self,
        user_input: &str,
    ) -> Result<ResponseModel, Box<dyn std::error::Error>> {
        // Build prompt with memory context
        let memory_context = self.memory.get_working_memory();
        let messages = vec![
            LlmMessage::system(&self.behaviour),
            LlmMessage::user(&format!(
                "This is what you remember:\n{}\n\n{}",
                serde_json::to_string_pretty(&memory_context)?,
                self.instructions
            )),
            LlmMessage::user(user_input),
        ];

        // Schema with memory field
        let schema = json!({
            "type": "object",
            "required": ["text"],
            "properties": {
                "text": {"type": "string"},
                "memory": {
                    "type": "object",
                    "description": "Add new information here."
                }
            }
        });

        // Generate response
        let response = self.broker.generate(
            messages,
            Some(schema),
            None,
            None,
            None,
            None
        ).await?;

        // Parse and update memory
        let result: ResponseModel = serde_json::from_str(&response.content)?;
        if let Some(ref learned) = result.memory {
            self.memory.merge_to_working_memory(learned);
        }

        Ok(result)
    }
}
}

Multi-Agent Coordination

Multiple agents can share the same memory instance:

#![allow(unused)]
fn main() {
use std::sync::Arc;

// Shared memory
let memory = Arc::new(SharedWorkingMemory::new(json!({
    "context": {}
})));

// Multiple agents
let researcher = MemoryAgent::new(
    Arc::clone(&broker),
    Arc::clone(&memory),
    "You are a research assistant.".to_string(),
    "Research topics thoroughly.".to_string(),
);

let writer = MemoryAgent::new(
    Arc::clone(&broker),
    Arc::clone(&memory),
    "You are a technical writer.".to_string(),
    "Write clear documentation.".to_string(),
);

// Researcher updates memory
let research = researcher
    .generate_with_memory("Research Rust async patterns")
    .await?;

// Writer uses updated memory (already shared)
let article = writer
    .generate_with_memory("Write an article about what was researched")
    .await?;
}

Use Cases

1. Conversational Chatbots

#![allow(unused)]
fn main() {
let memory = SharedWorkingMemory::new(json!({
    "conversation_history": [],
    "user_preferences": {}
}));
}

2. Workflow Automation

#![allow(unused)]
fn main() {
let memory = SharedWorkingMemory::new(json!({
    "workflow_state": "started",
    "completed_steps": [],
    "pending_tasks": []
}));
}

3. Knowledge Base Building

#![allow(unused)]
fn main() {
let memory = SharedWorkingMemory::new(json!({
    "entities": {},
    "relationships": [],
    "facts": []
}));
}

Best Practices

1. Structure Your Memory

Use clear, hierarchical keys:

#![allow(unused)]
fn main() {
json!({
    "User": {...},
    "Conversation": {...},
    "SystemState": {...}
})
}

2. Use Arc for Sharing

Share memory across threads/agents:

#![allow(unused)]
fn main() {
let memory = Arc::new(SharedWorkingMemory::new(initial));
let memory_clone = Arc::clone(&memory);
}

3. Validate Memory Updates

Check memory quality before accepting:

#![allow(unused)]
fn main() {
if let Some(ref learned) = result.memory {
    if is_valid_update(learned) {
        memory.merge_to_working_memory(learned);
    }
}
}

4. Handle Errors Gracefully

Memory operations can fail:

#![allow(unused)]
fn main() {
match agent.generate_with_memory(input).await {
    Ok(response) => {
        // Process response
    }
    Err(e) => {
        eprintln!("Failed to generate response: {}", e);
        // Don't update memory on error
    }
}
}

Example Application

See the complete working memory example:

cd mojentic-ru
cargo run --example working_memory

The example demonstrates:

• Initializing memory with user data
• RequestAgent that learns from conversation
• Event-driven coordination with AsyncDispatcher
• Memory persistence across interactions

API Reference

SharedWorkingMemory

#![allow(unused)]
fn main() {
impl SharedWorkingMemory {
    /// Create new memory with initial data
    pub fn new(initial: Value) -> Self

    /// Get current memory snapshot
    pub fn get_working_memory(&self) -> Value

    /// Deep merge updates into memory
    pub fn merge_to_working_memory(&self, updates: &Value)
}
}

See src/context/shared_working_memory.rs for full documentation.

Thread Safety

SharedWorkingMemory is thread-safe and can be shared across async tasks:

#![allow(unused)]
fn main() {
use tokio::task;

let memory = Arc::new(SharedWorkingMemory::new(initial));

let task1 = {
    let memory = Arc::clone(&memory);
    task::spawn(async move {
        memory.merge_to_working_memory(&updates1);
    })
};

let task2 = {
    let memory = Arc::clone(&memory);
    task::spawn(async move {
        memory.merge_to_working_memory(&updates2);
    })
};

task1.await?;
task2.await?;
}

Observability

Tracing and introspection help debug agentic flows, measure latency, and optimize tool usage.

Components:

• Tracer API for spans/events
• Instrumented broker calls
• Future: metrics export

Tracer System

The tracer system provides comprehensive observability into LLM interactions, tool executions, and agent communications. It records events with timestamps and correlation IDs, enabling detailed debugging and monitoring.

Goals:

• Performance insight
• Failure localization
• Reproducibility of agent flows

Architecture

The tracer system consists of several key components:

Core Components

• TracerEvent: Base trait for all event types with timestamps, correlation IDs, and printable summaries
• EventStore: Thread-safe storage for events with callbacks and filtering capabilities
• TracerSystem: Coordination layer providing convenience methods for recording events
• NullTracer: Null object pattern implementation for when tracing is disabled

Event Types

The system supports four main event types:

1. LlmCallTracerEvent: Records LLM calls with model, messages, temperature, and available tools
2. LlmResponseTracerEvent: Records LLM responses with content, tool calls, duration, and provider evidence (usage, provider model, finish reason, metadata)
3. ToolCallTracerEvent: Records tool executions with arguments, results, and duration
4. AgentInteractionTracerEvent: Records agent-to-agent communications

Usage

Basic Setup

#![allow(unused)]
fn main() {
use mojentic::tracer::TracerSystem;
use std::sync::Arc;

// Create a tracer system
let tracer = Arc::new(TracerSystem::default());

// Enable/disable tracing
tracer.enable();
tracer.disable();
}

Recording Events

LLM Calls

#![allow(unused)]
fn main() {
use std::collections::HashMap;

tracer.record_llm_call(
    "llama3.2",                    // model
    vec![],                        // messages (simplified)
    Some(0.7),                     // caller-selected temperature; None uses provider settings
    None,                          // tools
    "my_broker",                   // source
    "correlation-123"              // correlation_id
);
}

LLM Responses

#![allow(unused)]
fn main() {
tracer.record_llm_response(
    "llama3.2",                    // model
    "Response text",               // content
    None,                          // tool_calls
    Some(150.5),                   // call_duration_ms
    "my_broker",                   // source
    "correlation-123"              // correlation_id
);
}

Provider evidence in response events

LlmResponseTracerEvent has an evidence field of type ResponseEvidence. It holds what the provider reported about the response:

The event’s model field stays the model you configured on the broker. When the event is serialized, the evidence becomes the flat fields usage, provider_model, finish_reason and metadata.

The broker fills the evidence for generate_response, generate, generate_object and generate_stream_events. The older generate_stream does not trace provider evidence.

Mojentic does not estimate usage from text length or a tokenizer. If the provider does not report usage, usage stays None. OpenAI usage is the response’s usage object, unchanged. Ollama reports no usage object, so its usage is {"prompt_eval_count": ..., "eval_count": ...}, holding only the counts Ollama reported, under Ollama’s names.

To record evidence yourself, use record_llm_response_with_evidence:

#![allow(unused)]
fn main() {
use mojentic::llm::ResponseEvidence;

tracer.record_llm_response_with_evidence(
    "llama3.2",
    "Response text",
    None,
    Some(150.5),
    ResponseEvidence {
        finish_reason: Some("stop".to_string()),
        ..Default::default()
    },
    "my_broker",
    "correlation-123",
);
}

Tool Calls

#![allow(unused)]
fn main() {
use serde_json::json;

let mut arguments = HashMap::new();
arguments.insert("input".to_string(), json!("test data"));

tracer.record_tool_call(
    "date_tool",                   // tool_name
    arguments,                     // arguments
    json!({"result": "2024-01-15"}), // result
    Some("agent1".to_string()),    // caller
    Some(25.0),                    // call_duration_ms
    "tool_executor",               // source
    "correlation-123"              // correlation_id
);
}

Agent Interactions

#![allow(unused)]
fn main() {
tracer.record_agent_interaction(
    "agent1",                      // from_agent
    "agent2",                      // to_agent
    "request",                     // event_type
    Some("evt-456".to_string()),   // event_id
    "dispatcher",                  // source
    "correlation-123"              // correlation_id
);
}

Querying Events

Get All Events

#![allow(unused)]
fn main() {
let summaries = tracer.get_event_summaries(None, None, None);
for summary in summaries {
    println!("{}", summary);
}
}

Filter by Time Range

#![allow(unused)]
fn main() {
use std::time::{SystemTime, UNIX_EPOCH};

let now = SystemTime::now()
    .duration_since(UNIX_EPOCH)
    .unwrap()
    .as_secs_f64();

let one_hour_ago = now - 3600.0;

let recent_events = tracer.get_event_summaries(
    Some(one_hour_ago),  // start_time
    Some(now),           // end_time
    None                 // filter_func
);
}

Filter by Custom Predicate

#![allow(unused)]
fn main() {
// Find all events from a specific correlation ID
let correlation_id = "correlation-123";
let related_events = tracer.get_event_summaries(
    None,
    None,
    Some(&|event| event.correlation_id() == correlation_id)
);

// Count tool call events
let tool_call_count = tracer.count_events(
    None,
    None,
    Some(&|event| {
        event.printable_summary().contains("ToolCallTracerEvent")
    })
);
}

Get Last N Events

#![allow(unused)]
fn main() {
// Get last 10 events
let last_events = tracer.get_last_n_summaries(10, None);

// Get last 5 LLM-related events
let last_llm_events = tracer.get_last_n_summaries(
    5,
    Some(&|event| {
        let summary = event.printable_summary();
        summary.contains("LlmCallTracerEvent") ||
        summary.contains("LlmResponseTracerEvent")
    })
);
}

Managing Events

#![allow(unused)]
fn main() {
// Get event count
let total_events = tracer.len();
println!("Total events: {}", total_events);

// Check if empty
if tracer.is_empty() {
    println!("No events recorded");
}

// Clear all events
tracer.clear();
}

Correlation IDs

Correlation IDs are UUIDs that track related events across the system. They enable you to trace all events related to a single request or operation, creating a complete audit trail.

Best Practices

1. Generate Once: Create a correlation ID at the start of a request
2. Pass Through: Copy the correlation ID to all downstream operations
3. Query Together: Use correlation IDs to filter related events

Example flow:

User Request → Generate correlation_id
  ↓
LLM Call (correlation_id: "abc-123")
  ↓
LLM Response (correlation_id: "abc-123")
  ↓
Tool Call (correlation_id: "abc-123")
  ↓
LLM Call with tool result (correlation_id: "abc-123")
  ↓
Final LLM Response (correlation_id: "abc-123")

Event Store Callbacks

You can register a callback function that’s called whenever an event is stored:

#![allow(unused)]
fn main() {
use mojentic::tracer::{EventStore, EventCallback, TracerSystem};
use std::sync::{Arc, atomic::{AtomicUsize, Ordering}};

// Create a counter
let event_count = Arc::new(AtomicUsize::new(0));
let count_clone = Arc::clone(&event_count);

// Create callback
let callback: EventCallback = Arc::new(move |event| {
    count_clone.fetch_add(1, Ordering::SeqCst);
    println!("Event stored: {}", event.printable_summary());
});

// Create event store with callback
let event_store = Arc::new(EventStore::new(Some(callback)));

// Create tracer with custom event store
let tracer = Arc::new(TracerSystem::new(Some(event_store), true));

// Events will trigger the callback
tracer.record_llm_call("llama3.2", vec![], 1.0, None, "test", "corr-1");

println!("Total events: {}", event_count.load(Ordering::SeqCst));
}

Built-in events implement TracerEvent::as_any, so a callback can read typed fields. This callback prints the usage of each LLM response:

#![allow(unused)]
fn main() {
use mojentic::tracer::{EventCallback, LlmResponseTracerEvent};

let callback: EventCallback = Arc::new(|event| {
    if let Some(response) = event
        .as_any()
        .and_then(|any| any.downcast_ref::<LlmResponseTracerEvent>())
    {
        println!("usage: {:?}", response.evidence.usage);
    }
});
}

Null Tracer

For environments where tracing should be completely disabled without conditional checks:

#![allow(unused)]
fn main() {
use mojentic::tracer::NullTracer;
use std::collections::HashMap;
use serde_json::json;

let tracer = NullTracer::new();

// All operations are no-ops
tracer.record_llm_call("model", vec![], 1.0, None, "source", "id");
tracer.record_tool_call("tool", HashMap::new(), json!({}), None, None, "source", "id");

// Queries return empty results
assert!(tracer.is_empty());
assert_eq!(tracer.len(), 0);
assert_eq!(tracer.get_event_summaries(None, None, None).len(), 0);
}

Performance Considerations

1. Thread Safety: EventStore uses Arc<Mutex<>> for thread-safe access
2. Memory: Events are stored in memory; clear periodically for long-running processes
3. Overhead: Minimal when disabled; consider NullTracer for zero overhead
4. Callbacks: Keep callback functions fast to avoid blocking event recording

Integration with LlmBroker

The tracer integrates with LlmBroker to automatically record LLM interactions (implementation in progress):

use mojentic::llm::{LlmBroker, LlmMessage};
use mojentic::llm::gateways::OllamaGateway;
use mojentic::tracer::TracerSystem;
use std::sync::Arc;

// Create tracer
let tracer = Arc::new(TracerSystem::default());

// Create broker with tracer
let gateway = Arc::new(OllamaGateway::default());
let broker = LlmBroker::builder("llama3.2", gateway)
    .with_tracer(tracer.clone())
    .build();

// Broker will automatically record events
let response = broker.generate(
    &[LlmMessage::user("Hello!")],
    None,
    None
).await?;

// Query tracer for events
let events = tracer.get_event_summaries(None, None, None);
for event in events {
    println!("{}", event);
}

Testing

The tracer system includes comprehensive unit tests covering:

• Event creation and formatting
• Event storage and retrieval
• Callbacks and filtering
• TracerSystem coordination
• NullTracer behavior

Run tests with:

cargo test tracer

Implementation Status

✅ Layer 2 Tracer System - Core Complete

Current implementation:

• ✅ TracerEvent trait with 4 event types
• ✅ EventStore with callbacks and filtering
• ✅ TracerSystem coordination layer
• ✅ NullTracer for zero-overhead tracing
• ✅ 24 comprehensive unit tests (all passing)
• ✅ Correlation ID support
• ✅ Documentation complete

Pending integration:

• ⏳ LlmBroker integration (add tracer parameter)
• ⏳ Tool system integration (add tracer parameter)
• ⏳ Example application (tracer_demo.rs)

See Also

• Observability Overview
• Error Handling
• LLM Integration

Comprehensive Guide

End-to-end walkthrough of observing a multi-agent run.

(Placeholder for full example once agents crate stabilized.)

API Documentation

The full Rust API reference is generated with cargo doc and published under /api/ of the documentation site.

• Visit: /api/
• Crate landing page: /api/mojentic/

Note: During local development, run:

# Build API docs
cargo doc --no-deps --all-features
# Build the mdBook
mdbook build book
# Preview by serving book/book/ and opening ./api/ in a second tab from target/doc/

Contributing

We welcome issues and PRs. Please follow Rust style (rustfmt, clippy) and add tests for new behavior.

Extending Mojentic

This guide shows how to extend the Mojentic framework with new gateways and custom functionality.

Adding a New LLM Gateway

To add support for a new LLM provider, implement the LlmGateway trait:

1. Create the Gateway File

#![allow(unused)]
fn main() {
// src/llm/gateways/openai.rs
use crate::error::{MojenticError, Result};
use crate::llm::gateway::{CompletionConfig, LlmGateway};
use crate::llm::models::{LlmGatewayResponse, LlmMessage};
use crate::llm::tools::LlmTool;
use async_trait::async_trait;
use serde_json::Value;

pub struct OpenAiGateway {
    client: reqwest::Client,
    api_key: String,
    base_url: String,
}

impl OpenAiGateway {
    pub fn new(api_key: String) -> Self {
        Self {
            client: reqwest::Client::new(),
            api_key,
            base_url: "https://api.openai.com/v1".to_string(),
        }
    }
}

#[async_trait]
impl LlmGateway for OpenAiGateway {
    async fn complete(
        &self,
        model: &str,
        messages: &[LlmMessage],
        tools: Option<&[Box<dyn LlmTool>]>,
        config: &CompletionConfig,
    ) -> Result<LlmGatewayResponse> {
        // Implement OpenAI completion API call
        todo!()
    }

    async fn complete_json(
        &self,
        model: &str,
        messages: &[LlmMessage],
        schema: Value,
        config: &CompletionConfig,
    ) -> Result<Value> {
        // Implement OpenAI structured output
        todo!()
    }

    async fn get_available_models(&self) -> Result<Vec<String>> {
        // Implement model listing
        todo!()
    }

    async fn calculate_embeddings(
        &self,
        text: &str,
        model: Option<&str>,
    ) -> Result<Vec<f32>> {
        // Implement embeddings API
        todo!()
    }
}
}

2. Export the Gateway

#![allow(unused)]
fn main() {
// src/llm/gateways/mod.rs
pub mod ollama;
pub mod openai;  // Add this line

pub use ollama::{OllamaConfig, OllamaGateway};
pub use openai::OpenAiGateway;  // Add this line
}

3. Use the Gateway

use mojentic::prelude::*;
use mojentic::llm::gateways::OpenAiGateway;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<()> {
    let gateway = OpenAiGateway::new("your-api-key".to_string());
    let broker = LlmBroker::new("gpt-4", Arc::new(gateway));

    // Use as normal
    let messages = vec![LlmMessage::user("Hello!")];
    let response = broker.generate(&messages, None, None).await?;

    Ok(())
}

Creating Custom Message Types

You can create helper functions for specific message types:

#![allow(unused)]
fn main() {
use mojentic::llm::models::{LlmMessage, MessageRole};

impl LlmMessage {
    /// Create a message with an image
    pub fn user_with_image(content: impl Into<String>, image_path: impl Into<String>) -> Self {
        Self {
            role: MessageRole::User,
            content: Some(content.into()),
            tool_calls: None,
            image_paths: Some(vec![image_path.into()]),
        }
    }

    /// Create a system message with specific formatting
    pub fn system_instruction(instruction: impl Into<String>) -> Self {
        let content = format!("SYSTEM INSTRUCTION: {}", instruction.into());
        Self::system(content)
    }
}
}

Implementing Structured Output Types

Define your own structured output types with JSON Schema:

use serde::{Deserialize, Serialize};
use schemars::JsonSchema;

#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct ProductReview {
    rating: u8,
    pros: Vec<String>,
    cons: Vec<String>,
    recommendation: bool,
    summary: String,
}

#[tokio::main]
async fn main() -> mojentic::Result<()> {
    let gateway = OllamaGateway::new();
    let broker = LlmBroker::new("qwen3:32b", Arc::new(gateway));

    let messages = vec![
        LlmMessage::user(
            "Review this product: A wireless mouse with RGB lighting, \
             3200 DPI, 6 buttons, and 40-hour battery life."
        )
    ];

    let review: ProductReview = broker.generate_object(&messages, None).await?;

    println!("Rating: {}/5", review.rating);
    println!("Pros: {:?}", review.pros);
    println!("Cons: {:?}", review.cons);
    println!("Recommended: {}", review.recommendation);

    Ok(())
}

Advanced: Custom Gateway Configuration

Create configuration structs for your gateways:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone)]
pub struct CustomGatewayConfig {
    pub endpoint: String,
    pub timeout: std::time::Duration,
    pub retry_attempts: u32,
    pub custom_headers: HashMap<String, String>,
}

impl Default for CustomGatewayConfig {
    fn default() -> Self {
        Self {
            endpoint: std::env::var("CUSTOM_ENDPOINT")
                .unwrap_or_else(|_| "https://api.example.com".to_string()),
            timeout: std::time::Duration::from_secs(30),
            retry_attempts: 3,
            custom_headers: HashMap::new(),
        }
    }
}

pub struct CustomGateway {
    client: reqwest::Client,
    config: CustomGatewayConfig,
}

impl CustomGateway {
    pub fn new() -> Self {
        Self::with_config(CustomGatewayConfig::default())
    }

    pub fn with_config(config: CustomGatewayConfig) -> Self {
        let client = reqwest::Client::builder()
            .timeout(config.timeout)
            .build()
            .unwrap();

        Self { client, config }
    }
}
}

Testing Your Extensions

Create tests for your gateways and tools:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_gateway_models() {
        let gateway = OllamaGateway::new();
        let models = gateway.get_available_models().await.unwrap();
        assert!(!models.is_empty());
    }
}
}

Best Practices

1. Error Handling: Always use proper error types, never panic in library code
2. Async: All I/O operations should be async
3. Documentation: Add rustdoc comments to all public APIs
4. Testing: Write tests for your implementations
5. Configuration: Support environment variables for API keys and endpoints
6. Type Safety: Leverage Rust’s type system for compile-time guarantees

Need Help?

• Check the existing implementations in src/llm/gateways/ollama.rs
• Review the tool example in examples/tool_usage.rs
• See the Building Tools guide for tool development

Testing

• Unit tests: cargo test --all-features
• Lints: cargo clippy -- -D warnings
• Formatting: cargo fmt -- --check

User Personas

• Framework user: wants ready-to-run examples
• Library integrator: needs stable APIs and performance
• Contributor: deep dives into internals and design