LLM Broker Guide

The LlmBroker is the central interface for interacting with Large Language Models in Mojentic. It provides a consistent API across different LLM providers (gateways) and handles tool execution, error handling, and message management.

Overview

The Broker acts as an intermediary between your application and LLM providers:

Your App → Broker → Gateway → LLM Provider
                ↓
            Tool Execution

Creating a Broker

import { LlmBroker, OllamaGateway } from 'mojentic';

// Basic broker
const gateway = new OllamaGateway();
const broker = new LlmBroker('qwen3:32b', gateway);

Broker Structure

class LlmBroker {
  constructor(
    private readonly model: string,        // Model identifier
    private readonly gateway: LlmGateway   // Gateway implementation
  )
}

Text Generation

The primary use case is generating text responses:

import { Message, isOk } from 'mojentic';

const messages = [Message.user('Explain TypeScript in one sentence')];

const result = await broker.generate(messages);

if (isOk(result)) {
  console.log(result.value);
  // "TypeScript is a statically-typed superset of JavaScript..."
} else {
  console.error('Generation failed:', result.error);
}

With Configuration

const config: CompletionConfig = {
  temperature: 0.3,  // Lower = more focused
  maxTokens: 500
};

const result = await broker.generate(messages, undefined, config);

With Tools

import { DateResolverTool } from 'mojentic';

const tools = [new DateResolverTool()];
const result = await broker.generate(messages, tools, config);

Structured Output

Generate responses conforming to a JSON schema:

const schema = {
  type: 'object',
  properties: {
    title: { type: 'string' },
    summary: { type: 'string' },
    keywords: {
      type: 'array',
      items: { type: 'string' }
    }
  },
  required: ['title', 'summary']
};

const messages = [Message.user('Analyze: TypeScript is a typed language')];

const result = await broker.generateObject(messages, schema);

if (isOk(result)) {
  console.log(result.value);
  // {
  //   title: "TypeScript Language Analysis",
  //   summary: "Typed superset of JavaScript...",
  //   keywords: ["typed", "compiled", "JavaScript"]
  // }
}

Tool Execution Flow

When tools are provided, the Broker handles a recursive loop:

1. Send messages to LLM
2. If LLM requests tools:
   • Execute each tool
   • Add tool results to conversation
   • Return to step 1
3. Return final text response

// Example: Date resolution tool usage
const messages = [Message.user("What's the date next Monday?")];
const tools = [new DateResolverTool()];

// The broker will:
// 1. Call LLM with the question
// 2. LLM responds with tool call request
// 3. Broker executes DateResolverTool.run(args)
// 4. Broker adds result to conversation
// 5. Calls LLM again with tool result
// 6. LLM responds with final answer
const result = await broker.generate(messages, tools);

Tool Call Handling

The broker automatically:

• Matches tool calls to available tools
• Executes tools with provided arguments
• Handles tool errors gracefully
• Manages conversation state

Streaming

Generate responses in real-time:

const messages = [Message.user('Write a short poem about TypeScript')];

for await (const chunk of broker.generateStream(messages)) {
  if (isOk(chunk)) {
    process.stdout.write(chunk.value);
  } else {
    console.error('Stream error:', chunk.error);
    break;
  }
}

Message Management

The Broker accepts a list of messages representing the conversation:

const messages = [
  Message.system('You are a helpful coding assistant'),
  Message.user('How do I read a file in Node.js?'),
  Message.assistant('You can use fs.readFile()...'),
  Message.user('What about async/await?')
];

const result = await broker.generate(messages);

Message Types

• Message.system(content) - Set LLM behavior and context
• Message.user(content) - User input
• Message.assistant(content, toolCalls?) - LLM responses (for history)
• Message.tool(content, toolCallId, name) - Tool execution results

Error Handling

The Broker returns Result types for explicit error handling:

import { isOk, isErr, unwrap, unwrapOr } from 'mojentic';

const result = await broker.generate(messages);

// Pattern 1: Type guard
if (isOk(result)) {
  console.log(result.value);
} else {
  console.error(result.error);
}

// Pattern 2: Unwrap (throws on error)
try {
  const text = unwrap(result);
} catch (error) {
  console.error('Failed:', error);
}

// Pattern 3: Unwrap with default
const text = unwrapOr(result, 'Default response');

Error Types

import {
  MojenticError,   // Base error
  GatewayError,    // API/network errors
  ToolError,       // Tool execution errors
  ParseError,      // JSON parsing errors
  TimeoutError     // Timeout errors
} from 'mojentic';

Configuration Options

Fine-tune LLM behavior:

// Creative writing
const creativeConfig: CompletionConfig = {
  temperature: 1.5,
  maxTokens: 2000
};

// Factual responses
const factualConfig: CompletionConfig = {
  temperature: 0.1,
  maxTokens: 500
};

// JSON output
const jsonConfig: CompletionConfig = {
  responseFormat: {
    type: 'json_object'
  }
};

Reasoning Effort

Control how much the model thinks before responding:

// Deep reasoning for complex problems
const thinkingConfig: CompletionConfig = {
  reasoningEffort: 'high',
  temperature: 0.1
};

// Quick responses
const fastConfig: CompletionConfig = {
  reasoningEffort: 'low'
};

Ollama sends think: false for disabled, think: true for enabled, and the exact string for low, medium, high or xhigh. oMLX sends top-level enable_thinking: false/true for disabled/enabled and top-level reasoning_effort with the exact string for named levels. The oMLX fields are mutually exclusive. Omission sends no reasoning control and preserves provider defaults on both gateways.

OpenAI sends reasoning_effort for low, medium or high on reasoning models and warns and drops these levels on other models. Under the 3.0.0 rule, disabled, enabled and xhigh fail with ValidationError before HTTP dispatch on every model, including recovery-enabled requests. These mappings apply to ordinary, structured and streaming requests. See Reasoning Effort for full details.

Best Practices

1. Use System Messages

Set clear instructions:

const messages = [
  Message.system(`
    You are a helpful assistant that provides concise answers.
    Always format code examples with proper syntax highlighting.
  `),
  Message.user('How do I create a class in TypeScript?')
];

2. Handle Tool Errors

Tools can fail:

import { BaseTool, Ok, Err, ToolError } from 'mojentic';

class MyTool extends BaseTool {
  async run(args: ToolArgs): Promise<Result<ToolResult, Error>> {
    try {
      const result = await this.doWork(args);
      return Ok(result);
    } catch (error) {
      return Err(new ToolError(
        `Tool failed: ${error.message}`,
        'MyTool'
      ));
    }
  }
}

3. Manage Context Windows

Long conversations need truncation:

function keepRecentMessages(messages: LlmMessage[], maxCount = 10): LlmMessage[] {
  return messages.slice(-maxCount);
}

const recentMessages = keepRecentMessages(conversationHistory);
const result = await broker.generate(recentMessages);

Advanced Usage

Multiple Models

// Use different models for different tasks
const fastBroker = new LlmBroker('phi4:14b', new OllamaGateway());
const smartBroker = new LlmBroker('qwen3:32b', new OllamaGateway());

// Quick classification
const category = await fastBroker.generate(classifyMessages);

// Deep analysis
const analysis = await smartBroker.generate(analysisMessages);

Tool Composition

// Combine multiple tools
const tools = [
  new DateResolverTool(),
  new WeatherTool(),
  new CalculatorTool()
];

// LLM can use any tool as needed
const result = await broker.generate(messages, tools);

Schema Validation

// Strict schemas ensure valid output
const schema = {
  type: 'object',
  properties: {
    confidence: { type: 'number', minimum: 0, maximum: 1 },
    category: { type: 'string', enum: ['tech', 'business', 'other'] }
  },
  required: ['confidence', 'category'],
  additionalProperties: false
};

const result = await broker.generateObject(messages, schema);

if (isOk(result)) {
  // Guaranteed to have required fields with valid values
  if (result.value.confidence > 0.8) {
    processHighConfidence(result.value);
  }
}

See Also

• Getting Started
• Tool Usage
• Structured Output
• Error Handling

Caller-owned context and native responses

Use await broker.generateResponse(messages, tools, config) to receive one native gateway response without executing tools, extending history or making a follow-up request. Assemble the complete message array before each call. The broker traces the supplied request and returned response; it does not read repository guidance or apply a context policy.

The existing convenience completion method still executes tools and follows up. Choose a serial or parallel runner according to the tools' effects. Parallel execution does not make dependent edits safe.

Set { maxToolIterations: null } to disable the tool-round limit. Existing finite defaults remain unchanged. Concurrency controls simultaneous work; it is not a task budget or a loop detector.

The optional broker toolContext forwards cancellation and completion observations. Parallel concurrency must be a positive integer.

Native responses preserve the fields supplied by the gateway. Missing provider usage or termination evidence must remain unknown; configured model names and text length are not substitutes for reported metadata.

Provider evidence in response traces

When the broker has a tracer, each LLMResponseTracerEvent from generate, generateResponse, and generateObject keeps the evidence that the gateway reported: usage, providerModel, finishReason, and metadata. The event model field stays the configured model. A field that the provider did not report is null. The broker does not estimate usage.

The legacy generateStream API keeps its current trace, without this evidence. generateStreamEvents records it when its stream reaches the terminal event.

Ollama reports usage from prompt_eval_count and eval_count. Its done_reason becomes the finish reason, and its timings go into metadata. See Tracer System.

Single-turn streaming with terminal completion evidence

generateStreamEvents(messages, config?, { correlationId?, signal? }) streams one turn as content events that end in exactly one completed or error event. Use it when a truncated or unfinished response must never authorize an action. It supplies no tools, forces zero tool iterations, performs no retry, and sends one HTTP request. Stop the iteration or abort signal to cancel the request.

const controller = new AbortController();

for await (const event of broker.generateStreamEvents(
  [Message.user('Classify this ticket as JSON.')],
  { responseFormat: { type: 'json_object', schema } },
  { signal: controller.signal }
)) {
  if (event.type === 'completed') {
    // Only now is the content a complete result. Validate it before use.
  }
}

Content received before an error event is evidence, not a result. See Streaming for the events, completion rules, and error reasons.

Opt-in completion recovery

See completion recovery for policy configuration, structured failures, admission, cancellation, and provider limits. Both streaming forms accept the same opt-in recovery policy.

maxTokens is the optional output limit across providers. Omit it to let the provider and model choose the ceiling. In 3.0, replace the removed numPredict option with maxTokens; OpenAI and oMLX no longer apply an implicit 16,384-token limit, which can increase generation time and cost. See the gateway output-limit mapping.