Image Analysis

Mojentic supports multimodal capabilities, allowing you to analyze images with vision-capable LLM models. This guide shows you how to send images to LLMs and get intelligent responses.

Overview

Vision-capable models can analyze images and answer questions about them. Mojentic provides utilities to encode images and send them alongside text prompts in a single message.

Requirements

• A vision-capable model (e.g., llava, bakllava, qwen3-vl, gemma3:27b)
• Image files in common formats (JPEG, PNG, GIF, WebP, BMP, SVG)

Quick Example

import { LlmBroker, OllamaGateway, MessageRole, isOk } from 'mojentic';
import { imageContent, textContent } from 'mojentic/llm/utils/image';

const gateway = new OllamaGateway();
const broker = new LlmBroker('qwen3-vl:30b', gateway);

// Create a multimodal message with text and image
const message = {
  role: MessageRole.User,
  content: [
    textContent('What do you see in this image?'),
    imageContent('./path/to/image.jpg'),
  ],
};

const result = await broker.generate([message]);

if (isOk(result)) {
  console.log(result.value);
}

Image Utilities

imageToDataUri(filePath: string): string

Reads an image file and converts it to a base64 data URI suitable for LLM consumption.

import { imageToDataUri } from 'mojentic/llm/utils/image';

const dataUri = imageToDataUri('./photo.jpg');
// Returns: "data:image/jpeg;base64,/9j/4AAQSkZJRg..."

Supported formats:

• JPEG (.jpg, .jpeg)
• PNG (.png)
• GIF (.gif)
• WebP (.webp)
• BMP (.bmp)
• SVG (.svg)

imageContent(filePath: string)

Creates a ContentItem for an image, automatically handling the encoding.

import { imageContent } from 'mojentic/llm/utils/image';

const imageItem = imageContent('./diagram.png');
// Returns: { type: 'image_url', image_url: { url: 'data:image/png;base64,...' } }

textContent(text: string)

Creates a ContentItem for text. Use this when combining text and images in the same message.

import { textContent } from 'mojentic/llm/utils/image';

const textItem = textContent('Describe this diagram:');
// Returns: { type: 'text', text: 'Describe this diagram:' }

Multiple Images

You can include multiple images in a single message:

const message = {
  role: MessageRole.User,
  content: [
    textContent('Compare these two images:'),
    imageContent('./before.jpg'),
    imageContent('./after.jpg'),
    textContent('What are the key differences?'),
  ],
};

Manual Construction

If you need more control, you can construct multimodal messages manually:

import { MessageRole } from 'mojentic';
import { readFileSync } from 'fs';

function imageToDataUri(filePath: string): string {
  const imageBuffer = readFileSync(filePath);
  const base64 = imageBuffer.toString('base64');
  return `data:image/jpeg;base64,${base64}`;
}

const message = {
  role: MessageRole.User,
  content: [
    {
      type: 'text' as const,
      text: 'What is in this image?',
    },
    {
      type: 'image_url' as const,
      image_url: {
        url: imageToDataUri('./photo.jpg'),
      },
    },
  ],
};

Vision-Capable Models

Ollama Models

Popular vision models available via Ollama:

• qwen3-vl:30b - Excellent vision understanding and text extraction
• llava:latest - General-purpose vision model
• bakllava:latest - Enhanced LLaVA variant
• gemma3:27b - Google's multimodal model

Pull a model with:

ollama pull qwen3-vl:30b

Complete Example

Here's a complete working example that analyzes an image:

import { LlmBroker, OllamaGateway, MessageRole, isOk } from 'mojentic';
import { imageContent, textContent } from 'mojentic/llm/utils/image';
import { existsSync } from 'fs';
import { join } from 'path';

async function analyzeImage(imagePath: string, prompt: string) {
  // Verify image exists
  if (!existsSync(imagePath)) {
    throw new Error(`Image not found: ${imagePath}`);
  }

  // Initialize gateway and broker
  const gateway = new OllamaGateway();
  const broker = new LlmBroker('qwen3-vl:30b', gateway);

  // Create multimodal message
  const message = {
    role: MessageRole.User,
    content: [
      textContent(prompt),
      imageContent(imagePath),
    ],
  };

  console.log(`Analyzing: ${imagePath}`);
  console.log(`Prompt: ${prompt}\n`);

  // Generate response
  const result = await broker.generate([message]);

  if (isOk(result)) {
    console.log('Analysis:', result.value);
    return result.value;
  } else {
    throw result.error;
  }
}

// Usage
analyzeImage(
  './photo.jpg',
  'Describe what you see in this image in detail.'
).catch(console.error);

Use Cases

Document OCR

Extract text from images of documents, receipts, or signs:

const message = {
  role: MessageRole.User,
  content: [
    textContent('Extract all text from this document in markdown format.'),
    imageContent('./receipt.jpg'),
  ],
};

Image Comparison

Compare multiple images to identify differences or similarities:

const message = {
  role: MessageRole.User,
  content: [
    textContent('Compare these product photos and list the differences:'),
    imageContent('./product_v1.jpg'),
    imageContent('./product_v2.jpg'),
  ],
};

Diagram Understanding

Analyze technical diagrams, charts, or visualizations:

const message = {
  role: MessageRole.User,
  content: [
    textContent('Explain the architecture shown in this diagram:'),
    imageContent('./system_architecture.png'),
  ],
};

Accessibility

Generate alt text descriptions for images:

const message = {
  role: MessageRole.User,
  content: [
    textContent('Generate a concise alt text description for this image:'),
    imageContent('./photo.jpg'),
  ],
};

Implementation Notes

Base64 Encoding

imageContent reads the image file and changes it into a base64 data URI. All gateways accept data URIs. Refer to Gateway Support for the procedure that each gateway uses.

Memory Considerations

Large images consume significant memory when base64-encoded. Consider:

• Resizing images before encoding
• Processing images in batches rather than all at once
• Using appropriate model context windows

Gateway Support

OllamaGateway accepts the items that imageContent makes. It sends the base64 part of each data URI in the images field.

OpenAIGateway and OMLXGateway send each image item as an image_url content part. Their message adapter uses the value of image_url.url as follows:

• A data URI goes to the model without changes. imageContent makes this type of value.
• An http or https URL goes to the model without changes.
• The adapter reads all other values as paths to local files. It changes the file into a base64 data URI.

The adapter keeps the text items and the image items in the order that you give them. If the adapter cannot read a file, it writes an error to the console. It does not send that image.

Error Handling

import { isOk, GatewayError } from 'mojentic';

const result = await broker.generate([message]);

if (!isOk(result)) {
  if (result.error instanceof GatewayError) {
    console.error('Gateway error:', result.error.message);
    console.error('Status code:', result.error.statusCode);
  } else {
    console.error('Unexpected error:', result.error);
  }
}

Best Practices

1. Check file existence before attempting to read images
2. Use appropriate models - ensure your model supports vision
3. Provide clear prompts - be specific about what you want to analyze
4. Handle errors gracefully - network issues and missing files can occur
5. Consider image size - large images may hit token limits or memory constraints
6. Test with multiple models - different models have different strengths

See Also

• LLM Broker - Core message handling
• Tool Usage - Combining images with tool calls
• API Reference - Type definitions