mojentic-openai/com.mojentic.openai/JtokkitTokenizerGateway

JtokkitTokenizerGateway

class JtokkitTokenizerGateway(model: String) : TokenizerGateway

JVM-only TokenizerGateway backed by jtokkit.

Resolves the OpenAI encoding for a given model name via jtokkit's registry; falls back to the o200k_base (GPT-4o family) encoding when the model isn't recognised.

Constructors

constructor(model: String)

Types

Companion

object Companion

Functions

countTokens

open override fun countTokens(text: String): Int

decode

open override fun decode(tokens: List<Int>): String

encode

open override fun encode(text: String): List<Int>