@goliapkg/tiktoken-wasm

中文文档
4.1.2发布于 2026/08/13MIT OR Apache-2.0TypeScriptES Module零依赖近 30 天下载 1.8万

WASM bindings for the tiktoken BPE tokenizer

原始使用文档
文档目录23 节
  1. Install
  2. Build from source
  3. Usage
  4. ES Module (Browser / Vite / webpack)
  5. Bundler Configuration
  6. API Reference
  7. listEncodings(): string[]
  8. getEncoding(name: string): Encoding
  9. encodingForModel(model: string): Encoding
  10. modelToEncoding(model: string): string | null
  11. Encoding
  12. estimateCost(modelId, inputTokens, outputTokens): number
  13. getModelInfo(modelId): ModelInfo
  14. allModels(): ModelInfo[]
  15. modelsByProvider(provider): ModelInfo[]
  16. ModelInfo
  17. Supported Models (pricing)
  18. Notes
  19. Initialization
  20. Memory Management
  21. Demo
  22. Ecosystem
  23. License

@goliapkg/tiktoken-wasm

npm License

English | 简体中文 | 日本語

WebAssembly bindings for the tiktoken BPE tokenizer — run multi-provider tokenization directly in the browser or Node.js with near-native performance.

Install

npm install @goliapkg/tiktoken-wasm

Build from source

# requires wasm-pack: cargo install wasm-pack
cd tiktoken-wasm
wasm-pack build --target web --release

Output is in pkg/ — a complete npm-ready package containing:

  • tiktoken_wasm.js — ES module with WASM loader
  • tiktoken_wasm_bg.wasm — compiled WASM binary (6.28 MB, 5.55 MB gzipped, all 17 vocabularies)

Building it yourself, --no-default-features --features vocab-o200k_base (or any other vocab-* / vendor group forwarded from tiktoken) drops that to 1.92 MB / 1.20 MB gzipped.

  • tiktoken_wasm.d.ts — TypeScript type definitions

Usage

ES Module (Browser / Vite / webpack)

import init, {
  getEncoding,
  encodingForModel,
  listEncodings,
  modelToEncoding,
  estimateCost,
  getModelInfo,
  allModels,
  modelsByProvider,
  type Encoding,
  type ModelInfo,
} from '@goliapkg/tiktoken-wasm'

// initialize WASM module (required once, before any other calls)
await init()

// discover available encodings
const names: string[] = listEncodings()
// ["cl100k_base", "o200k_base", ..., "mistral_v3"]

// encode / decode
const enc: Encoding = getEncoding('cl100k_base')
const tokens: Uint32Array = enc.encode('hello world')
const text: string = enc.decode(tokens)   // "hello world"
const count: number = enc.count('hello world')  // 2

// special token handling
const countST: number = enc.countWithSpecialTokens('hi<|endoftext|>bye')

// vocabulary info
console.log(enc.vocabSize)         // 100256
console.log(enc.numSpecialTokens)  // 5

// by model name — supports OpenAI, Meta, DeepSeek, Qwen, Mistral
const enc2 = encodingForModel('gpt-4o')
const encName = modelToEncoding('llama-4-scout')  // "llama3"

// cost estimation (USD)
const cost: number = estimateCost('gpt-4o', 1000, 500)

// model metadata (fully typed)
const info: ModelInfo = getModelInfo('claude-opus-4')
console.log(info.id, info.provider, info.inputPer1m, info.contextWindow)

// browse all models or filter by provider
const all: ModelInfo[] = allModels()
const openai: ModelInfo[] = modelsByProvider('OpenAI')

// free WASM memory when done
enc.free()
enc2.free()

Bundler Configuration

Vite — add plugins to vite.config.ts:

import wasm from 'vite-plugin-wasm'
import topLevelAwait from 'vite-plugin-top-level-await'

export default defineConfig({
  plugins: [wasm(), topLevelAwait()],
})

webpack 5 — enable WASM experiments in webpack.config.js:

module.exports = {
  experiments: {
    asyncWebAssembly: true,
  },
}

Next.js — add to next.config.js:

module.exports = {
  webpack: (config) => {
    config.experiments = { ...config.experiments, asyncWebAssembly: true }
    return config
  },
}

API Reference

listEncodings(): string[]

List all available encoding names (17 encodings).

getEncoding(name: string): Encoding

Get a tokenizer by encoding name. Supported:

  • cl100k_base — GPT-4, GPT-3.5-turbo
  • o200k_base — GPT-4o, GPT-4.1, GPT-5–5.6, o1, o3, o4-mini
  • o200k_harmony — gpt-oss (harmony chat format)
  • p50k_base — text-davinci-002/003
  • p50k_edit — text-davinci-edit
  • r50k_base — GPT-3 (davinci, curie, etc.)
  • gpt2 — GPT-2 (alias for r50k_base)
  • llama3 — Meta Llama 3/4
  • deepseek_v3 — DeepSeek V3/R1
  • qwen2 — Qwen 2/2.5/3
  • mistral_v3 — Mistral/Codestral/Pixtral (Tekken)
  • deepseek_v4 — DeepSeek V4 Pro/Flash
  • kimi_k2 — Kimi K2/K2.5/K2.6
  • kimi_k3 — Kimi K3
  • glm4 — GLM-4.5/4.6/4.7
  • glm5 — GLM-5/5.2
  • minimax_m2 — MiniMax M2 family

encodingForModel(model: string): Encoding

Get a tokenizer by model name (e.g. gpt-4o, llama-4-scout, deepseek-r1, qwen3-max).

modelToEncoding(model: string): string | null

Map a model name to its encoding name without loading the encoding.

Encoding

Method / PropertyTypeDescription
encode(text)Uint32ArrayEncode text to token ids
encodeWithSpecialTokens(text)Uint32ArrayEncode with special token recognition
decode(tokens)stringDecode token ids to text
count(text)numberCount tokens (faster than encode().length)
countWithSpecialTokens(text)numberCount tokens with special token recognition
namestringEncoding name (getter)
vocabSizenumberNumber of regular tokens in vocabulary
numSpecialTokensnumberNumber of special tokens
free()voidRelease WASM memory

estimateCost(modelId, inputTokens, outputTokens): number

Estimate API cost in USD. Supports 116 models across 11 providers.

getModelInfo(modelId): ModelInfo

Get model metadata with full TypeScript typing.

The id is resolved the way estimateCost resolves it, so a model addressed as its API spells it (claude-haiku-4-5, anthropic.claude-opus-5) returns that model. The returned id is this table's own spelling.

allModels(): ModelInfo[]

List all 116 supported models with pricing info.

modelsByProvider(provider): ModelInfo[]

Filter models by provider: "OpenAI", "Anthropic", "Google", "Meta", "DeepSeek", "Alibaba", "Mistral", "Moonshot", "Zhipu", "MiniMax", "Voyage".

ModelInfo

PropertyTypeDescription
idstringModel identifier
providerstringProvider name
inputPer1mnumberInput cost per 1M tokens (USD)
outputPer1mnumberOutput cost per 1M tokens (USD)
cachedInputPer1mnumber | undefinedCached input cost per 1M tokens
contextWindownumberMax context window (tokens)
maxOutputnumberMax output tokens

Supported Models (pricing)

estimateCost / getModelInfo / allModels cover 116 models across 11 providers (2026-08 pricing). Per provider, newest first:

ProviderModelsLatest entries
OpenAI34gpt-5.6-sol/terra/luna, gpt-5.5(-pro), gpt-5.4(-mini/nano/pro), gpt-5.2(-pro), gpt-5.1, gpt-5(-mini/nano/pro), gpt-4.1, gpt-4o, o1/o3/o4-mini, …
Anthropic17claude-fable-5, claude-mythos-5, claude-opus-5, claude-sonnet-5, claude-opus-4.8/4.7/4.6, claude-haiku-4.5, …
Google13gemini-3.1-pro-preview, gemini-3.6/3.5-flash, gemini-2.5-pro/flash, …
Mistral12mistral-large/medium/small, codestral, devstral, ministral, pixtral-large, …
Alibaba9qwen3.8-max, qwen3.5-plus, qwen3-max/plus/coder, qwen2.5-72b, …
Meta6llama-4-scout/maverick, llama-3.3-70b, llama-3.1-405b/70b/8b
Zhipu5glm-5.2, glm-5, glm-4.7, glm-4.5(-air)
DeepSeek4deepseek-v4-pro, deepseek-v4-flash, deepseek-v3, deepseek-r1
MiniMax4minimax-m2.7, minimax-m2.5, minimax-m2.1, minimax-m2
Moonshot4kimi-k3, kimi-k2.7-code, kimi-k2.6, kimi-k2.5
Voyage8voyage-4-large/4/4-lite/code-4, voyage-3-large, voyage-3.5(-lite), voyage-code-3 (embeddings)

The authoritative list is the code: call allModels() at runtime, or see docs.rs. Anthropic and Google models are pricing-only (no public tokenizer to match); the other eight providers tokenize too.

Notes

Initialization

Call await init() once before any other API calls. This loads and compiles the WASM module. Subsequent calls are a no-op.

Memory Management

Encoding instances hold references to globally cached data and are lightweight. Calling .free() releases the JS wrapper — the underlying encoding data remains cached for reuse. In short-lived scripts you can skip .free(); in long-running apps, call it when you're done with the instance.

Demo

See examples/react-app for a complete Vite + React demo application.

Ecosystem

tiktoken · @goliapkg/tiktoken-wasm · instructors · chunkedrs · embedrs

License

Licensed under either of MIT or Apache-2.0, at your option.

版本与发布记录

@goliapkg/tiktoken-wasm 共发布 24 个版本,首次发布于 2026/03/07;过去 12 个月发布了 24 个版本。

版本发布日期距今
4.1.2latest2026/08/133 个月前
4.1.12026/08/133 个月前
4.0.12026/08/133 个月前
4.0.02026/08/133 个月前
3.8.32026/08/103 个月前
3.8.22026/08/093 个月前

关于 @goliapkg/tiktoken-wasm 的常见问题

@goliapkg/tiktoken-wasm 是什么?

WASM bindings for the tiktoken BPE tokenizer

如何安装 @goliapkg/tiktoken-wasm?

在项目目录运行 npm install @goliapkg/tiktoken-wasm,或使用 pnpm add / yarn add / bun add。

@goliapkg/tiktoken-wasm 支持 TypeScript 吗?

支持。包内自带 TypeScript 类型声明(package.json 声明了 types 入口),安装后即可获得类型提示,无需再安装 @types 包。

@goliapkg/tiktoken-wasm 支持 ESM 还是 CommonJS?

以 ES Module 形式发布,推荐使用 import 引入。在 CommonJS 项目中,Node.js 20.19+ / 22.12+ 可以直接 require() 不含顶层 await 的 ES Module,更早的版本需要改用动态 import()。

@goliapkg/tiktoken-wasm 可以商用吗?

@goliapkg/tiktoken-wasm 使用 MIT OR Apache-2.0 许可。可在 MIT、Apache-2.0 中任选其一遵守。选择 MIT 时,宽松许可:允许商业使用、修改和再分发,通常只需在分发时保留原版权与许可声明。以上为概括,不构成法律意见,请以许可原文为准。

@goliapkg/tiktoken-wasm 的下载量有多少?

根据 npm 官方下载统计,@goliapkg/tiktoken-wasm 在 2026/10/17 – 2026/11/15 这 30 天内被下载 17,657 次,日均约 589 次。下载量会把 CI 构建、镜像同步和作为其他包依赖的安装一并计入,适合横向比较热度,不能直接等同于使用人数。

@goliapkg/tiktoken-wasm 还在维护吗?

最新版本 4.1.2 发布于 2026/08/13(3 个月前);至今共发布 24 个版本,其中过去 12 个月发布了 24 个;首次发布于 2026/03/07。发布频率只是参考信号:功能稳定的工具可能很少需要更新,建议结合源码仓库的 issue 和提交记录判断。