@audio/neural-runtime
One inference adapter: load an ONNX model, run it in Node or the browser, worklet-ready.
Every neural atom needs the same three things — load bytes, run tensors through a model, free the session — on top of two different runtimes (onnxruntime-node on the server, onnxruntime-web in the browser/worker/worklet). This package is that one adapter: asr, align, separate, diarize, tts, denoise and anything else in the lane build on it instead of each wiring ONNX Runtime themselves.
npm install @audio/neural-runtime
import { load, tensor } from '@audio/neural-runtime'
let session = await load('https://example.com/model.onnx') // fetched, cached, progress-able
let out = await session.run({ input: tensor(new Float32Array(...), [1, 40, 1]) })
out.output.data // Float32Array | Int32Array | BigInt64Array | Uint8Array | string[]
session.free()
load accepts a URL string, Uint8Array, or ArrayBuffer. tensor(data, dims, type?) builds a plain { data, dims, type } — no ORT import at call time; the adapter converts it to ort.Tensor lazily, inside run(), once a backend is resolved.
Backends
opts.backend | Runtime | Environment |
|---|---|---|
'auto' (default) | picks 'node' in Node, 'wasm' in a browser/worker/worklet | either |
'node' | onnxruntime-node | Node only |
'wasm' | onnxruntime-web/wasm | browser/worker/worklet only |
'webgpu' | onnxruntime-web/webgpu; in Node, onnxruntime-node's WebGPU provider (Dawn) | browser/worker/worklet; Node |
'coreml' | onnxruntime-node's CoreML provider | Node on macOS |
'webgl' | onnxruntime-web/webgl | browser/worker/worklet only |
Neither onnxruntime-node nor onnxruntime-web (1.17 or later) is a hard dependency — both are optional peers (peerDependenciesMeta.optional). Each backend's build is a literal import(), which a bundler or a CDN such as esm.sh resolves, loaded the first time that backend is used and memoized after. Which runtime a build gets is package.json's choice, not a guess at the environment: "imports": { "#ort": { "node": "./ort-node.js", "default": "./ort-web.js" } } gives Node (and Bun, Deno) onnxruntime-node and everything else onnxruntime-web, so a browser build never meets onnxruntime-node, and a process polyfill (esm.sh's, a bundler's) can't send a page down the Node path. Asking for a backend your environment can't run ('wasm' in Node, 'node' in a browser) throws immediately, naming what's missing and where it does work; a resolvable but uninstalled package throws naming the install command. backends() reports what's actually usable here: ['node'] in Node, with 'webgpu' and 'coreml' where its onnxruntime-node has them, ['wasm'] or ['wasm', 'webgpu'] (when navigator.gpu gives an adapter that is no fallback, not SwiftShader) in a browser.
run() queues concurrent calls onto one sequential chain per session — safe regardless of whether the underlying ORT build tolerates overlapping run() calls. free() maps to session.release(), is idempotent, and run() after free() throws.
inputs/outputs on the session report { name, dims, type } when the installed ORT version exposes inputMetadata/outputMetadata (present since onnxruntime-common added shape/type reporting), else { name } only — never guessed.
Caching
load(url, opts) and fetchModel(url, opts) (the fetch primitive, also used by fetchJson for tokenizer/config files hosted next to a model) cache by default (opts.cache = false to disable):
- Node:
~/.cache/audiojs/neural/<sha256(url)>, override the base directory with$AUDIO_NEURAL_CACHE. A sidecar.sizefile records the expected byte count; a mismatch (truncated or otherwise corrupted entry) is treated as a cache miss and re-fetched. - Browser: the Cache API, store
'audio-neural', keyed by URL. file://URLs read straight from disk (Node only), bypassing the cache — there's nothing to cache against.opts.progress?.({ loaded, total })fires as bytes arrive;totalisnullwhen the server doesn't sendContent-Length.opts.fetchoverrides the ambientfetch(a custom client, auth headers, a mock in tests).
Worklets
onnxruntime-web loads its .wasm binary from a URL it resolves relative to its own module by default — a resolution an AudioWorkletProcessor can't do (no relative module location, no document). Pass opts.wasmPaths (a URL prefix string, or a { 'ort-wasm.wasm': url, ... } map) to point it at wherever you're serving the onnxruntime-web assets; it becomes ort.env.wasm.wasmPaths before the session is created. opts.threads sets ort.env.wasm.numThreads (SharedArrayBuffer + cross-origin isolation required for >1).
API
| Export | |
|---|---|
load(model, opts?) | → Promise<Session> — model: URL string | Uint8Array | ArrayBuffer |
tensor(data, dims, type?) | → plain Tensor; type inferred from data's constructor when omitted |
backends() | → Promise<string[]> — backends usable in this environment |
fetchModel(url, opts?) | → Promise<Uint8Array> — cached fetch with progress |
fetchJson(url, opts?) | → Promise<unknown> — same cache/progress path, JSON.parsed |
Session: { run(feeds, outputNames?) → Promise<Record<string, Tensor>>, inputs, outputs, backend, free() }.
load(model, opts) — opts: { backend = 'auto', threads?, cache = true, progress?, fetch?, executionProviders?, graphOptimizationLevel?: 'all' | 'basic' | 'disabled', sessionOptions?, logLevel?, wasmPaths? }. sessionOptions passes any other ort.InferenceSession option through; { enableCpuMemArena: false, enableMemPattern: false } returns activation memory after each run instead of pooling it (Hybrid Transformer Demucs, one 7.8 s segment in onnxruntime-node 1.30: peak RSS 2.7 GB against 3.2 GB).
Not for: choosing or shipping a model — this package has no model zoo, no bundled weights, and no opinion on architecture. neural-amp and the rest of the lane bring the model; this loads and runs it.
Part of the @audio/neural lane. Weights are out of scope here — see the umbrella README's weights-licensing policy for anything that ships pretrained parameters.
MIT © audiojs