@framers/agentos-ext-ml-classifiers

A content classifier guardrail for @framers/agentos: one local English toxicity model run through ONNX, with an LLM judge or keyword patterns when that model is not available.

What it does

The guardrail scores text in four categories, toxic, injection, nsfw and threat, and returns FLAG or BLOCK when a score passes its threshold. It checks user input and the agent's final response; it does not read streaming chunks. The pack also registers a classify_content tool that returns the same scores on demand.

Each text goes to the first tier that answers:

1. ONNX. The guardrail imports @huggingface/transformers (an optional dependency) and loads one model, Xenova/toxic-bert, the ONNX conversion of unitary/toxic-bert, with pipeline('text-classification', 'Xenova/toxic-bert', { device: 'cpu' }). The load is tried once per guardrail. When the package is missing or the model fails to load, the guardrail writes one warning with the error and the tier it uses instead (see Logging), and goes to the next tier.
2. LLM judge. When you pass an llmInvoker callback, the guardrail asks the model for a JSON verdict on all four categories. A failed call, or a reply with every score at zero, goes to the next tier.
3. Keywords. Regular expressions per category, always available.

What the ONNX model scores

toxic-bert has six labels. The guardrail maps them to its categories and keeps the highest score per category:

• nsfw from the ONNX tier is the model's obscene label, a score for obscene language. It is not a sexual-content classifier.
• injection is never scored by the ONNX tier: it stays at 0 there. Only the LLM judge and the keyword patterns score it.
• The model is for English text: its card lists its training data as the Wikipedia comments of the Jigsaw Toxic Comment Classification Challenge, an English dataset, and offers a separate multilingual model for other languages.

The model's card states its main limit (unitary/toxic-bert):

If words that are associated with swearing, insults or profanity are present in a comment, it is likely that it will be classified as toxic, regardless of the tone or the intent of the author e.g. humorous/self-deprecating. This could present some biases towards already vulnerable minority groups.

Install

npm install @framers/agentos-ext-ml-classifiers

Peer dependency: @framers/agentos. @huggingface/transformers is an optional dependency; without it the guardrail runs the LLM judge or the keyword tier.

Quickstart

import { AgentOS } from '@framers/agentos';
import { createMLClassifierGuardrail } from '@framers/agentos-ext-ml-classifiers';

const agentos = new AgentOS();
await agentos.initialize({
  extensionManifest: {
    packs: [
      {
        factory: () =>
          createMLClassifierGuardrail({
            categories: ['toxic', 'injection', 'threat'],
            flagThreshold: 0.5,
            blockThreshold: 0.8,
            thresholds: { toxic: { flag: 0.6, block: 0.9 } },
            // Optional: an LLM judge for when the ONNX tier is unavailable.
            llmInvoker: async (systemPrompt, userMessage) => callYourModel(systemPrompt, userMessage),
          }),
        enabled: true,
      },
    ],
  },
});

A score above flagThreshold (default 0.5) flags; above blockThreshold (default 0.8) blocks. thresholds overrides either per category.

Public API

• createMLClassifierGuardrail(options?, logger?) returns an ExtensionPack with the guardrail and the classify_content tool
• createExtensionPack(context) is the factory AgentOS's extension loader calls; it reads context.options and context.logger
• createMLClassifierPack is a deprecated alias of createMLClassifierGuardrail

See src/types.ts for MLClassifierOptions, LlmInvoker and MLClassifierLogger.

Moderation: @framers/agentos-ext-ml-classifiers/moderation

A second entry point for content people send each other: an OpenAI Moderation guard used as a signal, an OpenAI Moderation check for AgentOS's image guard, and a conduct classifier on a judge model you choose. Nothing in it loads a local model.

import { PhraseListGuardrail, StaticPhraseListSource, moderateContent } from '@framers/agentos';
import {
  InMemoryConductRefusalCache, createAgentOSConductJudge, createConductGuardrail, createOpenAIModerationGuardrail,
} from '@framers/agentos-ext-ml-classifiers/moderation';

// a sequential phrase list runs first: a text it blocks reaches neither vendor
const hardBan = await PhraseListGuardrail.create({
  id: 'hard-ban', sequential: true,
  source: new StaticPhraseListSource({ version: '1', reviewedAt: '2026-11-12', reviewedBy: 'team', entries: [{ phrase: 'banned phrase', match: 'word', onMatch: 'block', ruleId: 'BAN_1' }] }),
});
const openai = createOpenAIModerationGuardrail({ apiKey: process.env.OPENAI_API_KEY!, logger });
const conduct = createConductGuardrail({
  judge: createAgentOSConductJudge({ provider: 'anthropic', model: 'claude-haiku-5-5' }),
  fallbackJudge: createAgentOSConductJudge({ provider: 'openai', model: '<a second vendor model>', thinkingHeadroomTokens: 1024 }),
  attemptTimeoutMs: 1_800,
  cache: new InMemoryConductRefusalCache(),
  logger,
});

const decision = await moderateContent({ text }, {
  guards: [hardBan, openai, conduct],
  context: { userId, sessionId, moderation: { surface: 'public_room', roomId, rating: 'standard' } },
  stopVendorsOn: ['MINOR_SEXUAL_SIGNAL'],
});
// decision.outcome: 'allow' | 'flag' | 'refuse' | 'hold'; decision.unavailable is true when only an outage decided

Needs @framers/agentos 0.17.1 or newer, the release with moderateContent(), holds and frameUntrustedEvidence(). On an older core the subpath fails when it is linked. The root entry is unchanged and keeps working on every core its peer range names.

Both entry points' type declarations use .js specifiers, so they resolve under TypeScript's NodeNext module resolution as well as bundler: Node's ESM resolution, which NodeNext enforces, "requires relative paths to include file extensions" (TypeScript's module theory). The test suite packs the built package and type-checks a NodeNext consumer against it, skipLibCheck off.

The OpenAI Moderation guard

createOpenAIModerationGuardrail(options) sends the input's text, and nothing else, to POST /v1/moderations (model omni-moderation-latest by default; CreateModerationRequest and CreateModerationResponse in OpenAI's OpenAPI file). It reads the first result by walking its own list of thirteen categories (OPENAI_MODERATION_CATEGORIES): a key outside the list is ignored, and a value of another type makes the answer unusable. So does a result whose categories or category_scores object is missing or null, and so does one that leaves out a category's boolean or score: OpenAI's schema lists all thirteen in both objects, so an answer with empty or incomplete maps judged nothing and is never read as a clean one. A closed guard refuses it as an outage. One exception: the schema declares the booleans of illicit and illicit/violent nullable (OpenAI's OpenAPI file, CreateModerationResponse). A null boolean or a null score on those two reads as false with no score, and the other eleven decide; their keys must still be there. A null on any other category makes the answer unusable. The model named in a verdict is the one the guard was built with, never the answer's.

• Rules. Each category has a rule { flagAt?, holdAt?, blockAt?, reviewCode? }. A number is a score threshold from 0 to 1, met when the score is at or above it; 'vendor' means OpenAI's own boolean for the category. A block outranks a hold and a hold a flag. Any other threshold (NaN, Infinity, a number above 1 or below 0, a string) is a TypeError when the guard or the image check is built; its message names the option, never the value. An empty rule ({}) is valid and takes the category's rule away.
• Defaults. Every category flags at 'vendor'; sexual/minors also holds at 'vendor' with review code MINOR_SEXUAL_SIGNAL. Nothing blocks: OpenAI's scores refuse no message unless you set a blockAt.
• Your rules. A rule you pass for a category replaces the library's rule for that category whole. A rule you pass for 'sexual/minors' takes its hold away unless it names a holdAt itself. A hold whose rule names no reviewCode is held under the category's own code.
• Reason codes. OPENAI_MODERATION_ and the category in upper case, / and - written _ (OPENAI_MODERATION_SEXUAL_MINORS), plus the rules' review codes.
• Down. A network error, a non-2xx answer, a timeout (timeoutMs, default 3000, which bounds the whole request: the guard stops waiting for an apiKey function at that point, and nothing is sent) or an answer of another shape skips the guard with one moderation.signal_skipped line, and the other guards decide. failClosed: true makes it an outage instead. The body of an error answer is never read; it is cancelled at once, so its connection is freed. A successful answer is read up to 1 MiB (MAX_MODERATION_ANSWER_BYTES in the source; an answer for one input is a few kilobytes): a Content-Length past it is refused before the body is read, and a body that runs past it is cancelled there. Either is an answer of another shape.
• Phase. It is a parallel (phase 2) guard and reads the call's vendorSignal before its request, and again once its key is resolved (an apiKey function may take a while): once a code in stopVendorsOn has aborted it, the guard sends nothing. It reads the dispatcher's deadlineSignal at the same two points, the second right before the request: once the dispatcher has stopped waiting for the guard (its deadline, or a shorter one a required posture gives it), the text is sent nowhere.

The image check

openAIModerationImageCheck(options) is a classify check for AgentOS's image guard, ImageGuardrail, which needs @framers/agentos 0.17.1 or newer. It takes the OpenAI guard's options except stages and failClosed; its default id is openai-moderation-image.

• What it sends. One image per request, always as a data URL of the bytes the image guard captured (data:<mime type>;base64,...), never a URL. For an image_url input the guard fetches the image once through your loadImage, and OpenAI judges those bytes, the same ones a hash check saw.
• What it reads. The six categories OpenAI applies to images: self-harm, self-harm/intent, self-harm/instructions, sexual, violence and violence/graphic (category_applied_input_types in CreateModerationResponse, OpenAI's OpenAPI file). The seven others apply to text alone: the check reads none of them, so one left out or set null (the schema declares illicit and illicit/violent nullable) leaves the image's judgment as it is, and a rule you pass for one of them is not read. An answer that leaves out the boolean or the score of one of the six, empty maps included, judged nothing: it counts as an answer of another shape (see Down), never as a clean image. Its reason codes are the six categories' codes and your rules' review codes.
• Defaults. Every image category flags at OpenAI's own boolean. Nothing holds and nothing blocks: OpenAI's scores refuse no image unless you set a holdAt or a blockAt in rules, which work per category as they do on the OpenAI guard. A flag carries the six categories' scores, a sub-category written with . (violence.graphic), since the image guard keeps a score only under a key of letters, digits and _ . : -.
• Down. It is built with onError: 'skip'. A network error, a non-2xx answer, a timeout (timeoutMs, default 3000, which bounds the whole check: the check stops waiting for an apiKey function at that point, nothing is sent, and the image's data URL is made only once the key is resolved) or an answer of another shape (an answer past 1 MiB included, as on the OpenAI guard) is written by the image guard as one image_guard.check_failed line, and your other classify checks decide the image. When none of them answered for it, the image guard refuses the image as unavailable (IMAGE_CHECK_UNAVAILABLE). The check writes no line of its own.
• Stopping. The image guard reads the call's vendorSignal before each classify check; the check reads it again once its key is resolved, and then sends nothing and counts as a check that did not answer. A request already sent is answered as usual.
• Deadlines. With @framers/agentos 0.17.1 the image guard's deadline does not reach a check: the guard answers at its own timeoutMs (default 15,000), or at the shorter deadline a required posture gives it, while the check runs until its own timeoutMs (default 3,000). So set the check's timeoutMs at or below the guard's. If it is longer, a request may be sent after the guard has answered, and its answer is discarded. The check's timeoutMs bounds the whole check, the key lookup included: nothing is sent after it. A core that hands the check a deadlineSignal, on the context or on the image, stops the check at that deadline as well: once it is aborted nothing is sent, and a request in flight is cancelled.

import { ImageGuardrail, moderateContent } from '@framers/agentos';
import { openAIModerationImageCheck } from '@framers/agentos-ext-ml-classifiers/moderation';

const images = ImageGuardrail.create({
  id: 'images',
  requireMatch: true,
  checks: [
    hashCheck, // your hash service: { id, kind: 'match', check }
    openAIModerationImageCheck({ apiKey: process.env.OPENAI_API_KEY! }),
    visionJudge, // your own vision model: { id, kind: 'classify', check }
  ],
  loadImage, // fetches an image_url input once, under your own egress rules
});
const decision = await moderateContent({ images: uploads }, { guards: [images], context: { userId, sessionId } });

The conduct classifier

createConductGuardrail(options) (and ConductClassifier, which it wraps) asks a judge model for one JSON object of confidences over ten categories (CONDUCT_CATEGORIES: minor_sexual, suspected_minor, threat, sexual, violence, hate, harassment, self_harm, scam_spam, personal_data), in the form {"c":{"<category>":<confidence>}}. The judge writes no reason and no decision; code decides.

• Policy. policy is { version, sharedPrefix, ratings, defaultRating }. sharedPrefix holds your category definitions and the rules of every rating. It goes in one system block after the library's output contract and UNTRUSTED_EVIDENCE_NOTICE, with a cache breakpoint at its end, so every call shares one cached prefix. The user turn holds Rating: <rating> and the message framed by frameUntrustedEvidence(). A call's rating is context.moderation.rating when the policy lists it, else defaultRating (one conduct.rating_unknown line per unknown rating word). DEFAULT_CONDUCT_POLICY is plain and generic, with one rating, standard.
• Thresholds. thresholds is keyed by rating, with '*' for every rating; each entry is { flagAt?, holdAt?, refuseAt? } per category. Yours are laid over the library's category by category, so a category you do not name keeps the library's numbers. A category you name takes your entry whole, all three numbers: { '*': { minor_sexual: { refuseAt: 0.9 } } } leaves minor_sexual with no hold, so name every number you mean to keep. The library's: a refusal at 0.8 on every category but suspected_minor and self_harm; a hold at 0.5 on minor_sexual, suspected_minor and threat; a flag at 0.5 on the rest; self_harm only flags. Refuse outranks hold, hold outranks flag, and among categories that meet the same threshold the one with the highest confidence decides.
• Verdicts. A refusal is a BLOCK CONDUCT_<CATEGORY>. A hold is a BLOCK with hold: { reviewCode: '<CATEGORY>' } and reason code CONDUCT_<CATEGORY>. A flag is a FLAG. metadata holds the categories found, the deciding one, the rating, the policy version, the route (primary, fallback or cache) and the number of attempts.
• Strict answers. After white space and one enclosing code fence are removed, the answer must be exactly the contract: one object, the key c alone, categories among the ten, confidences from 0 to 1. Anything else, a sentence before the object included, is a failed attempt and never read as nothing found. So is an answer that stopped at the output cap, whether or not its text parses.
• The chain. judge is tried 1 + retries times (retries default 1), then fallbackJudge once. Each attempt has attemptTimeoutMs (default 3000). With no usable answer the guard writes conduct.unavailable and throws ConductUnavailableError. It is closed by default (failClosed: true), so AgentOS's dispatcher answers the outage itself: a BLOCK GUARDRAIL_ERROR that names the guard (metadata.guardrailId), which moderateContent() reports as unavailable: true and ranks below another guard's refusal or hold. With failClosed: false the guard answers nothing. The guard's own deadline defaults to room for every attempt, (1 + retries + (fallbackJudge ? 1 : 0)) * attemptTimeoutMs + 500: with one retry, a second route and 1,800 ms per attempt, 5.9 seconds. The half second covers the cache read. Once the dispatcher stops waiting for the guard (the deadlineSignal AgentOS hands it), no further attempt starts, so a text is not sent to the second route for an answer nobody reads.
• stopVendorsOn. Once the call's vendorSignal is aborted (OpenAI's MINOR_SEXUAL_SIGNAL hold with stopVendorsOn: ['MINOR_SEXUAL_SIGNAL']), no further attempt starts. An attempt already sent is still read; with no usable answer the guard writes conduct.stopped and fails as above, and the hold that stopped it decides.
• Length. A message over maxMessageChars (default 8,000) is refused with CONDUCT_TOO_LONG and no model call: a cut text would leave a part nobody read. The guard applies the limit; ConductClassifier.classify(), called on its own, does not.
• Numbers. retries is a whole number, 0 or more. attemptTimeoutMs and timeoutMs, here and on the OpenAI guard, are milliseconds above 0 and at most 2,147,483,647, the longest a Node timer waits; a longer one runs after 1 ms (Node's timers). A deadline the library works out is held to the same limit: this guard's default timeoutMs (the sum above) and the OpenAI guard's dispatcher deadline (its timeoutMs plus half a second). maxMessageChars is above 0 (Infinity sets no limit). cacheTtlMs is above 0 and finite, and InMemoryConductRefusalCache's maxEntries is a whole number, 1 or more. Any other value, the NaN that Number(process.env.NAME) gives for a name that is not set included, throws a TypeError when the guard or the cache is built.

The refusal cache

cache stores refusals alone, never an allow, a flag or a hold, for cacheTtlMs (default ten minutes). The key is a SHA-256 of the policy version, the rating, that rating's thresholds and the text exactly as received: no case folding, no Unicode normalisation, no look-alike mapping, because two texts that differ by one letter can name two different things. A hit answers with route: 'cache' and no model call. The key is unsalted, a fingerprint a guessed text can be matched to, so a shared store holds message data for the entry's lifetime. InMemoryConductRefusalCache (10,000 entries by default, oldest out first) serves one process; implement ConductRefusalCache (get, set) over a shared store when you run more than one. A read that throws, or that has not answered within 250 ms, is skipped with one conduct.cache_failed line, and the judge is asked. A write is not waited for: the refusal is returned at once, and a write that fails writes the same line when it fails. The entry stored is a copy, so an edit of the verdict leaves it as it was.

The judge call

createAgentOSConductJudge({ provider, model, apiKey?, baseUrl?, budget?, effort?, thinkingHeadroomTokens? }) sends each attempt through generateText() with thinking off (thinking: false), effort 'low' by default, no temperature, top_p or top-k, one attempt and no fallback provider, so the classifier's chain is the only one. A provider other than openai needs a model. On Claude Haiku 5.5, "Thinking is on by default and counts toward max_tokens" and thinking can be turned off at low, medium and high effort (Prompting Claude Haiku 5.5); its migration guide says to omit temperature, top_p and top_k (migration guide). The answer's 200 tokens are the answer's alone: a route whose model always thinks, or another vendor's reasoning model, gets thinkingHeadroomTokens, added to the request's output cap, sized from your own dry run.

An invoker you write yourself is (request) => Promise<{ text, stop }>, where stop is 'cap' when the model stopped at the output cap and 'end' otherwise, and it throws ConductJudgeDeclinedError(category) when its vendor declined the request. It may throw ConductJudgeEmptyAnswerError when its vendor can turn a refusal into an empty answer (see below).

A declined request

Claude Haiku 5.5 runs safety classifiers that can decline a request with stop_reason: "refusal", and sending the same request again usually returns another refusal (Safeguard refusals). Other vendors decline by a content filter: OpenAI's chat answer ends with finish_reason content_filter "if content was omitted due to a flag from our content filters" (OpenAI's OpenAPI file, CreateChatCompletionResponse). createAgentOSConductJudge() turns each of these into ConductJudgeDeclinedError: an error whose typed fields say content_filter, content_policy_violation or safety_violations (its code or type, the OpenAI provider's openaiErrorCode or openaiErrorType, or the vendor's own error object in details.error, as AgentOS's provider keeps a 400's code there), and an answer AgentOS reports with finish reason content_filter, whatever part of it arrived. The category is one of cyber, bio, frontier_llm, general_harms or other, read from the error's typed fields and never from its message; a filter's stop is other. The guard holds the message for a person: a BLOCK CONDUCT_JUDGE_DECLINED with review code JUDGE_DECLINED, with failClosed true or false. It is never retried, never sent to the second route and never cached, and one conduct.judge_declined line is written. How often your benign traffic is declined is for your own dry run to measure.

OpenAI also refuses outright: a chat answer whose message has a refusal string and null content (ChatCompletionResponseMessage in OpenAI's OpenAPI file), or a Responses answer whose content holds a part of type refusal (Refusals with Structured Outputs). Both end with an ordinary stop. AgentOS 0.17.1 keeps neither: generateText() returns empty text with finish reason stop, which on its own would read as a failed parse, be retried, and then let a second route's clean answer allow the refused text. So createAgentOSConductJudge() does two things:

• Where AgentOS keeps the refusal as a typed signal (a refusal field on the result, or finish reason refusal), the judge throws ConductJudgeDeclinedError('other'), and the message is held as above, with no retry. A refusal field that is true or a string, the empty string included, is a decline at any finish reason (an answer the output cap cut included) and whatever text came beside it. A field that is absent, undefined, null or false is not.
• Where it does not, an empty answer at an ordinary stop on the openai route, or on a route with no provider named, is thrown as ConductJudgeEmptyAnswerError. It is retried on its route like a failed attempt (one conduct.attempt_failed line, cause empty). If the route's attempts end with no usable answer, the message is held as a decline is (CONDUCT_JUDGE_DECLINED, review code JUDGE_DECLINED, declined: 'other'), with failClosed true or false. It is never sent to the second route, so a refusal never ends in an allow. A model that truly answers nothing is held the same way: the safe side is a person's review.

Events

MODERATION_EVENTS names the seven lines the guards write through the logger you pass, for a host on AgentOS's allow-list logger to add to its map: moderation.signal_skipped, conduct.attempt_failed, conduct.judge_declined, conduct.unavailable, conduct.stopped, conduct.cache_failed and conduct.rating_unknown. Their fields are ids, fixed words and numbers. No line holds a message's text, a vendor's answer, an HTTP body or an error's message, and with no logger the guards write nothing.

Logging

When the ONNX tier cannot load, the guardrail writes one line, once per guardrail instance:

ml-classifiers: the ONNX tier is unavailable (Xenova/toxic-bert could not load: <error>); classifying with the keyword tier

It goes to the logger passed to createMLClassifierGuardrail, else the logger in the pack context or the activation context, else the console.

Lazy loading and optional install

This package is an optional dependency of @framers/agentos-extensions-registry. The registry ships catalog metadata; createCuratedManifest() calls import.meta.resolve() per entry and skips anything not installed. npm install @framers/agentos-ext-ml-classifiers is the gate.

Nothing loads at activation. The guardrail imports @huggingface/transformers and loads the model on its first classification.

The guardrail registers with config.canSanitize = false and config.evaluateStreamingChunks = false: it never rewrites text, runs in Phase 2 of the two-phase dispatcher (parallel classifiers), and on output it reads only the final response.

For the extension model, see How extensions stay optional and lazy and the auto-loading guide.

What the package holds

The published package holds dist (the built JavaScript, its type declarations and their source maps), src (the TypeScript the maps point to), this README and the licence. Tests, workflows, scripts and build configuration are not published.

License

Apache 2.0, see the repo root LICENSE.