rag-pipeline-utils

RAG for Node.js that tells you when your LLM is making things up.

Your chatbot just told someone they get thirty vacation days. The handbook says twenty. This library catches that: it checks every sentence of an answer against the sources it was given, numbers, "not"s and conditions included, with no extra model calls. It also builds the pipeline for you, if you want one.

npm install @devilsdev/rag-pipeline-utils

Node.js 22+. CommonJS, ESM and TypeScript types out of the box.

New in 3.9

• Dropped conditions are caught. An answer that leaves out a condition its source sets, like "with manager approval", now fails the check.
• Much faster on long answers. A 1,000-sentence answer is checked in under half a second.
• Cleaner text from PDFs and web pages. It's normalised before it's compared.
• /edge without nodejs_compat (3.9.1). Cached pipelines and cost estimates on Cloudflare Workers no longer throw "process.emitWarning is not a function"; the warning goes to console.warn.

Release notes · Changelog

Upgrading from 3.8.1 or earlier? Upgrade to 3.8.2 or later for its security fixes; 3.9.0 is recommended. If you store documents with top-level access restrictions in Qdrant, Pinecone, Weaviate or pgvector, re-ingest them: see 3.8.2.

Catch the made-up bits

Bring an answer from anywhere: this library, LangChain, Mastra, the AI SDK or LlamaIndex. Pass it in with the sources it was given:

import { checkAnswer } from "@devilsdev/rag-pipeline-utils";

const check = await checkAnswer({
  answer:
    "Full-time employees receive thirty days of paid vacation per year. " +
    "Remote work is allowed two days per week.",
  sources: [
    {
      pageContent:
        "Full-time employees receive twenty days of paid vacation per year.",
    },
    {
      pageContent:
        "Remote work is allowed two days per week with manager approval.",
    },
  ],
});

console.log(check.grounded); // false
console.log(check.groundedness); // 0
console.log(check.unsupported); // ["Full-time employees receive thirty days of paid vacation per year.", "Remote work is allowed two days per week."]

Both sentences are caught: the first states the wrong number, and the second drops the "with manager approval" its source requires, which makes it claim more than the handbook says. That runs locally, with no API key and nothing sent anywhere. Worried about paraphrases? Add judge and your LLM gets asked about the sentences that failed on wording alone, and nothing else. Try it on StackBlitz with an answer of your own.

A pipeline in a dozen lines

No API key needed for this one either: a built-in TF-IDF embedder and an in-memory store. Swap in OpenAI, Anthropic, Ollama or any AI SDK model when you're ready.

import {
  createRagPipeline,
  LocalEmbedder,
  MemoryRetriever,
} from "@devilsdev/rag-pipeline-utils";

const docs = [
  { id: "vacation", content: "Full-time employees get twenty vacation days." },
  { id: "remote", content: "Remote work is allowed two days per week." },
];

const embedder = new LocalEmbedder();
embedder.train(docs.map((doc) => doc.content));
const retriever = new MemoryRetriever();
await retriever.store(
  docs.map((doc) => ({ ...doc, vector: embedder.embed(doc.content) })),
);

const pipeline = createRagPipeline({ embedder, retriever });
const result = await pipeline.run({ query: "How many vacation days?" });

console.log(result.results[0].id); // "vacation"

Rather start from a working app? npm create rag-pipeline@latest scaffolds one for Next.js with the AI SDK, Mastra, or Cloudflare Workers.

Why people reach for it

• Grounding you can see. Per-sentence citations and a groundedness score, with numbers, units, dates and negation actually checked. verifyCitations() double-checks the citations Claude and OpenAI hand back.
• Regressions fail the build. rag-pipeline eval scores a test set in CI and goes red when answers get worse. rag-eval-action posts the results on your pull request.
• Plays well with your stack. Grounding middleware for any Vercel AI SDK model, Mastra scorers and guardrails, Jest and Vitest matchers.
• Guardrails with receipts. Prompt-injection detection and PII redaction with published precision and recall, plus ACL filtering before anything reaches the model.
• Ready for production. Hybrid search, reranking, token budgets, timeouts and cancellation, retries, caching, and OpenTelemetry (or Langfuse) traces with the scores attached.
• Connectors included. OpenAI, Anthropic, Cohere, Ollama, pgvector, Qdrant, Pinecone and Weaviate. Anything with an embed() or generate() method works too.
• A real CLI. init, ingest, query, eval and serve, against a local index, offline if you like.
• Runs at the edge. A /edge build for Cloudflare Workers, Vercel Edge and Deno.
• Talk to it from Claude. @devilsdev/rag-mcp-server lets Claude search a folder of your docs and check its own answers.

Take only what you need

The main package has everything. The smaller ones are there if install size matters:

Go further

• Use cases with runnable code, most of them one click away on StackBlitz
• Docs: guides, the API reference and the CLI
• How it compares with LangChain.js, Mastra and the AI SDK
• Examples and the changelog

Found a bug or have an idea? Open an issue. A star on GitHub helps other devs find it.

Published from GitHub Actions with npm provenance. Security reports: see SECURITY.md. MIT licensed.