@arizeai/凤凰卫视

此软件包提供了一个 TypeScript 评价库. 它是供应商不可知的,可在任何框架或平台之外单独使用。 此套件仍在积极开发中,可能会有变化.

安装

# or yarn, pnpm, bun, etc...
npm install @arizeai/phoenix-evals

此软件包基于 AI SDK 软件 v7,仅限无害环境管理。 需要22.12节点或更新节点(共同JS的构建依赖于 require() (b) 无害环境管理。 使用AI SDK v7相容的模型提供者包(例如. @ai-sdk/openai (v4+) (中文(简体) ).

使用量

创建分类符

图书馆提供 createClassifier 函数,它允许您为不同的任务创建自定义评价器,如幻觉检测、相关性评分或任何二进制/多类分类.

import { createClassifier } from "@arizeai/phoenix-evals/llm";
import { openai } from "@ai-sdk/openai";

const model = openai("gpt-4o-mini");

const promptTemplate = `
In this task, you will be presented with a query, a reference text and an answer. The answer is
generated to the question based on the reference text. The answer may contain false information. You
must use the reference text to determine if the answer to the question contains false information,
if the answer is a hallucination of facts. Your objective is to determine whether the answer text
contains factual information and is not a hallucination. A 'hallucination' refers to
an answer that is not based on the reference text or assumes information that is not available in
the reference text. Your response should be a single word: either "factual" or "hallucinated", and
it should not include any other text or characters.

    [BEGIN DATA]
    ************
    [Query]: {{input}}
    ************
    [Reference text]: {{reference}}
    ************
    [Answer]: {{output}}
    ************
    [END DATA]

Is the answer above factual or hallucinated based on the query and reference text?
`;

// Create the classifier
const evaluator = await createClassifier({
  model,
  choices: { factual: 1, hallucinated: 0 },
  promptTemplate: promptTemplate,
});

// Use the classifier
const result = await evaluator({
  output: "Arize is not open source.",
  input: "Is Arize Phoenix Open Source?",
  reference:
    "Arize Phoenix is a platform for building and deploying AI applications. It is open source.",
});

console.log(result);
// Output: { label: "hallucinated", score: 0 }

见完整的例子 examples/classifier_example.ts。 。 。 .

建设前评价员

该图书馆包括若干负责共同评价任务的预建评价员。 这些评审人员带有最优化的提示并可以直接使用任何AI SDK模型.

所有预建评价员均可从 @arizeai/phoenix-evals/llm 模块 :

import {
  createFaithfulnessEvaluator,
  createConcisenessEvaluator,
  createCorrectnessEvaluator,
  createRetrievalRelevanceEvaluator,
  createRefusalEvaluator,
} from "@arizeai/phoenix-evals/llm";
import { openai } from "@ai-sdk/openai";

const model = openai("gpt-4o-mini");

// Faithfulness: checks if the output is grounded in the context
const faithfulnessEvaluator = createFaithfulnessEvaluator({ model });
const faithfulnessResult = await faithfulnessEvaluator.evaluate({
  input: "What is the capital of France?",
  context: "France is a country in Europe. Paris is its capital city.",
  output: "The capital of France is London.",
});
console.log(faithfulnessResult);
// Output: { label: "unfaithful", score: 0, explanation: "..." }

// Correctness: checks if the output is factually correct
const correctnessEvaluator = createCorrectnessEvaluator({ model });
const correctnessResult = await correctnessEvaluator.evaluate({
  input: "What is the capital of France?",
  output: "Paris is the capital of France.",
});
console.log(correctnessResult);
// Output: { label: "correct", score: 1, explanation: "..." }

// Retrieval Relevance: checks if retrieved information is relevant to the request
const relevanceEvaluator = createRetrievalRelevanceEvaluator({ model });
const relevanceResult = await relevanceEvaluator.evaluate({
  input: "What is the capital of France?",
  context: "Paris is the capital of France and a major European city.",
});
console.log(relevanceResult);
// Output: { label: "relevant", score: 1, explanation: "..." }

代码评价员

该图书馆还包括内置的、确定性(非LLM)编码评价员,用于共同的分类指标:精度、召回和F-β(包括F1)。 这些资料可从秘书处索取 @arizeai/phoenix-evals/code 模块和对一组预期的和预测的标签的工作,支持两个二进制分类(通过 positiveLabel)和多类分类(通过 macro编号 :micro编号 :weighted (平均).

• 精确度 ——所有被预测为一个类的事物中,其实是该类的分数(分数)TP / (TP + FP)) (中文(简体) ). 更低的精度意味着更多的假警报.
• 回顾 ——在实际属于某类的所有东西中,模型发现的分数(TP / (TP + FN)) (中文(简体) ). 低调召回意味着更多的失手.
• F - 贝塔 ——精确和回想的加权口琴表示. beta = 1 (F1,默认值)将它们相等地加权; beta > 1 重量的回想更多(如果缺少真实阳性则成本更高,例如医疗检查); beta < 1 加权精度更高(当假警报成本更高时使用,例如垃圾邮件过滤).

import {
  createPrecisionEvaluator,
  createRecallEvaluator,
  createF1Evaluator,
  createPrecisionRecallFScoreEvaluators,
} from "@arizeai/phoenix-evals/code";

const f1 = createF1Evaluator();
const result = await f1.evaluate({
  expected: ["cat", "dog", "cat", "bird"],
  output: ["cat", "cat", "cat", "bird"],
});
console.log(result);
// Output: { score: 0.6 }

// Or create matching precision/recall/F-score evaluators at once
const { precision, recall, fScore } = createPrecisionRecallFScoreEvaluators({
  average: "weighted",
});

使用 createFBetaEvaluator({ beta }) 对于F1以外的F分数(如:. beta: 2 重量比精确度高. createPrecisionRecallFScoreEvaluators 返回三个独立的评价员({ precision, recall, fScore }) 共享一个选项对象——每个选项都产生一个分数,匹配单个工厂.

对于多类数据, average 控制每个等级的分数如何合并为一个数字: "macro" (默认)对每一类进行同等的权重——有利于浏览稀有类是否被忽略; "weighted" 衡量每一类的发生频率——当总体业绩比各类的均等重要时,情况良好; "micro" 将每个真实/虚假的正负和假负都集中到第一类——因为单标签多类问题等于总体准确性.

见 examples/classification_metrics_example.ts 包括二进制分类 F-β取舍 以及所有三种平均策略 关于基本公式、精确度/回顾取舍和引用,见 精度/ 召回 / F- 分数文件。 。 。 .

说明: 与上述每行LLM评价员不同,这些分类计量评价员是分批/数据集级: expected编号 :output 是每个例子的标签的完整序列,而不是一行的标签。 别把他们直接绑起来 runExperiment 作为每行评价员——而是首先收集每一行的预期/预测标签,然后打电话 .evaluate({ expected, output }) 一次在全阵列上.

数据映射

当您的数据结构不符合评估者的期望时, 请使用 bindEvaluator 将您的字段映射到评估者的预期输入格式:

import { bindEvaluator } from "@arizeai/phoenix-evals";
import { createFaithfulnessEvaluator } from "@arizeai/phoenix-evals/llm";
import { openai } from "@ai-sdk/openai";

const model = openai("gpt-4o-mini");

type ExampleType = {
  question: string;
  context: string;
  answer: string;
};

const evaluator = bindEvaluator<ExampleType>(
  createFaithfulnessEvaluator({ model }),
  {
    inputMapping: {
      input: "question", // Map "input" from "question"
      context: "context", // Map "context" from "context"
      output: "answer", // Map "output" from "answer"
    },
  }
);

const result = await evaluator.evaluate({
  question: "Is Arize Phoenix Open Source?",
  context:
    "Arize Phoenix is a platform for building and deploying AI applications. It is open source.",
  answer: "Arize is not open source.",
});

映射支持简单属性( N)"fieldName"),点符号("user.profile.name"),数组访问("items[0].id", JSONPath 表达式 ("$.items[*].id"),和函数提取器((data) => data.customField) (中文(简体) ).

见完整的例子 examples/bind_evaluator_example.ts。 。 。 .

与凤凰的实验

此软件包与 @arizeai/phoenix-client 使实验工作流程成为可能。 您可以创建数据集,运行实验,以及微量评价呼吁进行分析和调试.

运行中的实验

为了运行你的评估,安装凤凰客户端

npm install @arizeai/phoenix-client

import { createFaithfulnessEvaluator } from "@arizeai/phoenix-evals/llm";
import { openai } from "@ai-sdk/openai";
import { createDataset } from "@arizeai/phoenix-client/datasets";
import {
  asExperimentEvaluator,
  runExperiment,
} from "@arizeai/phoenix-client/experiments";

// Create your evaluator
const faithfulnessEvaluator = createFaithfulnessEvaluator({
  model: openai("gpt-4o-mini"),
});

// Create a dataset for your experiment
const dataset = await createDataset({
  name: "faithfulness-eval",
  description: "Evaluate the faithfulness of the model",
  examples: [
    {
      input: {
        question: "Is Phoenix Open-Source?",
        context: "Phoenix is Open-Source.",
      },
    },
    // ... more examples
  ],
});

// Define your experimental task
const task = async (example) => {
  // Your AI system's response to the question
  return "Phoenix is not Open-Source";
};

// Create a custom evaluator to validate results
const faithfulnessCheck = asExperimentEvaluator({
  name: "faithfulness",
  kind: "LLM",
  evaluate: async ({ input, output }) => {
    // Use the faithfulness evaluator from phoenix-evals
    const result = await faithfulnessEvaluator({
      input: input.question,
      context: input.context,
      output: output,
    });

    return result; // Return the evaluation result
  },
});

// Run the experiment with automatic tracing
runExperiment({
  experimentName: "faithfulness-eval",
  experimentDescription: "Evaluate the faithfulness of the model",
  dataset: dataset,
  task,
  evaluators: [faithfulnessCheck],
});

实例

要运行示例,请使用 pnpm 运行 :

pnpm install
pnpx tsx examples/classifier_example.ts
# change the file name to run other examples

社区

加入我们的社区,与成千上万的人工智能建设者建立联系:

• 加入我们的行列 黑社会。 。 。 .
• 读取 凤凰网文献。 。 。 .
• * 提问并提供反馈 #凤凰支持 # 频道.
• 留下一颗星星在我们身上 GitHub 图像。 。 。 .
• 报告错误 GitHub 问题。 。 。 .
• X 跟着我们 页:1。 。 。 .
• 跟着我们 链接。 。 。 .
• 看看我们 路线图 看看我们下一步要去哪儿.