AI evals

The ai-evals block stores the test cases for a prompt, runs them against a prompt version and a model, and compares two runs so you see which cases got worse. It runs no model itself: you pass a task function that produces the answer, so any AI SDK, gateway or provider works.

better-supabase add ai-evals

Inputs and outputs can hold user content, so only the service role reads or writes the tables. Create the block with a service transport. prompt_name and prompt_version match the prompt_versions table of the prompts block, but the block keeps them as plain values and does not require it.

Build a datasetCopy Anchor Link

const dataset = await evals.datasets
  .create("support.reply", { organizationId })
  .orThrow();

await evals.items
  .add(dataset.id, [
    {
      input: { question: "Where is my order?" },
      expected: "Track it under Orders.",
    },
  ])
  .orThrow();

With the ai-chat block installed, items.addFromFeedback(datasetId) copies rated replies into the dataset. The user message before the reply is the input, the rating, reason and comment go to the item metadata, and a thumbs up keeps the reply as the expected output. It reads thumbs down by default (rating: -1), skips replies already in the dataset and only takes the dataset tenant's chats. Without ai-chat it fails with EVAL_FEEDBACK_UNAVAILABLE.

RunCopy Anchor Link

import {
  createEvalJob,
  exactMatch,
  jsonSchema,
  llmJudge,
} from "better-supabase/blocks/ai-evals";

const run = await evals.runs
  .create(dataset.id, {
    promptName: "support.reply",
    promptVersion: 3,
    model: "openai/gpt-4.1-mini",
    scorers: [
      exactMatch(),
      llmJudge(judge, { criteria: "Polite and correct" }),
    ],
  })
  .orThrow();

export const evalJob = createEvalJob({
  evals,
  scorers: [exactMatch(), llmJudge(judge, { criteria: "Polite and correct" })],
  task: async (item, run) =>
    callModel(run.model, run.promptVersion, item.input),
  budgetMs: 45_000,
});

await evalJob({ runId: run.id });

The job handler takes { runId }, so any queue can call it. It scores the items that have no result yet and closes the run when none are left. A task that throws becomes that item's error and does not stop the run. With a budgetMs it returns done: false when time runs out, and you enqueue the same payload again to continue.

ScorersCopy Anchor Link

The judge is a port: { apiVersion: 1, judge(input) } returns { score, reason? } between 0 and 1. The block never calls a model for you. A judge that throws gives a failed score with the message as its reason.

Compare runsCopy Anchor Link

const diff = await evals.runs.compare(baseRun.id, candidateRun.id).orThrow();

if (diff.regressions.length > 0) {
  throw new Error(`${diff.regressions.length} cases got worse`);
}

compare needs two runs of one dataset. It returns both summaries, the change in pass rate, the items that regressed or improved with both scores and outputs, and counts of unchanged and missing items.

Export to promptfooCopy Anchor Link

evals.exportPromptfoo(datasetId, { prompt, model, scorers }) returns a promptfoo config as plain JSON: one test per item with the input as vars and one assertion per scorer (equals, is-json and llm-rubric). Write it to promptfooconfig.json to run the same cases in promptfoo.

FunctionsCopy Anchor Link

Without the clientCopy Anchor Link

import "server-only";

import { createAiEvals, rpcTransport } from "better-supabase/blocks/ai-evals";

export const evals = createAiEvals({
  transport: rpcTransport(bsb.admin().$client, { schema: "api" }),
  schema: "api",
});