Docs
Evaluation

Datasets

Versioned collections of test cases that power evals.

Open in ClaudeView Markdown

A dataset is a workspace-scoped, versioned collection of test cases — one JSON object per case — together with a schema describing their shape. Datasets are what evals run against: every case becomes one trial of your prompt. Like prompts, a dataset is addressed as @workspace/name and versioned with semver.

Cases and schema

A case is a single JSON object — think of it as one row of test data. A dataset version holds many cases, and carries a JSON Schema describing their common shape.

The schema is inferred from your data the first time you upload cases. From then on it is used to validate every case and to type-check the CEL expressions in an eval — the case binding an eval reads from is typed by this schema.

A few cases, as JSON Lines (one object per line):

text
{"question": "What is 2 + 2?", "expected": "4", "tags": ["math"]}
{"question": "Capital of France?", "expected": "Paris", "tags": ["geography"]}

…produce an inferred schema like:

text
{
  "type": "object",
  "properties": {
    "question": { "type": "string" },
    "expected": { "type": "string" },
    "tags": { "type": "array", "items": { "type": "string" } }
  }
}

An eval can then reference these fields in its input mapping and assertions — case.question, case.expected, case.tags.

Versions

Datasets use the same draft-then-publish model as prompts:

  • A draft version is mutable — you can re-upload cases and adjust the schema while you iterate.
  • Published versions are immutable semver snapshots (1.0.0, 2.3.1, …). Once published, the cases and schema are frozen.

Creating and populating a dataset

You can author datasets in the web app, from the sufleur CLI, or programmatically via the API. The flow is the same regardless of surface:

  1. Create a dataset — this also opens its initial draft version.
  2. Upload cases. Accepted formats are JSONL, NDJSON, JSON (an array of objects), and CSV (up to 50 MB). The schema is inferred on the first upload; CSV columns are coerced to the inferred types.
  3. Review the validation report — it flags any case that doesn't match the schema — then publish the version to a semver.

Authoring from the CLI

The sufleur dataset command group covers authoring up to the point of publishing, so an agent (or a CI job) can build and validate a draft without leaving the terminal. The local loop mirrors prompt authoring — dump → edit → push → validate:

text
sufleur dataset create @acme/extraction-cases --description "..."  # creates the dataset + initial draft (private)
sufleur dataset dump @acme/extraction-cases@draft --to ./ds        # schema.json + cases.jsonl + dataset.yaml
 
# edit ./ds/cases.jsonl and ./ds/schema.json locally, then push them back
sufleur dataset cases push  @acme/extraction-cases@draft --file ./ds/cases.jsonl  # schema inferred on first upload
sufleur dataset schema set  @acme/extraction-cases@draft --file ./ds/schema.json  # optional: refine the inferred schema
 
sufleur dataset version validate @acme/extraction-cases@draft      # exits non-zero on any case violation
# when validation is clean, publish the version in the web app

The most useful commands:

NameDescription
dataset create @ws/nameCreate a dataset and its initial draft (private). --description to annotate.
dataset cases push @ws/name@draft --file …Upload cases from a .jsonl, .json (array), or .csv file. Format is detected from the extension; --format overrides it (required for --file - stdin).
dataset schema set @ws/name@draft --file …Replace the draft's JSON Schema. Only needed to refine what the first upload inferred.
dataset version validate @ws/name@draftCheck every case against the schema. Exits non-zero on a violation — publishing is gated on this.
dataset version draft @ws/nameOpen the next draft, carrying forward the latest published schema and cases.
dataset dump / cases pullSnapshot a version to disk — dump writes schema + cases + metadata, cases pull writes just the cases as JSONL.

Ingesting cases programmatically

For automation, cases can be uploaded over REST as multipart form data. The endpoint lives under the same API base your API key targets:

text
curl -X POST \
  "$SUFLEUR_API_BASE/dataset/<dataset-name>/versions/<version>/cases" \
  -H "Authorization: Bearer $SUFLEUR_API_KEY" \
  -F "[email protected]"

The response reports the resulting case count, the (inferred or existing) schema, and a per-case validation summary. The same operations — creating a dataset, adding a version, setting the schema, and publishing — are also available as GraphQL mutations (createDataset, createDatasetVersion, setDatasetSchema, publishDatasetVersion).

Referencing a dataset from an eval

An eval pins exactly one dataset version through its dataset.ref:

text
dataset:
  ref: "@acme/[email protected]"

Once an eval points at a dataset, each case is fed to your prompt and checked by the eval's assertions. See Evals for the full definition — input mapping, judges, assertions, and how a run turns cases into a verdict.

PreviousMustache extensions