# Datasets

Versioned collections of test cases that power evals.

A **dataset** is a workspace-scoped, versioned collection of **test cases** — one JSON object per case — together with a schema describing their shape. Datasets are what [evals](/docs/evals) run against: every case becomes one trial of your prompt. Like prompts, a dataset is addressed as `@workspace/name` and versioned with [semver](https://semver.org/).

## Cases and schema

A **case** is a single JSON object — think of it as one row of test data. A dataset version holds many cases, and carries a [JSON Schema](https://json-schema.org/) describing their common shape.

The schema is **inferred from your data the first time you upload cases**. From then on it is used to validate every case and to type-check the CEL expressions in an eval — the `case` binding an eval reads from is typed by this schema.

A few cases, as JSON Lines (one object per line):

```jsonl
{"question": "What is 2 + 2?", "expected": "4", "tags": ["math"]}
{"question": "Capital of France?", "expected": "Paris", "tags": ["geography"]}
```

…produce an inferred schema like:

```json
{
  "type": "object",
  "properties": {
    "question": { "type": "string" },
    "expected": { "type": "string" },
    "tags": { "type": "array", "items": { "type": "string" } }
  }
}
```

An eval can then reference these fields in its input mapping and assertions — `case.question`, `case.expected`, `case.tags`.

## Versions

Datasets use the same draft-then-publish model as prompts:

- A **draft** version is mutable — you can re-upload cases and adjust the schema while you iterate.
- **Published** versions are immutable semver snapshots (`1.0.0`, `2.3.1`, …). Once published, the cases and schema are frozen.

> [!TIP] Why immutable versions matter
>
> An eval pins an exact dataset version. Because a published version can never change, a run
> against `@acme/cases@2.0.0` always sees the same cases — so an eval result is reproducible and
> comparable across runs.

## Creating and populating a dataset

You can author datasets in the **web app**, from the **`sufleur` CLI**, or programmatically via the **API**. The flow is the same regardless of surface:

1. Create a dataset — this also opens its initial draft version.
2. Upload cases. Accepted formats are **JSONL**, **NDJSON**, **JSON** (an array of objects), and **CSV** (up to 50 MB). The schema is inferred on the first upload; CSV columns are coerced to the inferred types.
3. Review the validation report — it flags any case that doesn't match the schema — then publish the version to a semver.

### Authoring from the CLI

The `sufleur dataset` command group covers authoring up to the point of publishing, so an agent (or a CI job) can build and validate a draft without leaving the terminal. The local loop mirrors prompt authoring — **dump → edit → push → validate**:

```bash
sufleur dataset create @acme/extraction-cases --description "..."  # creates the dataset + initial draft (private)
sufleur dataset dump @acme/extraction-cases@draft --to ./ds        # schema.json + cases.jsonl + dataset.yaml

# edit ./ds/cases.jsonl and ./ds/schema.json locally, then push them back
sufleur dataset cases push  @acme/extraction-cases@draft --file ./ds/cases.jsonl  # schema inferred on first upload
sufleur dataset schema set  @acme/extraction-cases@draft --file ./ds/schema.json  # optional: refine the inferred schema

sufleur dataset version validate @acme/extraction-cases@draft      # exits non-zero on any case violation
# when validation is clean, publish the version in the web app
```

The most useful commands:

| Name | Description |
| --- | --- |
| `dataset create @ws/name` | Create a dataset and its initial draft (private). --description to annotate. |
| `dataset cases push @ws/name@draft --file …` | Upload cases from a .jsonl, .json (array), or .csv file. Format is detected from the extension; --format overrides it (required for --file - stdin). |
| `dataset schema set @ws/name@draft --file …` | Replace the draft's JSON Schema. Only needed to refine what the first upload inferred. |
| `dataset version validate @ws/name@draft` | Check every case against the schema. Exits non-zero on a violation — publishing is gated on this. |
| `dataset version draft @ws/name` | Open the next draft, carrying forward the latest published schema and cases. |
| `dataset dump / cases pull` | Snapshot a version to disk — dump writes schema + cases + metadata, cases pull writes just the cases as JSONL. |

> [!NOTE] Publishing and visibility stay in the web app
>
> Just like prompts, the CLI takes a dataset draft right up to publish but does not publish it or
> change its visibility — those are deliberate human steps in the app. Run `sufleur dataset --help`
> for the full surface, or hand `sufleur skill` to your coding agent.

### Ingesting cases programmatically

For automation, cases can be uploaded over REST as multipart form data. The endpoint lives under the same API base your API key targets:

```bash
curl -X POST \
  "$SUFLEUR_API_BASE/dataset/<dataset-name>/versions/<version>/cases" \
  -H "Authorization: Bearer $SUFLEUR_API_KEY" \
  -F "file=@cases.jsonl"
```

The response reports the resulting case count, the (inferred or existing) schema, and a per-case validation summary. The same operations — creating a dataset, adding a version, setting the schema, and publishing — are also available as GraphQL mutations (`createDataset`, `createDatasetVersion`, `setDatasetSchema`, `publishDatasetVersion`).

## Referencing a dataset from an eval

An eval pins exactly one dataset version through its `dataset.ref`:

```yaml
dataset:
  ref: "@acme/extraction-cases@2.0.0"
```

> [!WARNING] Use the raw version — no v prefix
>
> The version in a ref is the bare semver — `2.0.0`, not `v2.0.0`. Use the literal `draft` to point
> at the current draft version while you are still iterating on the cases.

Once an eval points at a dataset, each case is fed to your prompt and checked by the eval's assertions. See [Evals](/docs/evals) for the full definition — input mapping, judges, assertions, and how a run turns cases into a verdict.