Turn a natural-language description into a regex in a requested flavor (pcre/javascript/python/re2), with reasoning, a plain-string pattern, a human explanation, and self-supplied test cases (input + should_match). The self-supplied tests make outputs verifiable — and the eval runs the generated regex against the case's own strings via CEL .matches(), no judge needed.
Turn a natural-language description into a regex, in the flavor you ask for:
{
"reasoning": "A US ZIP is five digits, optionally followed by a hyphen and four more; anchor the whole string.",
"regex": "^\\d{5}(-\\d{4})?$",
"flavor": "re2",
"explanation": "Matches a 5-digit ZIP or ZIP+4. Rejects anything with letters or the wrong digit count.",
"test_cases": [
{ "input": "90210", "should_match": true },
{ "input": "90210-1234", "should_match": true },
{ "input": "9021", "should_match": false }
]
}
This is the collection's showcase of a genuinely executable, judge-free eval. Each dataset case carries its own positive and negative example strings; the eval runs the model's generated regex against them with CEL's RE2 .matches():
matches_positive_example → case.must_match_primary.matches(output.regex)rejects_negative_example → !case.must_not_match_primary.matches(output.regex)plus output.flavor == case.flavor. No LLM judge — the pattern either matches the right strings or it doesn't.
CEL uses RE2, so the dataset sticks to
re2/javascript-flavor cases whose patterns are RE2-expressible. For PCRE-only features (lookbehind, backreferences), verify with a judge instead — RE2 can't execute them.
Gated by @sufleur/regex-cases. Passing threshold 0.85. Recommended + eval model: claude-sonnet-4-6.