Judge prompt for error-explanation evals: given an error (with stack and code context) and the model's explanation, likely cause, and fixes, scores overall usefulness — is the diagnosis correct and are the fixes actionable — as a 0-1 score.
The LLM judge behind the error-explanation eval. Given the original error (plus any stack and code context) and the model's explanation, likely cause, and proposed fixes, it scores usefulness — is the cause correct for this error, and would the top fix actually resolve it — returning { reasoning, score } with score in [0, 1].
Grades correctness and actionability, not writing style. The deterministic half of the eval (the expected cause keyword must appear; at least one fix must be offered) is pure CEL; this judge covers whether the diagnosis is actually right.
Referenced as @sufleur/[email protected]; the eval asserts judge.usefulness.score > 0.7.