Skip to workstation

Judgewright

calibration instrument · in-browser worker

Calibrate your LLM judge against human raters — chance-corrected agreement, confusion heatmaps, and failure inspection with zero server egress.

100% in-browser
Chance-corrected κ
No account required

Drop your verdict file (.jsonl / .csv)

Parsed locally in Web Worker · zero server egress

Required columns: example_id, judge_label, human_label
one row per example
optional instructions for diagnosis
Paste or load data above to begin calibration

Why Judgewright

Judging a judge is usually done in a spreadsheet that stops at raw accuracy, or on a platform that wants your evaluation set uploaded. Judgewright keeps the arithmetic — chance-corrected agreement, the deflation against raw agreement, reproducible error bars — inside your browser, off the main thread, with no account and no upload.

CapabilityJudgewrightSpreadsheetHand-rolled scriptEval platform
Privacy & ads100% in-browser · zero adsStays local, no statisticsLocal scripts, drift-proneUploads your eval set
Chance correctionκ and α with the agreement-to-κ deflationRaw accuracy onlyWhatever you remembered to implementAccuracy dashboards
UncertaintySeeded bootstrap CIs, byte-reproducibleNoneOnly if you wrote it yourselfAggregate significance claims
UI stays responsiveOff-thread worker · watchdog · honest fallbackn/aBlocks the tab on large setsServer-bound round trips
Reproducibility receiptEvery number carries oneNot availableNot availableNot available
Account requiredNot requiredNot requiredNot requiredRequired

Reading a verdict set, and reading a reliability number

The input shapes Judgewright accepts, the column aliases it maps automatically, and the arithmetic behind chance-corrected agreement — written so a skeptic can recompute every figure from the paste alone.

The verdict row Judgewright reads

One JSON object per example, one example per line. Only three fields are required: an id, what the judge said, and what the humans said. Everything else — critiques, scores, extra raters — is optional and only unlocks more of the report.

{"example_id":"ex-001",
 "judge_label":"pass",          // what your judge emitted
 "human_label":"pass",          // what the human(s) said
 "judge_critique":"optional",   // shown side-by-side in the table
 "human_critique":"optional",
 "judge_score":0.93,            // optional, in [0,1]
 "human_labels":["pass","pass"] // optional: ≥2 raters unlock the human ceiling

Frequently answered questions

What Judgewright measures, how it keeps your verdicts in the browser, why raw agreement is not the headline, and how to reproduce every number it prints.

01What does Judgewright actually measure?
It measures a judge you already have labels for. You paste verdicts alongside human labels, and Judgewright reports how much of the agreement between them survives a chance correction, with error bars and a reproducibility receipt. It never runs a judge itself, never ranks an answer, and never scores a response.
02Do my verdicts leave my browser?
No. Parsing, every statistic and each export are computed inside an in-browser Web Worker. There is no upload, no telemetry on your payload, and no account requirement. The default path makes zero network requests — the Playwright request guard in this repository fails CI if that ever changes.
03Why is agreement not the headline?
Raw agreement counts ties that a coin flip would also have produced once you account for the label distribution, so a judge can look excellent against a skewed label set while carrying almost no information. Judgewright prints raw agreement, the chance-corrected coefficient, and the deflation between them side by side, so the gap cannot be missed.
04Can I reproduce the numbers myself?
Yes. Statistics are deterministic: the same rows, the same seed and the same replicate count produce byte-identical results. The receipt prints the exact seed, the replicate count that was actually used, the budget it ran under, and which engine produced the number.
05What happens on a very large verdict set?
The whole pipeline runs off the main thread behind an adaptive debounce, a cooperative deadline and a hard watchdog, so the interface stays responsive. Rows above the cap are capped before analysis while the true total is still counted and shown, and any budget that trips is disclosed rather than silently absorbed.
06Does it work without an account?
Completely. Sign-in, when a project is configured, only unlocks conveniences such as cross-device sync. Every deterministic feature — parsing, statistics, tables, exports, copying — works with no account and no backend at all.