AI Review Agreement Calculator

How to use AI Review Agreement Calculator

A model evaluation can be unstable when human reviewers disagree about the expected answer. Reconcile independent labels before treating a golden dataset as settled.

1. Freeze one rubric and obtain independent labels from two distinct reviewer IDs.
2. List all expected fixtures, then enter labels without replacing unresolved differences.
3. Adjudicate disagreements and missing labels with an accountable expert before using the dataset for acceptance.

Good to know

• Only complete two-reviewer pairs contribute to agreement and category marginals. Two to twenty exact categorical labels; mixed rubrics and duplicate reviewer-fixture records are refused.
• Kappa is undefined with no pairs or when expected agreement equals one. No universal quality threshold, correctness judgment, statistical confidence interval or model inference is supplied.
• Each input field is limited to 8,000 UTF-16 characters. CSV needs the exact displayed header and at most 100 data rows. Quoted commas, doubled quotes and quoted newlines are supported. Duplicate identities are refused unless the tool explicitly reports them as findings.
• Markdown and CSV are editable review artifacts, not certificates. Spreadsheet-like text is apostrophe-prefixed in CSV to reduce formula execution risk; check the actual importer before sharing. Decimal ratios are rounded to at most 12 places.
• Runs only on the records you enter, locally in your browser. No AI call, provider connection, account inspection, upload, automatic outreach or saved input. Use neutral identifiers rather than personal or secret content.