Agreement and Label Quality Explorer

Model Evaluation

Agreement and Label Quality Explorer

Measuring annotation reliability beyond raw accuracy.

What This Shows

Explore how inter-rater agreement changes under different labeling patterns. The applet highlights a frequent ML bottleneck: model quality is bounded by the reliability of the data labels, and agreement metrics can reveal problems that raw counts hide.

Label quality Cohen's kappa Evaluation Annotation workflows Data quality

Technical Challenge

Quantifying whether labels are consistent enough to support reliable model training and evaluation.

My Contribution

Built a simulator for interpreting agreement metrics under changing base rates and disagreement patterns.

Industry Transfer

Human-in-the-loop ML, annotation QA, medical review, content moderation, audit workflows.

Connected Work

Human validation in TOROS and applied ML data quality work.