Kappa

A measure of how far two raters agree, after removing the agreement expected by chance: 1 means perfect agreement and 0 means no better than chance. It was used to measure how often psychiatrists agree on a diagnosis.

In 1974 Spitzer and Fleiss recalculated earlier studies of diagnostic agreement on this scale. Agreement was satisfactory only for mental deficiency, organic brain syndrome and alcoholism; it was fair for psychosis and schizophrenia and poor for everything else. They also warned that a reliable classification is not guaranteed to be valid, but an unreliable one cannot be^[1].

In the DSM-III field trials reported in 1979 (281 adults), kappa for major diagnoses overall was 0.78 when two clinicians judged the same interview together and 0.66 when they interviewed separately^[2]. In the DSM-5 field trials of 2013, where two clinicians interviewed each patient on separate occasions, it was 0.46 for schizophrenia, 0.28 for major depressive disorder and 0.20 for generalized anxiety disorder. The trials were designed differently, so the numbers cannot be compared directly^[3].