Sonja Mensch

50 Chapter 2 Appendix A. (continued) Box B. Reliability: relative measures (including test-retest reliability, inter-rater reliability and intra-rater reliability) Statistical methods Excellent Good Fair Poor 11 for continuous scores: Was an intraclass correlation coefficient (ICC) calculated? ICC calculated and model or formula of the ICC is described ICC calculated but model or formula of the ICC not described or not optimal. Pearson or Spearman correlation coefficient calculated with evidence provided that no systematic change has occurred Pearson or Spearman correlation coefficient calculatedWITHOUT evidence provided that no systematic change has occurred or WITH evidence that systematic change has occurred No ICC or Pearson or Spearman correlations calculated 12 for dichotomous/nominal/ordinal scores: Was kappa calculated? Kappa calculated Only percentage agreement calculated 13 for ordinal scores: Was a weighted kappa calculated? Weighted Kappa calculated Unweighted Kappa calculated Only percentage agreement calculated 14 for ordinal scores: Was the weighting scheme described? e.g. linear, quadratic Weighting scheme described Weighting scheme NOT described