PhD candidate: Debby ten Hove, MSc
Interrater reliability (IRR), which involves the degree to which ratings are independent of raters, is imperative in social and behavioral research. It bounds the validity of measures, and serves as an indicator for measurement precision and loss of statistical power in subsequent analyses. Multiple conceptualizations and associated coefficients are available to assess the IRR. General guidelines on selecting such a coefficient are based on data characteristics, and the interpretation of the estimated IRR is typically based on arbitrary benchmarks.
We argue that choosing and interpreting an IRR coefficient should be guided by the use of a measure in the primary analyses. Moreover, existing IRR coefficients ignore the nested structure of (inter)dependent data, which may result in biased estimates, and be uninformative concerning the IRR at different components of (inter)dependent data. This PhD project aims to assess which IRR coefficients are useful in a research setting, investigate what these measures imply for factors such as measurement precision and statistical power, and develop and test IRR coefficients for (inter)dependent data.