6 February 2025, 13.00 Stockholm
Addressing measurement error in social science data
Speaker: Patricía Martinková, Charles University, Prague
Abstract: Measurement error is omnipresent in social science data. Assessing the sources and impact of the error is important for designing policies to increase measurement reliability, and for developing high-quality ratings. In this work, we discuss several statistical aspects addressing the measurement error in social science data. We introduce a flexible method for assessing heterogeneity in measurement error and reliability with variance component models. We also discuss the relationship between the reliability and the false positive rate and address the issue of zero estimates. Methods are demonstrated with real-data examples from teacher hiring and grant proposal peer-review.
References:
- Martinková, P., & Hladká, A. (2023). Computational Aspects of Psychometric Methods: With R. Chapman and Hall/CRC. https://doi.org/10.1201/9781003054313
- Martinková P., Bartoš F., & Brabec M. (2023). Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables. Journal of Educational and Behavioral Statistics, 48(3), 349–383. https://doi.org/10.3102/10769986221150517
- Bartoš, F., & Martinková P. (2024). Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability. British Journal of Mathematical and Statistical Psychology. https://doi.org/10.1111/bmsp.12343
- Erosheva E, Martinková P, & Lee CJ (2021). When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. Journal of the Royal Statistical Society — Series A, 184(3), 904-919. https://doi.org/10.1111/rssa.12681
Venue: SAM.A.317