Evidence-Centered Design method for measuring complex psychological constructs

The article describes validation process of the diagnostic tool for monitoring assessment of socio-emotional habits in elementary school. The tool is based on the Big Five model and includes three scales: Goal achievement, Cooperation and Emotional control. Two validation researches have been conducted, both on third-grade students from Moscow and Tatarstan elementary schools. Preliminary research (N=1318) examined the construct validity and reliability of the questionnaire. Confirmatory and exploratory factor analysis and methods of Modern Test Theory have confirmed hypothecated factorial structure. Subsequent research (N=2559) proved criterion validity of the questionnaire: it showed expected level and direction of correlation with the results of reading and mathematical skills testing. Presented questionnaire can be used for group monitoring assessment of socio-emotional development in elementary school.

1. Boyatsis R. Kompetentnyi menedzher. Model' effektivnoi raboty [Competent manager. Model of effective work]. Moscow: KhIPPO, 2008. 340 p. (In Russ.).
2. Avdeeva S.M. et al. Otsenka informatsionno-kommunikatsionnoi kompetentnosti uchashchikhsya: podkhody, instrument, validnost' i nadezhnost' rezul'tatov [Assessment of information and communication competence of students: approaches, tools, validity and reliability of results] [Elektronnyi resurs]. Voprosy obrazovaniya [Education issues], 2017, no. 4, pp. 104–132. Available at: https://cyberleninka.ru/article/n/otsenka-informatsionno-kommunikatsionnoy-kompetentnosti-uchaschihsya-podhody-instrument-validnost-i-nadezhnost-rezultatov (Accessed 04.07.2018). (In Russ.; Abstr. in Engl.).
3. Khlopotov M.V. Primenenie baiesovskoi seti pri postroenii modelei dlya otsenki urovnya sformirovannosti kompetentsii [Bayesian network application in building models for assessing the level of competence formation] [Elektronnyi resurs]. Internet-zhurnal Naukovedenie [Internet Journal of Science], 2014, no. 5(24), pp. 1–28. Available at: https://cyberleninka.ru/article/n/primenenie-bayesovskoy-seti-pri-postroenii-modeley-dlya-otsenki-urovnya-sformirovannosti-kompetentsiy (Accessed 26.08.2018). (In Russ.).
4. Almond R., Steinberg L., Mislevy R. Enhancing the design and delivery of assessment systems: A four-process architecture. The Journal of Technology, Learning and Assessment, 2002, vol. 1, no. 5, pp. 1–64. Available at: https://ejournals.bc.edu/ojs/index.php/jtla/article/view/1671/1509 (Accessed 26.08.2018).
5. Behrens J.T. et al. An evidence centered design for learning and assessment in the digital world [Elektronnyi resurs]: CRESST Report 778. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST), 2010. 48 p. Available at: https://files.eric.ed.gov/fulltext/ED520431.pdf (Accessed 26.08.2018).
6. Almond R.G. et al. Bayesian networks in educational assessment. New York: Springer,  2015. 662 p. doi:10.1007/978-1-4939-2125-6
7. de Klerk S., Veldkamp B.P., Eggen T.J.H.M., Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a Bayesian network example. Computers & education, 2015, vol. 85, pp. 23–34. doi:10.1016/j.compedu.2014.12.020
8. Almond R.G. et al. Debugging the evidence chain [Elektronnyi resurs]. In Almond R.G., Mengshoel O. (eds.) Proceedings of the 2013 uai application workshops: Big data meet complex models and models for spatial, temporal and network data (uai2013aw). Seattle, WA: Association For Uncertainty In Artificial Intelligence, 2013. pp. 1–10. Available at: http://ceur-ws.org/Vol-1024/paper-01.pdf (Accessed 26.08.2018).
9. Ennis R.H. Critical thinking assessment. Theory into practice, 1993, vol. 32, no. 3, pp. 179–186. doi:10.1080/00405849309543594
10. Facione P. Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction [Elektronnyi resurs]. Newark: California Academic Press, 1990. 112 p.. Available at: https://philarchive.org/archive/FACCTA (Accessed 26.08.2018).
11. Griffin P., Care E., McGaw B. The changing role of education and schools // Assessment and teaching of 21st century skills. Dordrecht: Springer, 2012. pp. 1–15. doi:10.1007/978-94-007-2324-5_1
12. Duckworth A.L. et al. Grit: perseverance and passion for long-term goals. Journal of personality and social psychology, 2007, vol. 92, no. 6, pp. 1087–1101. doi:10.1037/0022-3514.92.6.1087
13. Steinberg L.S. et al. Introduction to the biomass project: An illustration of evidence-centered assessment design and delivery capability [Elektronnyi resurs]: CSE Report 609. Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing. Los Angeles, CA, 2003. 57 p. Available at: https://files.eric.ed.gov/fulltext/ED483027.pdf (Accessed 26.08.2018).
14. Lai E.R. Critical thinking: A literature review [Elektronnyi resurs]: Research Report. Vol. 6.. Pearson, 2011. 49 p. Available at: http://images.pearsonassessments.com/images/tmrs/CriticalThinkingReviewFINAL.pdf (Accessed 26.08.2018).
15. Lewis A., Smith D. Defining higher order thinking. Theory into practice, 1993, vol. 32, no. 3, pp. 131–137. doi:10.1080/00405849309543588
16. Liu O.L., Frankel L., Roohr K.C. Assessing Critical Thinking in Higher Education: Current State and Directions for Next Generation Assessment. ETS Research Report Series, 2014, vol. 2014, no. 1, pp. 1–23. doi:10.1002/ets2.12009
17. Shute V.J. et al. Melding the power of serious games and embedded assessment to monitor and foster learning. In Ritterfeld U., Cody M., Vorderer P. (Eds.). Serious games: Mechanisms and effects. New York; London: Routledge; Taylor and Francis, 2009, pp. 295–321.
18. Messick S. Alternative modes of assessment, uniform standards of validity. Princeton, N.J.: Educational Testing Service, 1994. 25 p. (ETS Research Report Series. № 2). doi:10.1002/j.2333-8504.1994.tb01634.x
19. Mislevy R.J., Almond R.G., Lukas J.F. A brief introduction to evidence-centered design: Research Report. Princeton, N.J.: Educational Testing Service, 2003. 37 p. (ETS Research Report Series. Vol. 2003. № 1.). doi:10.1002/j.2333-8504.2003.tb01908.x
20. Mislevy R.J., Haertel G.D. Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 2006, vol. 25, no. 4, pp. 6–20. doi:10.1111/j.1745-3992.2006.00075.x
21. Mislevy R.J. et al. Modeling conditional probabilities in complex educational assessments [Elektronnyi resurs]. Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies, University of California. Los Angeles, 2001. 56 p. Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.322.4516&rep=rep1&type=pdf (Accessed 26.08.2018).
22. Paul R. Critical thinking: What, why, and how. New directions for community colleges, 1992, vol. 1992, no. 77, pp. 3–24. doi:10.1002/cc.36819927703
23. Polyak S.T., von Davier A.A., Peterschmidt K. Computational Psychometrics for the Measurement of Collaborative Problem Solving Skills. Frontiers in psychology, 2017, vol. 8, pp. 20–29. doi:10.3389/fpsyg.2017.02029
24. Mislevy R.J. et al. Psychometric considerations in game-based assessment. GlassLab Report, 2014. 158 p.
25. Shute V.J., Rahimi S. Review of computer-based assessment for learning in elementary and secondary education. Journal of Computer Assisted Learning, 2017, vol. 33, no. 1, pp. 1–19. doi:10.1111/jcal.12172
26. Toulmin S.E. The uses of argument. Cambridge: Cambridge University Press, 1958. 262 p.