The Application of Item Response Theory (IRT) for AUDEE Scale Psychometric Assessment

Using item response theory (IRT), we examine AUDEE Scale that measures six different aspects of university digital educational environment (DEE): DEE Learning Process Satisfaction, DEE Communication satisfaction and Learning Motivation, DEE Stress Tension, Need for Support in DEE Learning Activity, DEE Dishonest Strategy Prevalence, and DEE Accessibility. Five-point Likert scale is used for all 38 questions of the AUDEE Scale. The study involved 406 MSUPE students who completed e-courses in the online format (90.1 % female). The average age of the participants was 28,7 ± 9,6 years (median = 24 years, minimum = 19 years, maximum = 72 years). The application of the graded response model (GRM) separately to each of the six subscales of the AUDEE Scale showed that all the items have at least moderate item discrimination, and most of the items have high item discrimination. The threshold values for each question of the AUDEE Scale, corresponding to the relative indicator of difficulty, increase monotonically. Test information functions for each subscale of the AUDEE Scale indicate that almost all measurements of the components of the AUDEE Scale are quite accurate in the range of mean values, and only for the scale 6 “DEE Accessibility” the estimation accuracy is high for the lowest values and drops sharply for medium and high values.

1. Avanesov V.S. Istoki i osnovnye ponyatiya matematicheskoi teorii pedagogicheskikh izmerenii (Item Response Theory) [The Origins and Basic Concepts of Item Response Theory] // Pedagogichesk­ie Izmereniya = Pedagogical measurements, 2007, no. 3, pp. 3–36. (In Russ.)
2. Kroker L., Algina D. Vvedenie v klassicheskuyu i sovremennuyu te­oriyu testov [Introduction to classical and modern test theory]. Mos­cow: Logos, 2010. 668 p. (In Russ.)
3. Neiman Yu.M., Khlebnikov V.A. Vvedenie v teoriyu modelirovaniya i parametrizatsii pedagogicheskikh testov [Introduction to the Theo­ry of Modeling and Parametrization of Educational Tests]. Moscow: Prometei, 2000. 168 p. (In Russ.)
4. Sorokova M.G., Odintsova M.A., Radchikova N.P. Shkala otsenki tsifrovoi obrazovatel’noi sredy (TsOS) universiteta [Scale for Assess­ing University Digital Educational Environment (AUDEE Scale)] // Psikhologicheskaya nauka i obrazovanie = Psychological Science and Education, 2021, Vol. 26, no. 2, pp. 52–65. DOI: https://doi.org/10.17759/pse.2021260205 (In Russ., Abstr. in Engl.).
5. Baker F.B. The Basis of Item Response Theory, 2nd ed. ERIC Clear­inghouse on Assessment and Evaluation, College Park, 2001. 186 p.
6. Burnham K.P., Anderson D.R. Model selection and multimodel in­ference: A practical information-theoretic approach (2nd ed.). New York: Springer-Verlag, 2002. 488 p.
7. Kang T., Chen T.T. Performance of the generalized S-X2 item fit in­dex for polytomous IRT models // Journal of Educational Measure­ment, 2008, no. 45 (4), pp. 391–406.
8. Muraki E. A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992. pp. 159–176.
9. Orlando M., Thissen D. Likelihood-based item-fit indices for dichot­omous item response theory models // Applied Psychological Meas­urement, 2000, no. 24 (1), pp. 50–64.
10. Samejima F. Graded Response Model. In W.J. van der Linden, & R.K. Hambleton (Eds.). Handbook of Modern Item Response The­ory. New York, NY: Springer, 1997. pp. 85–100.
11. Sorokova, Marina; Odintsova, Maria; Radchikova, Nataly (2021): Scale for Assessing University Digital Educational Environment (AUDEE Scale): validation database. Psychological Research Data & Tools Repository. Dataset. https://doi.org/10.25449/ruspsyda­ta.14742816