Automatic engagement detection in the education: critical review

This paper reviews the key research of the automatic engagement detection in education. Automatic engagement detection is necessary in enhancing educational process, there is a lack of out-of-the-box technical solutions. Engagement can be detected while tracing learning-centered affects: interest, confusion, frustration, delight, anger, boredom, and their facial and bodily expressions. Most of the researchers reveal these emotions on video using Facial Action Coding System (FACS). But there doesn’t exist a set of ready-made criteria to detect engagement and many scientists use additional techniques like self-reports, audio-data, physiological indicators and others. In this paper we present a review of most recent researches in the field of automatic affect and engagement detection and present our theoretical model of engagement in educational process based on the learning-centered affects’s detection. Engagement is understood as an affective and cognitive state, accompanying learning process. While reaching optimal engagement students experience various affects, where highly positive and negative feelings mean that a student is close to be engaged in the learning process.

1. Il'in E.P. Psikhofiziologiya sostoyanii cheloveka [Psychophysiology of human states]. St. Petersburg: Piter, 2005. 412 p.(In Russ.).
2. Kupriyanov R.B. Primenenie tekhnologii komp'yuternogo zreniya dlya avtomaticheskogo sbora dannykh ob emotsiyakh obuchayushchikhsya vo vremya gruppovoi raboty [Application of computer vision technologies for automatic collection of data about students' emotions during group work]. Informatika i obrazovanie [Informatics and Education],2020. Vol. 314, no. 5,pp. 56–63. DOI:10.32517/0234-0453-2020-35-5-56-63(In Russ.).
3. Altuwairqi K. et al. A new emotion–based affective model to detect student’s engagement. Journal of King Saud University – Computer and Information Sciences, 2019. In Press. DOI:10.1016/j.jksuci.2018.12.008
4. Zeng Z. et al. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009. Vol. 31, no. 1, pp. 39–58. DOI:10.1109/TPAMI.2008.52
5. Craig S. et al. Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 2004. Vol. 29, no. 3, pp. 241–250. DOI:10.1080/1358165042000283101
6. Ainley M. Connecting with learning: Motivation, affect and cognition in interest processes. Educational Psychology Review, 2006. Vol. 18, pp. 391–405. DOI:10.1007/s10648-006-9033-0
7. Bosch N. et al. Automatic detection of learning-centered affective states in the wild. In Proceedings of the 20th International Conference on Intelligent User Interfaces. New York: Association for Computing Machinery, 2015, pp. 379–388. DOI:10.1145/2678025.2701397
8. Baker R.S.J., Rodrigo M.M.T., Xolocotzin U.E. The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. International Conference on Affective Computing and Intelligent Interaction. Affective Computing and Intelligent Interaction. Lisbon: ACII, 2007, pp. 666–677. DOI:10.1007/978-3-540-74889-2_58
9. Beck J. Engagement tracing: using response times to model student disengagement. In Chee-Kit Looi (ed.), Artificial Intelligence in Education: Supporting Learning Through Intelligent and Socially Informed Technology. Amsterdam: IOS Press, 2005, pp. 88–95.
10. Bosch N. Detecting student engagement: Human versus machine Association for Computing Machinery. In Vassileva J. et al. (eds.), UMAP '16: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization. New York: Association for Computing Machinery, 2016, pp. 317–320. DOI:10.1145/2930238.2930371
11. Bosch N., Chen Y., D’Mello S. It’s Written on Your Face: Detecting Affective States from Facial Expressions while Learning Computer Programming. Intelligent Tutoring Systems. Lecture Notes in Computer Science, 2014. Vol. 8474, pp. 39–44. DOI:10.1007/978-3-319-07221-0_5
12. Calvo M.G., Nummenmaa L. Processing of Unattended Emotional Visual Scenes. Journal of Experimental Psychology: General, 2007. Vol. 136, no. 3, pp. 347–369. DOI:10.1037/0096-3445.136.3.347
13. Calvo R.A., D’Mello S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 2010. Vol. 1, no. 1, pp. 18–37. DOI:10.1109/T-AFFC.2010.1
14. Christenson S.L., Wylie C., Reschly A.L. Handbook of Research on Student Engagement. New York: Springer, 2012. 840 p.DOI:10.1007/978-1-4614-2018-7
15. Conati C., MacLaren H. Empirically building and evaluating a probabilistic model of user affect.User Modeling and User-Adapted Interaction, 2009. Vol. 19, pp. 267–303. DOI:10.1007/s11257-009-9062-8
16. D’Mello S. A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 2013. Vol. 105, no. 4, pp. 1082–1099. DOI:10.1037/a0032674
17. D’Mello S., Graesser A. Dynamics of affective states during complex learning. Learning and Instruction, 2012. Vol. 22, no. 2, pp. 145–157. DOI:10.1016/j.learninstruc.2011.10.001
18. Dixson M.D. Measuring student engagement in the online course: The online student engagement scale (OSE).Journal of Asynchronous Learning Network, 2015. Vol. 4, no. 19. DOI:10.24059/olj.v19i4.561
19. Ekman P., Freisen W.V., Ancoli S. Facial signs of emotional experience. Journal of Personality and Social Psychology, 1980. Vol. 39, no. 6, pp. 1125–1134. DOI:10.1037/h0077722
20. Ekman P., Friesen W. V. Facial Action Coding System: A Technique for the Measurement of Facial Movement. California: Consulting Psychologists Press, 1978. 197 p.
21. Arroyo I. et al. Emotion Sensors Go to School. In Dimitrova V. et al. (eds.), Artificial Intelligence in Education. Amsterdam: IOS Press, 2009, pp. 17–24. (Frontiers in Artificial Intelligence and Applications).
22. Bergdahl N. et al. Engagement, disengagement and performance when learning with technologies in upper secondary school [Elektronnyi resurs]. Computers & Education, 2020. Vol. 149, Article number 103783. URL: https://www.sciencedirect.com/science/article/abs/pii/S0360131519303331 (Accessed 18.08.2020).
23. Fredricks J.A., Blumenfeld P.C., Paris A.H. School Engagement: Potential of the Concept, State of the Evidence. Review of Educational Research, 2004. Vol. 74, no. 1, pp. 59–109. DOI:10.3102/00346543074001059
24. Frenzel A.C., Pekrun R., Goetz T. Perceived learning environment and students’ emotional experiences: A multilevel analysis of mathematics classrooms. Learning and Instruction, 2007. Vol. 17, no. 5, pp. 478–493. DOI:10.1016/j.learninstruc.2007.09.001
25. Frijda N.H. Emotion, cognitive structure, and action tendency. Cognition and Emotion, 1987. Vol. 1, no. 2, pp. 115–143. DOI:10.1080/02699938708408043
26. Kupriyanov R. et al. Intelligent tool for developing student’s social skills [Elektronnyi resurs]. Proceedings of Edulearn20 Conference: 6th-7th July 2020. Mallorca, 2020, pp. 2408–2413. URL: http://132.66.16.6/~ilia1/744.pdf (Accessed 18.08.2020).
27. Kapoor A., Burleson W., Picard R.W. Automatic prediction of frustration. International Journal of Human Computer Studies, 2007. Vol. 65, no. 8, pp. 724–736. DOI:10.1016/j.ijhcs.2007.02.003
28. Marks H.M. Student Engagement in Instructional Activity: Patterns in the Elementary, Middle, and High School Years. American Educational Research Journal, 2000. Vol. 37, no. 1, pp. 153–184. DOI:10.3102/00028312037001153
29. Matthews G., Davies D.R. Individual differences in energetic arousal and sustained attention: A dual-task study. Personality and Individual Differences, 2001. Vol. 31, no. 4, pp. 575–589. DOI:10.1016/S0191-8869(00)00162-8
30. Appleton J.J. et al. Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 2006. Vol. 44, no. 5, pp. 427–445. DOI:10.1016/j.jsp.2006.04.002
31. Mota S., Picard R. Automated Posture Analysis for detecting Learner’s Interest Level [Elektronnyi resurs].Conference on Computer Vision and Pattern Recognition Workshop. Vol. 5. Madison: IEEE, 2003., pp. 49. URL: https://affect.media.mit.edu/pdfs/03.mota-picard.pdf (Accessed 18.08.2020).
32. Newmann F., Wehlage G., Lamborn S.D. The significance and sources of student engagement. In Newman F. Student Engagement and Achievement in American Secondary School. New York: Teachers College Press, 1992, pp. 11–39. URL: https://files.eric.ed.gov/fulltext/ED371047.pdf#page=16 (Accessed 18.08.2020).
33. Pantic M., Rothkrantz L. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the Institute of Electrical and Electronics Engineers, 2003, pp. 1370–1390. DOI:10.1109/JPROC.2003.817122
34. Pontual Falcão T. et al. Participatory methodologies to promote student engagement in the development of educational digital games. Computers and Education, 2018. Vol. 116, pp.161–175. DOI:10.1016/j.compedu.2017.09.006
35. Matthews G. et al. Profiling task stress with the dundee state questionnaire. In Cavalcanti L., Azevedo S. (eds.), Psychology of Stress. New York: Nova Science Publishers, 2013, pp. 49–91.
36. Roseman I. Cognitive determinants of emotion: A structural theory. Review of Personality & Social Psychology, 1984. Vol. 5, pp. 11–36.
37. Russell J.A. Core Affect and the Psychological Construction of Emotion. Psychological Review, 2003. Vol. 110, no. 1, pp. 145–172. DOI:10.1037/0033-295X.110.1.145
38. Salovey P. Introduction: Emotion and Social Processes. In Davidson R.J., Scherer K.R., Goldsmith H.H. (eds.),Series in affective science. Handbook of affective sciences. Oxford: Oxford University Press, 2003, pp. 3–7.
39. Scherer K.R. Facets of emotion: recent research. London: Psychology Press, 1988. 280 p.
40. Schutz P., Pekrun R. Emotion in Education. London: Academic Press, 2007. 368 p.
41. Whitehill J. et al. The faces of engagement: Automatic recognition of student engagement from facial expressions.IEEE Transactions on Affective Computing, 2014. Vol. 5, no. 1, pp. 86–98. DOI:10.1109/TAFFC.2014.2316163
42. Xie K., Heddy B.C., Greene B.A. Affordances of using mobile technology to support experience-sampling method in examining college students’ engagement.Computers and Education, 2019. Vol. 128, pp. 183–198. DOI:10.1016/j.compedu.2018.09.020