Perception of the Prosodic Formative of Multimodal Affective States

20

Abstract

The features of the expression and perception of vocal expressions as one of the forming multimodal affective states of a person are studied. The experiment, designed on the basis of the Russian-language version of the Geneva Emotion Recognition Test (GERT), involved two groups of women aged 18-45, 72 women each. One group was shown audio-video clips of 14 affective states, lasting 3-5 seconds, played by 10 professional actors according to the Stanislavsky system. The other group was presented with audio clips extracted from the same clips through headphones. It was demanded to recognize the state of the actors using an interactive image of the Geneva Emotion Wheel. It is shown that vocal expressions differ from the corresponding multimodal states by a low average accuracy of expression and identification, a different configuration and a high spread of estimates, a closer dependence on the degree of arousal, individual ways of expression and a fuzzy extended structure of categorical fields. The range of possibilities that vocal expressions have, ready to be included in cross-modal integrations, is indicated.

General Information

Keywords: : multimodal dynamic states, vocal expression, accuracy of expression and identification of emotion, structure of categorical fields, cross-modal integration

Journal rubric: Face Science

Article type: scientific article

DOI: https://doi.org/10.17759/exppsy.2024170303

Funding. The work was supported by the RNF, project No. 24-18-00904 “Mechanisms of perception of the emotional state of a person in the processes of non-verbal communication”.

Received: 20.07.2024

Accepted:

For citation: Barabanschikov V.A., Suvorova E.V., Malionok A.V. Perception of the Prosodic Formative of Multimodal Affective States. Eksperimental'naâ psihologiâ = Experimental Psychology (Russia), 2024. Vol. 17, no. 3, pp. 30–51. DOI: 10.17759/exppsy.2024170303. (In Russ., аbstr. in Engl.)

References

  1. Andreeva E.V. Fenomen krossmodal'nykh vzaimodeistvii: sovremennoe sostoyanie problemy v zarubezhnoi psikhologii. Rossiiskii nauchnyi zhurnal = Russian Scientific Journal, 2013. Vol. 1, no. 32, pp. 219—225. DOI:10.17759/rsj.20130132219 (In Russ.).
  2. Barabanschikov V.A., Zhegallo A.V., Korolkova O.A. Pertseptivnaya kategorizatsiya vyrazhenii litsa [Perceptual categorization of facial expressions]. Moscow: Publ. Kogito-Tsentr, 2016. 376 p. DOI:10.17759/9785990501513 (In Russ.).
  3. Barabanshchikov V.A., Korolkova O.A. Vospriyatie ekspressii “zhivogo” litsa. Eksperimental'naya psikhologiya = Experimental Psychology, 2020. Vol. 13, no. 3, pp. 55—73. DOI:10.17759/exppsy.2020130305 (In Russ.).
  4. Barabanshchikov V.A., Suvorova E.V. Ocenka emocional'nogo sostoyaniya cheloveka po ego videoizobrazheniyam. Eksperimental'naya psikhologiya = Experimental Psychology, 2020. Vol. 13, no. 4, pp. 4—24. DOI:10.17759/exppsy.2020130401 (In Russ.).
  5. Barabanshchikov V.A., Suvorova E.V. Ocenka multimodal'nykh ekspressii litsa v laboratorii I onlain. Lico cheloveka v kontekstakh prirody, tekhnologii i kul'tury = The Human Face in Contexts of Nature, Technology, and Culture / Eds. K.I. Ananyeva, V.A. Barabanshchikov, A.A. Demidov. Moscow: Kogito-Tsentr, 2021. Pp. 310—322. DOI:10.17759/kogito.202103220 (In Russ.).
  6. Barabanshchikov V.A., Suvorova E.V. Gendernyi faktor v raspoznavanii emocional'nogo sostoyaniya cheloveka po ego audiovideoizobrazheniyam. Rossiiskii psikhologicheskii zhurnal = Russian Psychological Journal, 2022. Vol. 19, no. 2, pp. 6—20. DOI:10.17759/rpj.2022190201 (In Russ.).
  7. Barabanshchikov V.A., Suvorova E.V. Individual'nye formy vyrazheniya I identifikaciya multimodal'nykh dinamicheskikh sostoyanii cheloveka. Poznanie i perezhivanie = Cognition and Experience, 2022. Vol. 3, no. 2, pp. 6—35. DOI:10.17759/kogito.202203220 (In Russ.).
  8. Barabanshchikov V.A., Suvorova E.V. Vyrazhenie I vospriyatie multimodal'nykh emocional'nykh sostoyanii. Natsional'nyi psikhologicheskii zhurnal = National Psychological Journal, 2023. Vol. 18, no. 3(51), pp. 106—127. DOI:10.11621/npj.2023.0311 (In Russ.).
  9. Borisova A.A. Vospriyatie emocional'nogo sostoyaniya cheloveka po intonacionnomu risunku rechi. Voprosy psikhologii = Questions of Psychology, 1989. No. 1, pp. 117—124. DOI:10.17759/voppsy.1989010117 (In Russ.).
  10. Vitt N. Lichnostno-situacionnaya oposredovannost' vyrazheniya I raspoznavaniya emocii v rechi. Voprosy psikhologii = Questions of Psychology, 1991. No. 2, pp. 95—107. DOI:10.17759/vp.1991020095 (In Russ.).
  11. Lupenko E.A., Korolkova O.A. Semanticheskaya èkvivalentnost' kak osnova intermodal'noi integracii. Eksperimental'naya psikhologiya = Experimental Psychology, 2022. Vol. 15, no. 3, pp. 159—177. DOI:10.17759/exppsy.20221503159 (In Russ.).
  12. Morozov V.P. Neverbal'naya kommunikaciya: ėksperimental'no-psikhologicheskie issledovaniya: monografiya. Moscow: Institut psikhologii RAN, 2011. 528 p. DOI:10.17759/rusbook.20110528 (In Russ.).
  13. Potapova R.K., Potapov V.V., Komalova L.R. Vospriyatie multimodal'noi mono- I polietnicheskoi kommunikacii: monografiya / Ed. L.R. Komalova. Moscow: INION RAN, 2020. 210 p. DOI:10.17759/iniom.2020210 (In Russ.).
  14. Yureva M.V., Khoze E.G. Sozdanie klassifikatora vokal'nykh emocional'nykh ekspressii: personalizirovannye bazy dannykh. Lico cheloveka v kontekstakh prirody, tekhnologii i kul'tury = The Human Face in Contexts of Nature, Technology, and Culture/ Eds. K.I. Ananyeva, V.A. Barabanshchikov. Moscow: Kogito-Tsentr; Moskovskii institute psikhoanaliza, 2020. Pp. 364—372. DOI:10.17759/kogito.2020364372 (In Russ.).
  15. Baart M., Vroomen J. Recalibration of Vocal Affect by a Dynamic Face. Experimental Brain Research, Vol. 236, no. 1, pp. 1911—1918. DOI:10.1007/s00221-018-5270-5
  16. Barkhuysen P., Krahmer E., Swerts M. Crossmodal and Incremental Perception of Audiovisual Cues to Emotional Speech. Language and Speech, Vol. 53, no. 1, pp. 3—30. DOI:10.1177/0023830910362212
  17. Bänziger T., Mortillaro M., Scherer K.R. Introducing the Geneva Multimodal Expression Corpus for Experimental Research on Emotion Perception. Emotion, Vol. 12, no. 5, pp. 1161—1179. DOI:10.1037/a0025827
  18. Bänziger T., Patel S., Scherer K.R. The Role of Perceived Voice and Speech Characteristics in Vocal Emotion Communication. Journal of Nonverbal Behavior, 2014. Vol. 38, no. 1, pp. 31—52. DOI:10.1007/s10919-013-0165-x
  19. Bryant G.A. The Evolution of Human Vocal Emotion. Emotion Review, 2021. Vol. 13, no. 1, pp. 25—33. DOI:10.1177/1754073919870985
  20. Chronaki G., Hadwin J.A., Garner M., Maurage P., Sonuga-Barke E.J.S. The Development of Emotion Recognition from Facial Expressions and Non-Linguistic Vocalizations During Childhood. The British Journal of Developmental Psychology, 2015. Vol. 33, no. 1, pp. 218—236. DOI:10.1111/bjdp.12075
  21. Cortes D.S., Tornberg C., Bänziger T., Elfenbein H.A., Fischer H., Laukka P. Effects of Aging on Emotion Recognition from Dynamic Multimodal Expressions and Vocalizations. Scientific Reports, Vol. 11, no. 1, pp. 1—12. DOI:10.1038/s41598-021-94993-4
  22. Cowen A.S., Elfenbein H.A., Laukka P., Keltner D. Mapping 24 Emotions Conveyed by Brief Human Vocalization. The American Psychologist, Vol. 74, no. 1, pp. 698—712. DOI:10.1037/amp0000399
  23. Dolan R.J., Morris J.S., de Gelder B. Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences of the United States of America, 2001. Vol. 98, no. 17, pp. 9465—9470. DOI:10.1073/pnas.171288598
  24. El Ayadi M., Kamel M.S., Karray F. Survey on speech emotion recognition: Features, classification schemes and databases. Pattern Recognition, Vol. 44, No. 3, pp. 572—587. DOI:10.1016/j.patcog.2010.09.014
  25. Gelder B. De, Vroomen J. The Perception of Emotions by Ear and by Eye. Cognition &Emotion, Vol. 14, no. 3, pp. 289—311. DOI:10.1080/026999300378824
  26. Jürgens R., Grass A., Drolet M., Fischer J. Effect of Acting Experience on Emotion Expression and Recognition in Voice: Non-Actors Provide Better Stimuli Than Expected. Journal of Nonverbal Behavior, Vol. 39, no. 1, pp. 195—214. DOI:10.1007/s10919-014-0201-6
  27. Lambrecht L., Kreifelts B., Wildgruber D. Age-Related Decrease in Recognition of Emotional Facial and Prosodic Expressions. Emotion, Vol. 12, no. 1, pp. 529—539. DOI:10.1037/a0026827
  28. Lausen A., Schacht A. Gender Differences in the Recognition of Vocal Emotions. Frontiers in Psychology, Vol. 9, no. 1, Article 882. DOI:10.3389/fpsyg.2018.00882
  29. Patel S., Scherer K.R., Björkner E., Sundberg J. Mapping Emotions into Acoustic Space: The Role of Voice Production. Biological Psychology, Vol. 87, no. 1, pp. 93—98. DOI:10.1016/j.biopsycho.2011.02.010
  30. Paulmann S., Pell M.D. Is There an Advantage for Recognizing Multi-Modal Emotional Stimuli? Motivation and Emotion, Vol. 35, no. 1, pp. 192—201. DOI:10.1007/s11031-011-9211-y
  31. Pittam J., Scherer K.R. Vocal Expression and Communication of Emotion/ In Lewis M., Haviland J.M. (Eds.). Handbook of Emotions. The Guilford Press. 1993. Pp. 185—197.
  32. Sauter D.A., Eisner F., Calder A., Scott S. Perceptual Cues in Nonverbal Vocal Expressions of Emotion. Quarterly Journal of Experimental Psychology, Vol. 63, no. 1, pp. 2251—2272. DOI:10.1080/17470211003721642
  33. Scherer K.R. Vocal Affect Expression: A Review and a Model for Future Research. Psychological Bulletin, Vol. 99, no. 2, pp. 143—165. DOI:10.1037/0033-2909.99.2.143
  34. Scherer K.R. What Are Emotions? And How Can They Be Measured. Social Science Information, 2005. Vol. 44, no. 1, pp. 695—729. DOI:10.1177/0539018405058216
  35. Scherer K., Banse R. Acoustic Profiles in Vocal Emotion Expression. Journal of Personality and Social Psychology, 1996. Vol. 70, no. 1, pp. 614—636. DOI:10.1037/0022-3514.70.3.614
  36. Scherer K.R., Banse R., Wallbott H.G., Goldbeck T. Vocal Cues in Emotion Encoding and Decoding. Motivation and Emotion, Vol. 15, no. 2, pp. 123—148. DOI:10.1007/BF00995674
  37. Schirmer A., Adolphs R. Emotion Perception from Face, Voice, and Touch: Comparisons and Convergence. Trends in Cognitive Sciences, 2017. Vol. 21, no. 1, pp. 216—228.DOI:10.1016/j.tics.2017.01.001
  38. Schlegel K., SchererK.R.The nomological network of emotion knowledge and emotion understanding in adults: evidence from two new performance-based tests. Cognition and Emotion, 2017. 32. pp. 1514—1530. DOI:10.1080/02699931.2017.1414687
  39. Schlegel K., Grandjean D., Scherer K.R. Emotion Recognition: Unidimensional Ability or a Set of Modality- and Emotion-Specific Skills? Personality and Individual Differences, 2012. Vol. 53, no. 1, pp. 16—21.DOI:10.1016/j.paid.2012.01.026
  40. Schlegel K., Grandjean D., Scherer K. R. Introducing the Geneva Emotion Recognition Test: An Example of Rasch-Based Test Development. Psychological Assessment, Vol. 26, no. 2, pp. 666—672. DOI:10.1037/a0025827
  41. Sen A., Isaacowitz D., Schirmer A. Age Differences in Vocal Emotion Perception: On the Role of Speaker Age and Listener Sex. Cognition & Emotion, Vol. 32, no. 1, pp. 1189—1204. DOI:10.1080/02699931.2017.1390725
  42. Williams C., Stevens K. Emotions and speech: some acoustical correlates.The Journal of the Acoustical Society of America, 1972. Vol. 52, no. 4(2), pp. 1238—1250. DOI:10.1121/1.1913238
  43. Wilting J., Krahmer E., Swerts M. Real vs acted emotional speech. Proceedings of Interspeech 2006. Pittsburgh, 2006. DOI:10.21437/Interspeech.2006-250

Information About the Authors

Vladimir A. Barabanschikov, Doctor of Psychology, Professor, Director, Institute of Experimental Psychology, Moscow State University of Psychology and Education, Dean of the Faculty of Psychology, Moscow Institute of Psychoanalysis, Moscow, Russia, ORCID: https://orcid.org/0000-0002-5084-0513, e-mail: vladimir.barabanschikov@gmail.com

Ekaterina V. Suvorova, Research Associate, Institute of Experimental Psychology, Moscow State University of Psychology & Education, PhD Student, Moscow Institute of Psychoanalysis, Moscow, Russia, ORCID: https://orcid.org/0000-0001-8834-2037, e-mail: esresearch@yandex.ru

Arina V. Malionok, Institute of Experimental Psychology, Moscow State University of psychology and education (MSUPE), Moscow, Russia, ORCID: https://orcid.org/0009-0001-7375-4343, e-mail: arinamalionok333@gmail.com

Metrics

Views

Total: 70
Previous month: 28
Current month: 42

Downloads

Total: 20
Previous month: 8
Current month: 12