Experimental Psychology (Russia)
2024. Vol. 17, no. 3, 30–51
doi:10.17759/exppsy.2024170303
ISSN: 2072-7593 / 2311-7036 (online)
Perception of the Prosodic Formative of Multimodal Affective States
Abstract
The features of the expression and perception of vocal expressions as one of the forming multimodal affective states of a person are studied. The experiment, designed on the basis of the Russian-language version of the Geneva Emotion Recognition Test (GERT), involved two groups of women aged 18-45, 72 women each. One group was shown audio-video clips of 14 affective states, lasting 3-5 seconds, played by 10 professional actors according to the Stanislavsky system. The other group was presented with audio clips extracted from the same clips through headphones. It was demanded to recognize the state of the actors using an interactive image of the Geneva Emotion Wheel. It is shown that vocal expressions differ from the corresponding multimodal states by a low average accuracy of expression and identification, a different configuration and a high spread of estimates, a closer dependence on the degree of arousal, individual ways of expression and a fuzzy extended structure of categorical fields. The range of possibilities that vocal expressions have, ready to be included in cross-modal integrations, is indicated.
General Information
Keywords: : multimodal dynamic states, vocal expression, accuracy of expression and identification of emotion, structure of categorical fields, cross-modal integration
Journal rubric: Face Science
Article type: scientific article
DOI: https://doi.org/10.17759/exppsy.2024170303
Funding. The work was supported by the RNF, project No. 24-18-00904 “Mechanisms of perception of the emotional state of a person in the processes of non-verbal communication”.
Received: 20.07.2024
Accepted:
For citation: Barabanschikov V.A., Suvorova E.V., Malionok A.V. Perception of the Prosodic Formative of Multimodal Affective States. Eksperimental'naâ psihologiâ = Experimental Psychology (Russia), 2024. Vol. 17, no. 3, pp. 30–51. DOI: 10.17759/exppsy.2024170303. (In Russ., аbstr. in Engl.)
References
- Andreeva E.V. Fenomen krossmodal'nykh vzaimodeistvii: sovremennoe sostoyanie problemy v zarubezhnoi psikhologii. Rossiiskii nauchnyi zhurnal = Russian Scientific Journal, 2013. Vol. 1, no. 32, pp. 219—225. DOI:10.17759/rsj.20130132219 (In Russ.).
- Barabanschikov V.A., Zhegallo A.V., Korolkova O.A. Pertseptivnaya kategorizatsiya vyrazhenii litsa [Perceptual categorization of facial expressions]. Moscow: Publ. Kogito-Tsentr, 2016. 376 p. DOI:10.17759/9785990501513 (In Russ.).
- Barabanshchikov V.A., Korolkova O.A. Vospriyatie ekspressii “zhivogo” litsa. Eksperimental'naya psikhologiya = Experimental Psychology, 2020. Vol. 13, no. 3, pp. 55—73. DOI:10.17759/exppsy.2020130305 (In Russ.).
- Barabanshchikov V.A., Suvorova E.V. Ocenka emocional'nogo sostoyaniya cheloveka po ego videoizobrazheniyam. Eksperimental'naya psikhologiya = Experimental Psychology, 2020. Vol. 13, no. 4, pp. 4—24. DOI:10.17759/exppsy.2020130401 (In Russ.).
- Barabanshchikov V.A., Suvorova E.V. Ocenka multimodal'nykh ekspressii litsa v laboratorii I onlain. Lico cheloveka v kontekstakh prirody, tekhnologii i kul'tury = The Human Face in Contexts of Nature, Technology, and Culture / Eds. K.I. Ananyeva, V.A. Barabanshchikov, A.A. Demidov. Moscow: Kogito-Tsentr, 2021. Pp. 310—322. DOI:10.17759/kogito.202103220 (In Russ.).
- Barabanshchikov V.A., Suvorova E.V. Gendernyi faktor v raspoznavanii emocional'nogo sostoyaniya cheloveka po ego audiovideoizobrazheniyam. Rossiiskii psikhologicheskii zhurnal = Russian Psychological Journal, 2022. Vol. 19, no. 2, pp. 6—20. DOI:10.17759/rpj.2022190201 (In Russ.).
- Barabanshchikov V.A., Suvorova E.V. Individual'nye formy vyrazheniya I identifikaciya multimodal'nykh dinamicheskikh sostoyanii cheloveka. Poznanie i perezhivanie = Cognition and Experience, 2022. Vol. 3, no. 2, pp. 6—35. DOI:10.17759/kogito.202203220 (In Russ.).
- Barabanshchikov V.A., Suvorova E.V. Vyrazhenie I vospriyatie multimodal'nykh emocional'nykh sostoyanii. Natsional'nyi psikhologicheskii zhurnal = National Psychological Journal, 2023. Vol. 18, no. 3(51), pp. 106—127. DOI:10.11621/npj.2023.0311 (In Russ.).
- Borisova A.A. Vospriyatie emocional'nogo sostoyaniya cheloveka po intonacionnomu risunku rechi. Voprosy psikhologii = Questions of Psychology, 1989. No. 1, pp. 117—124. DOI:10.17759/voppsy.1989010117 (In Russ.).
- Vitt N. Lichnostno-situacionnaya oposredovannost' vyrazheniya I raspoznavaniya emocii v rechi. Voprosy psikhologii = Questions of Psychology, 1991. No. 2, pp. 95—107. DOI:10.17759/vp.1991020095 (In Russ.).
- Lupenko E.A., Korolkova O.A. Semanticheskaya èkvivalentnost' kak osnova intermodal'noi integracii. Eksperimental'naya psikhologiya = Experimental Psychology, 2022. Vol. 15, no. 3, pp. 159—177. DOI:10.17759/exppsy.20221503159 (In Russ.).
- Morozov V.P. Neverbal'naya kommunikaciya: ėksperimental'no-psikhologicheskie issledovaniya: monografiya. Moscow: Institut psikhologii RAN, 2011. 528 p. DOI:10.17759/rusbook.20110528 (In Russ.).
- Potapova R.K., Potapov V.V., Komalova L.R. Vospriyatie multimodal'noi mono- I polietnicheskoi kommunikacii: monografiya / Ed. L.R. Komalova. Moscow: INION RAN, 2020. 210 p. DOI:10.17759/iniom.2020210 (In Russ.).
- Yureva M.V., Khoze E.G. Sozdanie klassifikatora vokal'nykh emocional'nykh ekspressii: personalizirovannye bazy dannykh. Lico cheloveka v kontekstakh prirody, tekhnologii i kul'tury = The Human Face in Contexts of Nature, Technology, and Culture/ Eds. K.I. Ananyeva, V.A. Barabanshchikov. Moscow: Kogito-Tsentr; Moskovskii institute psikhoanaliza, 2020. Pp. 364—372. DOI:10.17759/kogito.2020364372 (In Russ.).
- Baart M., Vroomen J. Recalibration of Vocal Affect by a Dynamic Face. Experimental Brain Research, Vol. 236, no. 1, pp. 1911—1918. DOI:10.1007/s00221-018-5270-5
- Barkhuysen P., Krahmer E., Swerts M. Crossmodal and Incremental Perception of Audiovisual Cues to Emotional Speech. Language and Speech, Vol. 53, no. 1, pp. 3—30. DOI:10.1177/0023830910362212
- Bänziger T., Mortillaro M., Scherer K.R. Introducing the Geneva Multimodal Expression Corpus for Experimental Research on Emotion Perception. Emotion, Vol. 12, no. 5, pp. 1161—1179. DOI:10.1037/a0025827
- Bänziger T., Patel S., Scherer K.R. The Role of Perceived Voice and Speech Characteristics in Vocal Emotion Communication. Journal of Nonverbal Behavior, 2014. Vol. 38, no. 1, pp. 31—52. DOI:10.1007/s10919-013-0165-x
- Bryant G.A. The Evolution of Human Vocal Emotion. Emotion Review, 2021. Vol. 13, no. 1, pp. 25—33. DOI:10.1177/1754073919870985
- Chronaki G., Hadwin J.A., Garner M., Maurage P., Sonuga-Barke E.J.S. The Development of Emotion Recognition from Facial Expressions and Non-Linguistic Vocalizations During Childhood. The British Journal of Developmental Psychology, 2015. Vol. 33, no. 1, pp. 218—236. DOI:10.1111/bjdp.12075
- Cortes D.S., Tornberg C., Bänziger T., Elfenbein H.A., Fischer H., Laukka P. Effects of Aging on Emotion Recognition from Dynamic Multimodal Expressions and Vocalizations. Scientific Reports, Vol. 11, no. 1, pp. 1—12. DOI:10.1038/s41598-021-94993-4
- Cowen A.S., Elfenbein H.A., Laukka P., Keltner D. Mapping 24 Emotions Conveyed by Brief Human Vocalization. The American Psychologist, Vol. 74, no. 1, pp. 698—712. DOI:10.1037/amp0000399
- Dolan R.J., Morris J.S., de Gelder B. Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences of the United States of America, 2001. Vol. 98, no. 17, pp. 9465—9470. DOI:10.1073/pnas.171288598
- El Ayadi M., Kamel M.S., Karray F. Survey on speech emotion recognition: Features, classification schemes and databases. Pattern Recognition, Vol. 44, No. 3, pp. 572—587. DOI:10.1016/j.patcog.2010.09.014
- Gelder B. De, Vroomen J. The Perception of Emotions by Ear and by Eye. Cognition &Emotion, Vol. 14, no. 3, pp. 289—311. DOI:10.1080/026999300378824
- Jürgens R., Grass A., Drolet M., Fischer J. Effect of Acting Experience on Emotion Expression and Recognition in Voice: Non-Actors Provide Better Stimuli Than Expected. Journal of Nonverbal Behavior, Vol. 39, no. 1, pp. 195—214. DOI:10.1007/s10919-014-0201-6
- Lambrecht L., Kreifelts B., Wildgruber D. Age-Related Decrease in Recognition of Emotional Facial and Prosodic Expressions. Emotion, Vol. 12, no. 1, pp. 529—539. DOI:10.1037/a0026827
- Lausen A., Schacht A. Gender Differences in the Recognition of Vocal Emotions. Frontiers in Psychology, Vol. 9, no. 1, Article 882. DOI:10.3389/fpsyg.2018.00882
- Patel S., Scherer K.R., Björkner E., Sundberg J. Mapping Emotions into Acoustic Space: The Role of Voice Production. Biological Psychology, Vol. 87, no. 1, pp. 93—98. DOI:10.1016/j.biopsycho.2011.02.010
- Paulmann S., Pell M.D. Is There an Advantage for Recognizing Multi-Modal Emotional Stimuli? Motivation and Emotion, Vol. 35, no. 1, pp. 192—201. DOI:10.1007/s11031-011-9211-y
- Pittam J., Scherer K.R. Vocal Expression and Communication of Emotion/ In Lewis M., Haviland J.M. (Eds.). Handbook of Emotions. The Guilford Press. 1993. Pp. 185—197.
- Sauter D.A., Eisner F., Calder A., Scott S. Perceptual Cues in Nonverbal Vocal Expressions of Emotion. Quarterly Journal of Experimental Psychology, Vol. 63, no. 1, pp. 2251—2272. DOI:10.1080/17470211003721642
- Scherer K.R. Vocal Affect Expression: A Review and a Model for Future Research. Psychological Bulletin, Vol. 99, no. 2, pp. 143—165. DOI:10.1037/0033-2909.99.2.143
- Scherer K.R. What Are Emotions? And How Can They Be Measured. Social Science Information, 2005. Vol. 44, no. 1, pp. 695—729. DOI:10.1177/0539018405058216
- Scherer K., Banse R. Acoustic Profiles in Vocal Emotion Expression. Journal of Personality and Social Psychology, 1996. Vol. 70, no. 1, pp. 614—636. DOI:10.1037/0022-3514.70.3.614
- Scherer K.R., Banse R., Wallbott H.G., Goldbeck T. Vocal Cues in Emotion Encoding and Decoding. Motivation and Emotion, Vol. 15, no. 2, pp. 123—148. DOI:10.1007/BF00995674
- Schirmer A., Adolphs R. Emotion Perception from Face, Voice, and Touch: Comparisons and Convergence. Trends in Cognitive Sciences, 2017. Vol. 21, no. 1, pp. 216—228.DOI:10.1016/j.tics.2017.01.001
- Schlegel K., SchererK.R.The nomological network of emotion knowledge and emotion understanding in adults: evidence from two new performance-based tests. Cognition and Emotion, 2017. 32. pp. 1514—1530. DOI:10.1080/02699931.2017.1414687
- Schlegel K., Grandjean D., Scherer K.R. Emotion Recognition: Unidimensional Ability or a Set of Modality- and Emotion-Specific Skills? Personality and Individual Differences, 2012. Vol. 53, no. 1, pp. 16—21.DOI:10.1016/j.paid.2012.01.026
- Schlegel K., Grandjean D., Scherer K. R. Introducing the Geneva Emotion Recognition Test: An Example of Rasch-Based Test Development. Psychological Assessment, Vol. 26, no. 2, pp. 666—672. DOI:10.1037/a0025827
- Sen A., Isaacowitz D., Schirmer A. Age Differences in Vocal Emotion Perception: On the Role of Speaker Age and Listener Sex. Cognition & Emotion, Vol. 32, no. 1, pp. 1189—1204. DOI:10.1080/02699931.2017.1390725
- Williams C., Stevens K. Emotions and speech: some acoustical correlates.The Journal of the Acoustical Society of America, 1972. Vol. 52, no. 4(2), pp. 1238—1250. DOI:10.1121/1.1913238
- Wilting J., Krahmer E., Swerts M. Real vs acted emotional speech. Proceedings of Interspeech 2006. Pittsburgh, 2006. DOI:10.21437/Interspeech.2006-250
Information About the Authors
Metrics
Views
Total: 70
Previous month: 28
Current month: 42
Downloads
Total: 20
Previous month: 8
Current month: 12