Semantic Equivalence as the Basis for Intermodal Integration

We studied the mechanisms of combining sensory information across different modalities. The results of the experimental study indicate that when comparing objects in different modalities, observers rely not on modal-specific characteristics, but on general semantic content. Similar results were obtained for two types of comparison: verbal (the words denoting a particular emotion were compared with pictures of emotions) and non-verbal, where pictures of emotions and emotional faces were the objects of comparison. In addition, the data of the conducted study indicate that the semantic content is successfully extracted both during a sufficiently long stimuli presentation (2 s) and when solving a near-threshold problem (reducing the exposure time to 30 ms, as well as to 30 ms with forward mask), i.e. in the absence of time for the formation of a representation, which indicates the presence of early semantic information processing. Importantly, reducing the exposure time to near-threshold values in some cases leads to a more efficient comparison of objects. We suggest that the same cognitive mechanism underlies the comparison and integration of heterogeneous objects of any modality.

1. Andreeva E.V. Fenomen krossmodal’nykh vzaimodeistvii: sovremennoe sostoyanie problemy v zarubezhnoi psikhologii [The phenomenon of cross-modal interactions: the current state of the problem in foreign psychology]. Rossiiskii nauchnyi zhurnal [Russian Scientific Journal], 2013. Vol. 1, no. 32, pp. 219—225. (In Russ.).
2. Velichkovskii B.M. Sovremennaya kognitivnaya psikhologiya [Modern cognitive psychology]. Moscow: Moscow State University Publishing House, 1982. 336 p. (In Russ.).
3. Velichkovskii B.M., Zinchenko V.P., Luriya A.R. Psikhologiya vospriyatiya [Psychology of perception]. Moscow: Moscow State University Publishing House, 1973. 247 p. (In Russ.).
4. Wundt V. Psikhologiya dushevnykh volnenii [Psychology of emotional unrest]. In V.K. Vilyunas, Yu.B. Gippenreiter (eds.) / Psikhologiya emotsii. Teksty [The Psychology of Emotions. Texts]. Moscow: Moscow State University Publishing House, 1984. P. 47—63. (In Russ.).
5. Izmailov Ch.A., Sokolov E.N., Korshunova S.G. Tsvetovoe prostranstvo cheloveka, osnovannoe na dannykh korkovykh vyzvannykh potentsialov [Human color space based on cortical evoked potential data]. Sensornye sistemy [Sensory Systems], 2003. Vol. 17, no. 1, pp. 32—44. (In Russ.).
6. Izmailov Ch.A., Sokolov E.N., Korshunova S.G., Kadik A.A. Semanticheskii komponent vyzvannogo potentsiala razlicheniya [The semantic component of the evoked discrimination potential]. In A.L. Zhuravlev, N.V. Tarabrina (ed.) / Psikhologiya. Sovremennye napravleniya mezhdistsiplinarnykh issledovanii [Psychology. Modern directions of interdisciplinary research]. Moscow: IPRAS Publ., 2003. P. 295—308. (In Russ.).
7. Kravkov S.V. Vzaimodeistvie organov chuvstv [The interaction of the senses]. Moscow: Publishing House of the Academy of Sciences of the USSR, 1948. 128 p. (In Russ.).
8. Leont’ev A.N. Razvitie pamyati: Eksperimental’noe issledovanie vysshikh psikhologicheskikh funktsii [Memory Development: An Experimental Study of Higher Psychological Functions]. Moscow — Leningrad: Uchpedgiz, 1931. 276 p. (In Russ.).
9. Lupenko E.A. Intermodal’noe skhodstvo kak rezul’tat kategorizatsii [Intermodal similarity as a result of categorization]. Eksperimental’naya psikhologiya [Experimental Psychology (Russia)], 2009. Vol. 2, no. 2, pp. 84—103. (In Russ., abstr. in Engl.).
10. Lupenko E.A. Issledovanie psikhologicheskoi prirody intermodal’noi obshchnosti oshchushchenii [Investigation of the psychological nature of the intermodal community of sensations]. Psikhologicheskii zhurnal [Psychological journal], 2008, no. 1, pp. 69—78. (In Russ.).
11. Lupenko E.A., Korol’kova O.A. Intermodal’noe sopostavlenie ob”ektov s emotsional’nym soderzhaniem [Intermodal comparison of objects with emotional content]. In V.A. Barabanshchikov (ed.), Estestvenno-nauchnyi podkhod v sovremennoi psikhologii [Scientific Approach in Contemporary Psychology]. Moscow: IPRAS Publ., 2014. P. 426—433. (In Russ.).
12. Lupenko E.A., Korol’kova O.A. Emotsional’noe obobshchenie kak kognitivnyi mekhanizm [Emotional generalization as a cognitive mechanism]. In Yu.I. Aleksandrov, K.V. Anokhin (eds.), Sed’maya mezhdunarodnaya konferentsiya po kognitivnoi nauke: Tezisy dokladov. Svetlogorsk, 20—24 iyunya 2016 g. [The Seventh International Conference on Cognitive Science: Abstracts. Svetlogorsk, June 20—24, 2016]. Svetlogorsk: IPRAS Publ., 2016. P. 398—399. (In Russ.).
13. Luria A.R. Lektsii po obshchei psikhologii [Lectures on General Psychology]. Saint-Petersburg: Piter Publishing House, 2006. 320 p. (In Russ.).
14. Luria A.R. Malen’kaya knizhka o bol’shoi pamyati [A small book about great memory]. Moscow: Moscow State University Publishing House, 1980. 88 p. (In Russ.).
15. Natadze R.G. K voprosu o psikhologicheskoi prirode intermodal’noi obshchnosti oshchushchenii [On the question of the psychological nature of the intermodal community of sensations]. Voprosy psikhologii [Questions of Psychology], 1979. no. 6, pp. 49—57. (In Russ.).
16. Sidorov-Dorso A.V. Sovremennye issledovaniya sinestezii estestvennogo razvitiya [Modern research on natural developmental synesthesia]. Voprosy psikhologii [Questions of Psychology], 2013. Vol. 4, pp. 147—158. (In Russ.).
17. Smirnov S.D. Ponyatie «obraz mira» i ego znachenie dlya psikhologii poznavatel’nykh protsessov [The concept of “image of the world” and its meaning for the psychology of cognitive processes]. In A.N. Leont’ev i sovremennaya psikhologiya [A.N. Leontiev and modern psychology]. Moscow: Moscow State University Publishing House, 1983. P. 149—154. (In Russ.).
18. Shakhbazova E.S. Metod issledovaniya mezhsistemnykh i mezhmodal’nykh vzaimodeistvii [Research method for intersystem and intermodal interactions]. Psikhologicheskie issledovaniya: elektronnyi nauchnyi zhurnal [Psychological Research: Electronic Scientific Journal], 2011. Vol. 4, no. 18. (In Russ.). DOI:10.54359/ps.v4i18.839
19. Shakhbazova E.S. Pertseptivno-verbal’nye vzaimodeistviya v kognitivnoi sfere pri rasseyannom skleroze v podrostkovom vozraste. Avtoref. dis. … kand. psikhol. nauk [Perceptual-verbal interactions in the cognitive sphere in multiple sclerosis in adolescence. Abstract of thesis ... cand. psychol. sciences]. 2012. (In Russ.).
20. Baron-Cohen S. Is there a normal phase of synaesthesia in development. Psyche, 1996. Vol. 2, no. 27, pp. 223—228.
21. Baron-Cohen S., Harrison J.E. Synaesthesia: Classic and Contemporary Readings. Oxford: Blackwell Publishing Ltd, 1996. 296 p.
22. Campen C. van. The Hidden Sense. Cambridge, MA: The MIT Press, 2007. 185 p. DOI:10.7551/mitpress/7492.001.0001
23. Cytowic R.E. Synesthesia: A Union of the Senses (2nd edition). Cambridge, Mass.: MIT Press, 2002. 394 p.
24. Deroy O., Spence C. Why we are not all synesthetes (not even weakly so). Psychonomic Bulletin & Review, 2013. Vol. 20, no. 4, pp. 643—664. DOI:10.3758/s13423-013-0387-2
25. Ekman P. Pictures of facial affect. Oakland, CA: Paul Ekman, 1993.
26. Gottfried A.W., Rose S.A., Bridger W.H. Effects of visual, haptic, and manipulatory experiences on infants’ visual recognition memory of objects. Developmental Psychology, 1978. Vol. 14, no. 3, pp. 305—312. DOI:10.1037/0012-1649.14.3.305
27. Laurienti P.J., Kraft R.A., Maldjian J.A., Burdette J.H., Wallace M.T. Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research, 2004. Vol. 158, no. 4, pp. 405—414. DOI:10.1007/s00221-004-1913-2
28. Lehmann S., Murray M.M. The role of multisensory memories in unisensory object discrimination. Cognitive Brain Research, 2005. Vol. 24, no. 2, pp. 326—334. DOI:10.1016/j.cogbrainres.2005.02.005
29. Lewkowicz D.J. Development of intersensory perception in human infants. In D.J. Lewkowicz, R. Lickliter (ed.), The development of intersensory perception: Comparative perspectives. N.Y.: Lawrence Erlbaum Associates, 1994. P. 165—203.
30. Lewkowicz D.J., Turkewitz G. Cross-modal equivalence in early infancy: Auditory-visual intensity matching. Developmental Psychology, 1980. Vol. 16, no. 6, pp. 597—607. DOI:10.1037/0012-1649.16.6.597
31. Marks L.E. The Unity of the Senses: Interrelationships Among the Modalities. New York: Academic Press, 1978. 289 p.
32. Martín-Loeches M., Hinojosa J.A., Fernández-Frías C., Rubia F.J. Functional differences in the semantic processing of concrete and abstract words. Neuropsychologia, 2001. Vol. 39, no. 10, pp. 1086—1096. DOI:10.1016/S0028-3932(01)00033-1
33. Martino G., Marks L.E. Synesthesia: Strong and Weak. Current Directions in Psychological Science, 2001. Vol. 10, no. 2, pp. 61—65. DOI:10.1111/1467-8721.00116
34. Maurer D. Neonatal Synesthesia: Implications for the Processing of Speech And Faces. Developmental Neurocognition: Speech and Face Processing in the First Year of Life. Dordrecht: Springer Netherlands, 1993. P. 109—124. DOI:10.1007/978-94-015-8234-6_10
35. Meltzoff A.N., Borton R.W. Intermodal matching by human neonates. Nature, 1979. Vol. 282, no. 5737, pp. 403—404. DOI:10.1038/282403a0
36. Polich J., Donchin E. P300 and the word frequency effect. Electroencephalography and Clinical Neurophysiology, 1988. Vol. 70, no. 1, pp. 33—45. DOI:10.1016/0013-4694(88)90192-7
37. Ramachandran V.S. A Brief Tour of Human Consciousness: From Impostor Poodles to Purple Numbers. New York: Pi Press, 2004. 208 p.
38. Sagiv N., Ilbeigi A., Ben-Tal O. Reflections on synaesthesia, perception, and cognition. Intellectica, 2011. Vol. 55, no. 1, pp. 81—94.
39. Spence C. Crossmodal correspondences: A tutorial review. Attention, Perception, and Psychophysics, 2011. Vol. 73, no. 4, pp. 971—995. DOI:10.3758/s13414-010-0073-7
40. The Oxford Handbook of Synesthesia. In J. Simner, E. Hubbard (ed.), Oxford: Oxford University Press, 2013. 1104 p. DOI:10.1093/oxfordhb/9780199603329.001.0001
41. Woltz D.J. Perceptual and conceptual priming in a semantic reprocessing task. Memory and Cognition, 1996. Vol. 24, no. 4, pp. 429—440. DOI:10.3758/BF03200932