Use of Evidence-Based Evaluation Criteria for Inclusive Practices in Competitive Procedures

The work presents the results of approbation of the approach to expert evaluation of inclusive practices in educational institutions developed at the Institute for Problems of Inclusive Education of the Moscow State University of Psychology and Education based on the integration of inclusiveness criteria and evidence requirements. The assessment involved 25 federal experts on inclusive education, who evaluated 30 cases in three nominations of inclusive practices submitted to the federal stage of the competition. The assessors scored the applications by completing expert protocols, which were processed to determine the level of inclusivity of the practice (zero, initial, basic, or advanced). Then a reflective discussion was organized on the application by experts of the approach we developed to assessing inclusive practices according to the criteria of inclusiveness and evidence, the results of this discussion were subjected to thematic analysis. The study revealed that the distribution of expert assessments of inclusive practices by levels differs from uniform (p < 0,01), confirming the effectiveness of the criteria as an assessment tool. At the same time, the reproducibility of the results of the expert assessment turned out to be at an acceptable level (83%), taking into account the unreliability of differentiation of the zero and initial levels (sample of repeated measurements: N = 12). The thematic analysis of expert reflection showed the usefulness of evaluation criteria as a tool not only for examination, but also for the development of expert thinking of the experts themselves, as well as the professional development of the contestants. Also, the results of the thematic analysis revealed the need for a broad discussion and further study of the key categories of inclusion (diversity, participation, acceptance, etc.) to operationalize them more accurately as inclusion criteria.

Introduction

Since inclusive education (IE) became a requirement of Russian federal legislation (following the ratification of the UN Convention on the Rights of Persons with Disabilities in 2012), professionals are looking for new stimulos  to wider implementIE in educational institutions (EO) [14]. Contests among Russian educational organizations and nominations for the best inclusive practices (IP) are beginning to have an impact on the development of inclusive education.

In this study, we usedthe operational definition of the Inclusive educational practices (IEP) adopted at the Institute for Problems of Inclusive Education of the Moscow State University of Psychology and Education. IEP  is a set of interrelated organizational, administrative, pedagogical and social actions aimed to widly support an active participation of all parties of the educational process by providing a diversity of educational needs and the inclusion of all students in the educational process. We can also discuss the concept of the inclusive practices since this article refers to inclusion as implemented and assessed in contestant procedures not only in public organizations, but alsoin recreational institutions for children, where it may not be so much about educational inclusion as about social inclusion. At a preliminary level, such a definition will practically repeat what we stated above, except that instead of inclusion in the educational process, we will be talking about involvement students as active participants in any activity relevant to the given conditions. Therefore, further in the text we mainly use the abbreviation IP.

IPs are components of the inclusive educational environment (IEE) of the organization. There are several approaches to increasing the evidence of assessment procedures in social work and education that take into account both the objective and subjective components of the achieved results, which also include assessment of the beneficiaries’ satisfaction  ( in the case of basic education - parents and their children studying in public educational institutions). Expert assessment which requires developing more equal and detailed criteria to help experts evaluate IP in the process of holding contests [16] occupies a significant place among assessment methods..

It should be taken into account that when we apply an evidence-based approach in the field of assessing the effectiveness of educational institutions, a number of specific problems arise. These problems have become the subject of discussion within the international community [24; 29; 30; 32; 33]. One of them is the difficulty to conduct controlled randomized studies due to the wide variety of special educational needs (SEN) of students with disabilities [24], insufficient consideration of the specifics of SEN in students with severe and multiple disabilities or severe intellectual disability [30] etc.

Discussion goes on relating to methods for determining evidence-based practices in domestic and foreign literature, such as discussion of methodology issues. One of the most important issues of discussion is the use of qualitative methods to substantiate the effectiveness of practices [7; 8; 9; 10; 26; 27; 31; 34; 35; 36; 37; 38].

The Institute for Problems of Inclusive Education has developed criteria[2; 3; 4; 5]  for assessing IP in general educational institutions, that are based on a system analysis, taking into account scientific literature, integrated with the requirements of an evidence-based approach in assessing social practices [7; 8; 9; 10]. The criteria were tested and applied in the expert assessment of applications recieved from participants of the  contest “Best Inclusive School of Russia”.

The goal of this article is to demonstrate the results of this testing, including the reflection of the experts who applied these criteria as their application seems productive and what problems arise.

The article puts forward a hypothesis,thatthe expert assessment will reveal differences in inclusiveness between the assessed inclusive practices used in the environmental conditions of kindergartens, schools, recreational institutions for children, etc. and that the assessment of various experts, who used the proposed criteria, will not differ significantly. Apart from that, it has been assumed that the reflection  carried out by the experts would reveal a number of proposals relating to the contest procedure and the evaluation criteria that were used by the experts.

Research methodology

The developers of the procedure for expert assessment of inclusive practices based on an evidence-based approach are the team of authors of the Institute for Problems of Inclusive Education, Moscow State University of Psychology and Education.

The development of the expert assessment procedure has been carried out in several stages.

1. The expert survey helped to identify the key principles of inclusive education [2; 4].
2. The criteria for assessing of the inclusive practices were developed according to an evidence-based approach and key principles of inclusive education [2; 3; 4].
3. The analysis of the evidence standard made it possible to identify the main parameters of inclusive educational practices, which were included in the requirements for describing the best inclusive practices within the framework of the contest “Best Inclusive School of Russia”. The description of a successful IP with an evidence-based approach involved the presentation of a practical case.
4. An expert protocol was developed as a tool for assessing IP in the form of the table that includes 6 criteria for the inclusiveness of practices (acceptance, participation, accessibility, variability, adaptability, support and individual approach) and 4 indicators of evidence (regularity, validity of data, achievement of educational results, validity of data on the educational results of the practices), grouped into three levels of practice – initial (1–3), basic (4–7), advanced (8–10).
5. 30 cases from 3 nominations came to the federal stage of the call for applications and were evaluated by 25 federal experts. They have been examined in order to test the expert protocol 
6. The data from expert protocols and a video recording of a webinar with the experts, which reflected their opinions on the assessment procedure and criteria as well as  their proposals for improving the assessment procedure and tool, including the expert protocol, were considered  as a result of the assessment of cases by the experts. Some experts also provided their thoughts and suggestions in written from.
7. The data obtained from the expert protocols, as well as the results of the experts’ reflection, were processed. The arithmetic mean was calculated for each expert protocol based on the points assigned by the expert and the number of headings of the completed protocol. the IP level was determined Based on this average: zero, if the average was less than 1, initial - from 1 to less than 4, basic - from 4 to less than 8, advanced - from 8 to 10. Thus another category for evaluation of competitive bids was added, i.e. a zero level, since many experts not only spoke in favor of its introduction at the reflective webinar, but also used it in practice during the assessment process. Following that we have tested a statistical hypothesis about the difference in the frequencies of competitive bids falling into various categories (zero, initial, basic, and advanced levels). The results of the reflection were subjected to thematic analysis. A one-sample chi-square goodness-of-fit criterion was applied as a method of statistical processing of the distribution by category of competitive bids, which helped to verify the difference between the empirical distribution and the uniform distribution.
8. The reflective approach was used as a thematic analysis methodology, which was considered a form of systemic theoretical activity aimed at creating and understanding the social product, its actions and their laws based on G.P. Shchedrovitsky and his followers’ theory [1; 19; 20; 21]. The reflective approach involved experts analyzing their first experience of working with protocols developed on the basis of criteria for evidence of inclusive practices while evaluating competitive works. Particularly valuable for us was the reflection on the understanding of the criteria and indicators developed by various experts, who are experienced teachers in the field of inclusive education, in relation to various competitive works. This way we have implemented a cycle of the experts´ reflective activity: analysis and evaluation of competitive works (control); criticism (correction of criteria and evaluation protocol); proposals for regulation of the criteria. The reflective approach not only allows to organize a discussion based on complex communication, highlight expert opinions about the strengths and weaknesses of the assessment tool, but also helps design changes within the assessment tool [1; 16; 19; 20; 21].

Selection. The assessment involved 25 federal experts on inclusive education, selected according to the criteria of having at least 5 years of experience in inclusive education, a scientific degree and publications on the issues of inclusive education.

30 cases from three categories of inclusive practices (“kindergarten”, “school”, “health and recreational intuitions for children”) submitted to the federal stage of the contest of applications and  offered for expert evaluation have been carefully reviewed.

Results

Table 1 refers to the results of the expert scoring of competitive bids (inclusive practices presented in three nominations of the contest).

Table 1 Expert evaluation of competitive inclusive practices

Note: zero level – the average expert assessment for the case is less than 1; entry level – 1–4; basic level – 4–8; advanced level – 8–10.

The presented results show that only 3 out of 12 paired expert assessments differ from each other in the level to which competitive bids are distributed. These are inclusive practices, the assessments of which are presented in the 2nd, 15th and 22nd rows of the table, which is 25 %   of all cases that passed double examination. Accordingly, in 75% of cases, the experts’ assessments in the distribution of cases by level coincided. Moreover, the discrepancy in line 15 may be caused by the fact that initially the experts were asked to divide practices into three levels, i.e. without zero, since the criterion for differentiating between zero and initial levels is unreliable. Taking this into account, from our point of view, the percentage of matches turned out to be quite good (i.e. 10 out of 12, or 83%), which is an argument in favor of the reproducibility of expert assessments, although the small selection would not allow us to evaluate this reproducibility quantitatively. This may indicate that experts understand the assessment criteria in a similar way, and that the subjective factor has no undue influence on the assessment results in this selection of experts.

Table 2 demonstrates the results of a frequency analysis of expert assessments by level, including the zero level we introduced post factum, since it was actually used by the experts themselves.

Table 2 Results of the expert classification of competitive inclusive practices by level

Note: Pearson chi-square goodness-of-fit test (goodness-of-fit with uniform distribution). Result: χ 2 Emp =19.332, the null hypothesis is rejected at p< 0.01, df =3 (χ 2 Crit =11.345).

The data and their statistical analysis referred to in Table 2 show that the frequencies of cases assessed as meeting the zero and initial levels greatly predominate in the competitive selection- 27% and 57%, respectively. At the same time, experts classified only 17% in total as basic and advanced levels. These results clearly demonstrate that the vast majority of practices, according to experts, poorly meet the inclusiveness criteria and standards of evidence used.

Following that we have carried out a thematic analysis of the results of the experts’ reflection transformed into written or initially represented in the form of the text, and identified on its basis three categories of expert opinions and proposals regarding (1) the usefulness (beneficial effect) of the assessment tool, (2) changes in the procedure for interacting with contestants, and (3) changes to the assessment tool itself (criteria for inclusivity and evidence, which includes procedure and expert´s protocol).

Table 3 represents expert opinions regarding the usefulness (beneficial effect) of the assessment tool used.

Table 3 The effect (usefulness) of the criteria for experts and contestants, according to experts

The results represented in the Table 3 demonstrate that experts find the tool they used in order to evaluate competitive practices based on the criteria of inclusivity and evidence very useful both for themselves and, potentially, for the contestants. A fairly common and important suggestion is that the assessment tool should be used during the preparatory stage of a contest to create criteria-based recommendations to familiarize contestants with its requirements and following that bring practices into contest format as far as the requirements for inclusivity and evidence are concerned. We have also discussed the usefulness of criteria for identifying the strengths and weaknesses of practices, existing deficiencies both in their implementation and in the presentation of results achieved on their basis, the requirement to recognize the risks of the practices, etc.

Experts made proposals regarding specific changes in the requirements for the contestants:

1. Introduce a mandatory requirement to describe inclusivity as a result of the practice and means to confirm the result, including based on substantive criteria, and not just formal quantitative ones.
2. Draw up methodological recommendations for the contestants and/or a checklist for preparing a description of the IP for the contests, which would include recommendations on methods for justifying the effects of the IP.
3. Set contestants specific requirements for regulations on describing IP.

The experts' recommendations refer to desirable changes in the requirements for contestants. To a large extent, they come down to the two main ones. The first generally recommends the creation of more detailed regulations for describing practice, which can be presented in the form of recommendations and/or, for example, a checklist for the contestant, and the second refers to the content of these requirements, which must necessarily include a description of the inclusiveness of the practice and, accordingly, what the contestant considers inclusion, recommended evidence-based procedures/methods, description of goals and objectives, as well as the results of practice and whether and how the task of ensuring inclusion has been solved and how it is confirmed.

Experts made proposals for changes to the criteria and assessment protocol:

1. Detail the criteria within the levels.
2. Eliminate duplication in criteria.
3. Enter a zero IP level into the assessment protocol.
4. Enter a comment column into the evaluation protocol.
5. Create a glossary of criteria for experts.
6. Expand the result of practice as an object of assessment: introduce other types of results in addition to the educational ones.
7. Clarify the characteristics of each level.
8. Clarify the wording (acceptance, participation, support, individualization etc.) and a set of criteria.

The criteria for evaluating practices themselves generated the greatest number of comments and suggestions, but there are also important proposals regarding the examination procedure and protocol. That said a number of experts proposed introducing a comment column for the expert into the protocol, as well as expanding the number of assessed levels by adding a zero level. Moreover, there is a recommendation to clarify and detail the criteria for assignment to various levels and scaling within them, in particular by adding an indicator of practice reproducibility for the advanced level, and a requirement for internal professional expertise for the basic level. Experts point out the desirability of having a dictionary (glossary) that explains the main parameters of evidence and the terminology used in the criteria. It has also been proposed to expand the concept of result beyond the educational one (for example, social inclusion etc.) , due to the presence of different nominations (for example, holiday camps, kindergartens). Experts highlighted the presence of duplication in the criteria, which may unjustifiably inflate or underestimate the score. The key criteria for assessing inclusivity, such as participation and acceptance, received particular number of comments, while other names have also been proposed for these criteria, which, according to the expert, clarify their content and eliminate ambiguity.

The discussion aboutthe results

The results obtained, from our point of view, need to be discussed in several interrelated aspects. The criteria become the subject of reflective discussion in the community of experts, and thereby develop expert thinking in the field of inclusion; it becomes the object of change by experts and a means of self-change in the thinking of experts. This reflection as a process of self-change in expert thinking in the field of inclusion is not something accidental: it is the result of the presence of disputes and disagreements in international practice regarding the very idea of inclusion [22; 23; 24; 25; 26; 29; 32; 33], which are not removed even by the efforts of the UN Committee on the Rights of Persons with Disabilities, which issued General Comment No. 4 on the right to inclusive education [17].

A key point of discussion was the interpretation of what the “participation” criterion means. whether it consists of the realization of the right of everyone to participate in general educational conditions through reasonable adaptation of these conditions, however with the main emphasis on the implementation of this right, or whether it is in deciding the issue of placement of students based on the balance in the implementation of this right along with other – the right to receive the highest quality education, i.e. the one that meets the needs of students with special education needs in the least restrictive environment, without however compromising the exercise of their last right? T.O. Archakov and E.Sh. Garifulina further developed the matter of in their research paper by means of introducing into the context of this problem the issue of taking into account the opinion of the children when making important decisions affecting their interests, which is a requirement of the UN Convention on the Rights of the Child (Article 12) [6; 15]. The need to take into account the views of children when making important decisions poses many complex problems, such as finding a balance between the objective developmental needs of children and achieving an acceptable level of their subjective well-being, developing their competence in expressing their opinions and attention to their subjective preferences, building partnership with them and accepting social and cultural norms that are important to the community they enter as they grow up.

A similar duality exists regarding the concepts of individual approach and individualization, where the question has been raised about supporting the subjectivity of each student on the one hand, and about an individual approach to his training and education, where he is not the subject, but the recipient of educational influences on the other hand [13; 17].

These are just a few aspects of the complex problems associated with the implementation of expert reflection in the assessment of inclusive practices.

Conclusion

As we see, the results of testing the criteria for expert assessments of inclusiveness and evidence-based educational and social practices, expressed in the results of expert assessments and reflection on the experience of using the assessment tool and its criteria, obtained in the work, prove that the tool is productivem the assessments are reproducible and  suitabile  for the validation of the IP expertise and for the professional development of contestants. The hypotheses posed in the article have been confirmed. The peer review tool has demonstrated that it can be used by experts to differentiate competitive practices according to the levels of inclusivity and evidence. The reproducibility of expert assessments can be defined as acceptable, although it doesn't have quantitative characteristics of reproducibility yet. The results of expert reflection revealed a number of important issues for further development and specification of both the basic concepts underlying the assessment tool and its parameters and procedures.

First of all, the experts confirmed the importance and usefulness of the developed IP assessment criteria, not only for examination, but also, as a potential tool for the professional development of the contestants.

In the second place, the results of reflection have shown that a number of key categories for the concept of inclusion, such as diversity, participation, acceptance  retain a significant amount of uncertainty for domestic experts, and therefore for practicing specialists in general (school teachers, educators, psychologists, speech pathologists, etc.).

This means that they not only need situational clarification, for example, in the proposed examination criteria, but they also need further broad discussion and empirical verification in the scientific literature and at scientific and practical conferences that's why they have to beoperationalized and reliable verification methods should be chosen.

1. Alekseev N.G. Printsipy i kriterii ekspertizy programm razvitiya obrazovaniya [Principles and criteria for the examination of educational development programs]. Voprosy metodologii =  Voprosy metodologii, 1994, no. 2, pp. 59–68. (In Russ.).
2. Alekhina S.V., Mel'nik Yu.V., Samsonova E.V., Shemanov A.Yu. K voprosu otsenki inklyuzivnogo protsessa v obrazovatel'noi organizatsii: pilotazhnoe issledovanie [On the issue of assessing the inclusive process in an educational organization: a pilot study]. Psikhologo-pedagogicheskie issledovaniya = Psychological and pedagogical research, 2019. Vol. 11, no. 4, pp. 121–132. DOI:10.17759/psyedu.2019110410 (In Russ., abstr. in Engl.).
3. Alekhina S.V., Mel'nik Yu.V., Samsonova E.V., Shemanov A.Yu. Otsenka inklyuzivnogo protsessa kak instrument proektirovaniya inklyuzii v obrazovatel'noi organizatsii [Evaluation of the inclusive process as a tool for designing inclusion in an educational organization]. Psikhologicheskaya nauka i obrazovanie = Psychological Science and Education, 2021. Vol. 26, no. 5, pp. 116–126. DOI:10.17759/pse.2021260509 (In Russ., abstr. in Engl.).
4. Alekhina S.V., Mel'nik Yu.V., Samsonova E.V., Shemanov A.Yu. Ekspertnaya otsenka parametrov inklyuzivnogo protsessa v obrazovanii [Expert assessment of the parameters of the inclusive process in education]. Klinicheskaya i spetsial'naya psikhologiya = Clinical and Special Psychology, 2020. Vol. 9, no. 2, pp. 62–78. DOI:10.17759/cpse.2020090203 (In Russ., abstr. in Engl.).
5. Alekhina S.V., Samsonova E.V., Shemanov A.Yu. Podkhod k modelirovaniyu inklyuzivnoi sredy obrazovatel'noi organizatsii [Approach to modeling the inclusive environment of an educational organization]. Psikhologicheskaya nauka i obrazovanie = Psychological science and education, 2022. Vol. 27, no. 5, pp. 69–84. DOI:10.17759/pse.2022270506 (In Russ., abstr. in Engl.).
6. Archakova T.O., Garifulina E.Sh. Uchastie detei v Rossii: teoreticheskoe osmyslenie i razvitie praktiki [Participation of children in Russia: theoretical understanding and development of practice]. Sotsial'nye nauki i detstvo = Social sciences and childhood, 2020. Vol. 1, no. 1, pp. 68–87. DOI:10.17759/ssc.2020010106 (In Russ., abstr. in Engl.).
7. Busygina N.P., Gorobtsova A.V. Kachestvennaya metodologiya i dokazatel'nye praktiki v psikhologii i obrazovanii [Qualitative methodology and evidence-based practices in psychology and education]. Psikhologicheskaya nauka i obrazovanie = Psychological science and education, 2021. Vol. 26, no. 6, pp. 117–127. DOI:10.17759/pse.2021260609 (In Russ., abstr. in Engl.).
8. Busygina N.P., Podushkina T.G., Zevina D.A. Metodologicheskie problemy dokazatel'nogo opisaniya psikhologicheskikh i sotsial'nykh praktik [Methodological problems of evidence-based description of psychological and social practices]. Sotsial'nye nauki i detstvo = Social sciences and childhood, 2021. Vol. 2, no. 1, pp. 24–36. DOI: 10.17759/ssc.2021020102 (In Russ., abstr. in Engl.).
9. Busygina N.P., Podushkina T.G., Stanilevskii V.V. Dokazatel'nyi podkhod v obrazovanii: kriticheskii analiz aktual'nykh diskussii [Evidence-based approach in education: a critical analysis of current discussions]. Psikhologo-pedagogicheskie issledovaniya = Psychological and pedagogical research, 2021. Vol. 13, no. 4, pp. 162–176. DOI:10.17759/psyedu.202113-410 (In Russ., abstr. in Engl.).
10. Busygina N.P., Podushkina T.G., Stanilevskii V.V. Dokazatel'nyi podkhod v sotsial'noi sfere: osnovnye ponyatiya i printsipy, istoriya, perspektivy [Evidence-based approach in the social sphere: basic concepts and principles, history, perspectives]. Sotsial'nye nauki i detstvo = Social sciences and childhood, 2020. Vol. 1, no. 1, pp. 8–26. DOI:10.17759/ssc.2020010101 (In Russ., abstr. in Engl.).
11. Gutsykova S.V. Metod ekspertnykh otsenok: teoriya i praktika [Method of expert assessments: theory and practice]. Moscow: Institut psikhologii RAN, 2011. 144 p. (In Russ.).
12. Zaretskii V.K. U istokov refleksivno-deyatel'nostnogo podkhoda: k 120-letiyu Petra Yakovlevicha Gal'perina, k 90-letiyu Nikity Glebovicha Alekseeva [At the origins of the reflexive-activity approach: to the 120th anniversary of Pyotr Yakovlevich Galperin, to the 90th anniversary of Nikita Glebovich Alekseev]. Konsul'tativnaya psikhologiya i psikhoterapiya = Counseling Psychology and Psychotherapy, 2022. Vol. 30, no. 4, pp. 11–27. DOI:10.17759/cpp.2022300402 (In Russ., abstr. in Engl.).
13. Kovaleva T.M. Oformlenie novoi professii t'yutora v rossiiskom obrazovanii [Formation of a new profession of a tutor in Russian education]. Voprosy obrazovaniya = Educational Issues, 2011, no. 2, pp. 163–181. (In Russ.).
14. Konventsiya OON o pravakh invalidov: prinyata rezolyutsiei 61/106 General'noi Assamblei ot 13 dekabrya 2006 g. (ratificirovana Federal'nym zakonom ot 3 maja 2012 g. № 46-FZ) [UN Convention on the Rights of Persons with Disabilities: adopted by General Assembly resolution 61/106 of December 13, 2006 (ratified by the Federal Law of May 3, 2012 No. 46-FZ)]. [Elektronnyi resurs]. Available at: https://www.un.org/ru/documents/decl_conv/conventions/disability.shtml; https://base.garant.ru/70170066/  (Accessed 03.03.2023). (In Russ.).
15. Konvencija OON o pravah rebenka: prinjata rezoljuciej 44/25 General'noj Assamblei ot 20 nojabrja 1989 g. [UN Convention on the Rights of the Child: adopted by General Assembly resolution 44/25 of November 20, 1989] [Eelektronnyj resurs]. Available at: https://www.un.org/ru/documents/decl_conv/conventions/childcon.shtml (Accessed 03.03.2023). (In Russ.).
16. Leont'ev D.A., Ivanchenko G.V. Kompleksnaya gumanitarnaya ekspertiza. Metodologiya i smysl [Comprehensive humanitarian expertise. Methodology and meaning]. Moscow: Smysl, 2008. 133 p. (In Russ.).
17. OON: Zamechanie obshchego poryadka № 4 (2016) o prave na inklyuzivnoe obrazovanie [UN: General comment No. 4 (2016) on the right to inclusive education]. Komitet OON po pravam invalidov [Elektronnyi resurs]. Available at: https://tbinternet.ohchr.org/_layouts/15/treatybodyexternal/Download.aspx?symbolno=CRPD/C/GC/4&Lang=en (Accessed 03.03.2023). (In Russ.).
18. Samsonova E.V., Bystrova Yu.A., Shemanov A.Yu., Kutepova E.N. Kompetentsii t'yutora v inklyuzivnom obrazovanii: spetsifika programm professional'noi podgotovki [Tutor competencies in inclusive education: the specifics of professional training programs]. Psikhologo-pedagogicheskie issledovaniya = Psychological and pedagogical research, 2022. Vol. 14, no. 2, pp. 84–99. DOI:10.17759/psyedu.2022140206 (In Russ., abstr. in Engl.).
19. Chupina V.A. Refleksivnyi metod i ego rol' v razvitii professional'nogo myshleniya upravlencheskikh kadrov [Reflexive method and its role in the development of professional thinking of managerial personnel]. Obrazovanie i nauka = Obrazovanie i nauka, 2010, no. 11 (79), pp. 12–22. (In Russ.).
20. Shchedrovitskii G.P. Problemy metodologii sistemnogo issledovaniya [Problems of system research methodology]. Moscow: Znanie, 1964. 489 p. (In Russ.).
21. Shchedrovitskii G.P. Refleksiya i ee problemy [Reflection and its problems]. Refleksivnye protsessy i upravlenie = Reflexive processes and management, 2001. Vol. 1, no. 1, pp. 16–28. (In Russ.).
22. Anastasiou D., Felder M., De Miranda Correia L.A., Shemanov A., Zweers I., Ahrbeck B. Chapter 11. The impact of article 24 of the CRPD on special and inclusive education in Germany, Portugal, the Russian federation, and the Netherlands. On Educational Inclusion: Meanings, History, Issues, and International Perspectives. J.M. Kauffman (ed.). Vol. I. Connecting Research with Practice in Special and Inclusive Education. Series edited by Philip Garner. Routledge, London, New York, 2020. P. 216–248.
23. Annamma S.A., Ferri B.A., Connor D.J. Disability critical race theory: Exploring the intersectional lineage, emergence, and potential futures of DisCrit in education. Review of Research in Education, 2018. Vol. 42, no. 1, pp. 46–71.
24. Boyle C., Koutsouris G., Mateu A.S., Anderson J. The matter of ‘evidence’ in the inclusive education debate. Oxford Research Encyclopedia of Education. Oxford University Press. USA, 2020. DOI:10.1093/acrefore/9780190264093.013.1019
25. Connor D.J., Ferri B.A. The conflict within: resistance to inclusion and other paradoxes in special education. Disability and Society, 2007. Vol. 22, no. 1, pp. 63–77. DOI:10.1080/09687590601056717
26. De Weger E., Vooren N.J.E. van, Wong G., Dalkin S., Marchal B., Drewes H.W., Baan C.A. What’s in a Realist Configuration? Deciding Which Causal Configurations to Use, How, and Why. International Journal of Qualitative Methods, 2020. Vol. 19, pp. 1–8. DOI:10.1177/1609406920938577
27. Francis-Auton E., Sarkies M.N., Pomare C., Long J.C., Hardwick R., Nguyen H.M., Braithwaite J. Real Talk: A Realist Dialogic Approach in a Realist Evaluation. International Journal of Qualitative Method, 2022. Vol. 21. DOI:10.1177/16094069221120748
28. Gordon-Gould P., Hornby G. The progress of inclusion and the elephant in the classroom. Inclusive Education at the Crossroads: Exploring Effective Special Needs Provision in Global Contexts. P. Gordon-Gould, G. Hornby (eds.). Routledge. London & New York: Taylor & Francis Group, 2023. P. 1–9. DOI: 10.4324/9781003262701-1
29. Hornby G. Are Inclusive Education or Special Education Programs More Likely to Result in Inclusion Post-School? Educ. Sci, 2021. Vol. 11, 304.  DOI:10.3390/educsci11060304
30. Imray P., Kossyvaki L., Sissons M. Evidence-based practice: the use and abuse of research. Support for Learning, 2023. Vol. 38, no. 1, pp. 51–66. DOI:10.1111/1467-9604.12438
31. Jess K., Petersén A.C. From “What Works” to “Why it Works” – From a Humean to an Aristotelian Understanding of Causality. Research on Social Work Practice, 2023, pp. 1–9. DOI:10.1177/10497315231154493
32. Kauffman J.M., Badar J. Definitions and other issues. On Educational Inclusion: Meanings, History, Issues, and International Perspectives. J.M. Kauffman (ed.). Vol. I. Connecting Research with Practice in Special and Inclusive Education. Series edited by Philip Garner. Routledge, London, New York, 2020. P. 1–24.
33. Kauffman J.M., Burke M.D., Anastasiou D. Hard LRE Choices in the Era of Inclusion: Rights and Their Implications. Journal of Disability Policy Studies, 2022, pp. 1–12. DOI:10.l177/10442073221113074
34. McDuffie K.A., Scruggs T.E. The Contributions of Qualitative Research to Discussions of Evidence-Based Practice in Special Education. Interventions in School and Clinic, 2008. Vol. 44, no. 2, pp. 91–97. DOI:10.1177/1053451208321564
35. Mukumbang F.C., Marchal B., Belle S. van, Wyk B. van. Using the realist interview approach to maintain theoretical awareness in realist studies. Qualitative Research, 2020. Vol. 20, no. 4, pp. 485–515. DOI:10.1177/1468794119881985
36. Rutten R. Applying and Assessing Large-N QCA: Causality and Robustness from a Critical Realist Perspective. Sociological Methods & Research, 2022. Vol. 51, no. 3, pp. 1211–1243.  DOI:10.1177/0049124120914955
37. Tellings A. Evidence-Based Practice in the social sciences? A scale of causality, interventions, and possibilities for scientific proof. Theory & Psychology, 2017. Vol. 27, no. 5, pp. 581–599.  DOI:10.1177/0959354317726876
38. Watharow A., Wayland S. Making Qualitative Research Inclusive: Methodological Insights in Disability Research. International Journal of Qualitative Method, 2022. Vol. 21.  DOI:10.1177/16094069221095316