Psychometry of Latent Characteristics: Tools, Problems of Rank Information Processing, Solutions

The paper examines the problems of correctness and adequacy of the use of psychometric tools in psychosocial research and applications. The main attention is paid to the problem of correct processing of rank information, which is non-numeric information that does not allow the use of classical mathematical operations, starting with the addition operation. The approach of forming the results of processing expert, ranked information in the scale of relations based on the Analytic Hierarchy Process (AHP), which includes a measurement apparatus and decision-making algorithms, is presented. An example is given that simulates the situation of an expert council and demonstrates unacceptable contradictions that arise with the "numerical" approach to processing rank information. The algorithms of the AHP normative approach are presented, which allow obtaining numerical, personalized assessments of psychosocial characteristics. The basic concepts of the Rasch Measurement (RM) for comparing the capabilities of AHP and RM in psychosocial applications are described. Some critical remarks about the methods of psychometry are considered. It is shown that the origins of the criticisms are associated with a superficial understanding of the basic concepts of psychometric measurements and the scope of their applications, as well as with elementary errors when using the apparatus of psychometric instruments.

Introduction

In psychology, psychiatry, neurology, pedagogy, and sociology, psychometric ranking instruments are used to measure and assess the level of expression of latent characteristics of a person (patient). These characteristics include intellectual abilities, neurocognitive functions, aggression, depression, anxiety, stigma, interpersonal relationships, and other personality traits.

At present, two main directions can be distinguished in the use of psychometric instruments, which we rely on to illustrate and solve the tasks set within this work.

1. Traditional direction

Information obtained from ranking psychometric scales, tests, and questionnaires is considered (without any additional conditions and considerations, particularly probabilistic in nature) as numerical information that can be processed by any mathematical means.

Within such a numerical approach to ranking information, researchers in their works operate with such impermissible characteristics as the sum of points and average points on the items of diagnostic scales and subscales, variance and standard deviations, errors of average values, etc. Such examples can be found when reviewing articles in almost any journal in the field of psychosocial research using ranking scales. Numerous references can be found in work [20] and as fresh examples [6; 7; 19; 23].

The inadmissibility of using classical mathematical operations within a ranking (ordinal) scale has been known for more than 50 years, starting from the foundational works of J. Pfanzagl [25]. This fact is described in foreign and domestic manuals and textbooks on the theory of scales and methods of mathematical data processing (see, for example, [8; 14; 18; 24]).

2. Modern direction

1) Within this direction, researchers, knowing the inadmissibility of processing ranking information by classical mathematical methods, propose using metric measurement systems. Such systems include, in particular, the Analytic Hierarchy Process (AHP), which includes measurement tools and decision-making algorithms [2; 5; 10-13; 15; 22; 28-30], as well as the Rasch measurement system (RMS) [1; 3; 4; 21; 26; 27]. Note that in our work, the classical version of RMS is discussed, as modern criticism of psychometric instruments, presented in article [17], is associated with this model. The classical version of RMS is also incorrectly used within the Russian Unified State Exam (USE) system (see, for example, [9]), which is one of the reasons for the emergence of criticism of psychometric instruments.

2) In recent years, there have been works discussing critical arguments regarding psychometry. The main message of these arguments is that psychometry cannot be considered a tool for measuring latent characteristics of a person at all (e.g., [17; 31]).

It is argued that the idea of psychometry as a means of obtaining assessments of a person's psychosocial characteristics is based on fuzzy values of basic terms, which are often confused, as well as erroneous assumptions about how measurements can be implemented when studying the psyche.

Research problem

Thus, there is a problem of assessing the correctness and adequacy of the application of psychometric instruments in psychosocial research. In this work, the main focus is on the possibilities and prospects of using AHP in processing ranking information obtained using reliable psychometric instruments. Comparing the capabilities of AHP and RMS methods requires a separate study, but some remarks on the features of using RMS, as well as on the issue of criticism of psychometric instruments, will be made after outlining the possibilities of applying AHP to solve psychometry problems.

Materials and methods

The work uses materials from Russian and foreign scientific publications, as well as the results of our own research devoted to the use and processing of heterogeneous data presented in different scales, from categorical to ratio scales.

Data analysis methods: statistical, ranking methods, AHP method. All calculations based on the proposed algorithms were performed in the MS Excel environment.

Results and discussion

1.  Incorrectness of the numerical approach to processing ranking data in psychometry

The reason for the continuing practice of the numerical approach to processing ranking information lies in the abstract nature of the methods of measurement theory and scales used to prove, for example, the inadmissibility of using such operations as summation and calculating the arithmetic mean in ranking scales. However, at a qualitative level of understanding of ranking information, it is clear that the sum of knowledge of a failing student (2 points) and an average student (3 points) is not equal to the level of knowledge of an excellent student (5 points).

The main reason for such facts in psychosocial research is the unevenness of the labels of ordinal (ranking) scales, for which only ordering operations can be used: equality, greater, lesser, or monotonic transformation without violating the original order (e.g., [14; 18]).

Consider a simple example modeling the situation of an expert council and demonstrating the unacceptable contradictions arising from the "numerical" approach to processing ranking information. Table 1 presents the results of expert evaluation of the condition of two patients, P1 and P2, obtained from 11 specialists of equal qualification who used a 7-point ranking scale (the levels of this scale are from 1 to 7, ordered by increasing severity of the symptom sign).

Table 1. Expert assessments of the condition of patients P1 and P2 on a 7-point scale

А. Obviously, based on the principle of absolute majority, the assessment of the condition of patient P1 is 6 points, and for patient P2, respectively, 5 points. This principle, which in decision theory allows avoiding paradoxes that exist in various selection systems. An additional argument is the assessments from descriptive statistics: the median and mode, which coincide for P1 (6 points) and for P2 (5 points).

В. If we proceed from the "numerical" approach to the ranked data in Table 1, then for the average assessments (C1 and C2) of the condition of P1 and P2, we get (with an accuracy of up to 0.01):

С1 = (8*6 + 3*3) / 11 = 5,18; С2 = (9*5 + 2*6) / 11 = 5,18.

Taking into account the standard error of the mean calculation, we get:

С1 = 5,18 ± 0,42; С2 = 5,18 ± 0,12

The obtained result leads at best to an uncertain situation and, at worst, to an opposite conclusion compared to point A. A logical way out of this situation is to recognize the fact that the ranked data from Table 1 do not allow the calculation of the arithmetic mean.

An additional confirmation of this conclusion is the results of a comprehensive clinical study [3], which revealed that the criteria for assessing the severity of depression on the Hamilton Rating Scale for Depression (HRSD, Hamilton), developed based on ranked data within the "numerical" approach, do not agree with clinical data.

Commentary on the example.

Incorrect ("numerical") processing of ranked information, as follows from the given example, leads to uncertain situations or errors in assessing the severity of disorders, which obviously negatively affects the effectiveness of relevant psychosocial studies and interventions.

The given simple example is further used to demonstrate correct AHP algorithms in processing ranked information.

2. Processing Psychometric Ranking Data Based on the Analytic Hierarchy Process Method

2.1 Basic Information about the Analytic Hierarchy Process Method

In modern decision theory for multi-criteria, weakly structured problems (it is evident that major psychosocial problems fall into this category), the most natural and effective method for processing heterogeneous information is the Analytic Hierarchy Process (AHP). The author of the method, who is over 50 years old, is considered to be the American mathematician T. Saaty (T.L. Saaty), but it should be noted that domestic authors (B.N. Brook and V.N. Burkov [5]) proposed a method for processing ranked information (earlier than T.L. Saaty's works), which is essentially the mathematical foundation of the AHP apparatus.

Since 1988 (every 2 years), world symposia on the problems of applications and development of AHP (International Symposium on Analytic Hierarchy Process, ISAHP) have been held; the last (17th) took place in December 2022 (virtual format). The number of theoretical and applied works based on AHP in practically all fields (science, business, industry, healthcare, education, etc.) measures in the thousands (see, for example, reviews of works using AHP [2; 22; 30]).

AHP is a systematic decision-making method based on: 1) a hierarchical representation of the problem (in our case, the hierarchy of the procedure for measuring latent characteristics); 2) the collection of expert information (quantitative and qualitative) gathered within this hierarchy using pairwise comparisons of hierarchy objects; 3) recording the results of pairwise comparisons using the fundamental AHP scale (ratio scale), which is the basic element of the AHP apparatus; 4) obtaining quantitative assessments (in the ratio scale) of latent characteristics using the AHP apparatus.

AHP does not rely on probabilistic approaches, i.e., it is not a statistical procedure. AHP is used both for individual decisions (naturally, the expert must understand the problem at hand and be proficient with the AHP apparatus) and for group (collective) decisions (in practice, such a collective consists of up to 10-20 experts).

Hence, it follows that AHP and the classical variant of the Analytic Hierarchy Process (AHP) (as well as the family of models developed to improve AHP) are completely different models with their own areas of application. In practice, AHP is usually used to assess up to 10 latent characteristics of one or several subjects.

The advantages of AHP include the presence of a simple mathematical apparatus (linear algebra and matrix analysis), as well as standard computational procedures based on MS Excel.

The main achievements of T.L. Saaty and his colleagues (psychologists, psychophysiologists, mathematicians, and statisticians) include, first and foremost, the development and use of the ratio scale (fundamental AHP scale) for measuring expert preferences in pairwise comparison procedures and the assessment of objects of any nature [2; 10; 15; 28; 29].

This numerical scale was derived based on the ratios of the process of nerve excitation, which lead to the well-known psychophysiology law "stimulus-response," and has shown its effectiveness in numerous diverse applications and studies.

The scale is an infinite interval (0; ∞), and the main values of the scale, which are convenient to use in most applications, are numbers from the interval (0; 9). This scale does not include zero, as otherwise objects with zero weight, i.e., insignificant for the considered problem, would be taken into account.

For the convenience and ease of the expert’s work in pairwise comparisons of objects, the main numbers of the AHP scale have linguistic descriptions: 1 – equal importance; 3 – slight preference; 5 – significant preference; 7 – strong preference; 9 – absolute preference; 2, 4, 6, 8 – for intermediate cases. The scale also includes corresponding reciprocal values (results of reverse comparisons). At the same time, the team of T.L. Saaty has taken into account the psychophysical characteristics of humans when processing information.

2.2 Main Stages of Information Processing Tasks within AHP

The problem of assessing the condition of patients (within the example) can be represented as a hierarchy (top-down): the 1st level – the final assessment of the condition; the 2nd level – specialists-experts; the 3rd level – assessment criteria (in our example, one criterion is used, but in general, there can be several), measured on appropriate psychometric scales; the 4th level – patients.

To solve the considered example of processing ranked information, the normative AHP approach [2; 10-12; 15] should be used. The approach is based on using expert pairwise comparisons of ranked scale assessments (using the fundamental AHP scale) to form a numerical scale of intensities of the corresponding ranked scale assessments.

Let’s consider the matrix (Table 2) of expert assessments of pairwise comparisons of levels of the 7-point scale (“1”,…, “7”), using the main values of the fundamental AHP scale to record the results of pairwise comparisons. Recall that the 7-point ranking scale is used in the considered example (Table 1). The levels of the ranking scale from the minimum ("1") to the maximum ("7") are ordered by the severity of the symptom expression.

Table 2. Matrix of pairwise comparison results of rankings ("1", ..., "7") of the rating scale based on expert consensus (E).

The elements of the matrix in Table 2 are numbers from the fundamental AHP scale, representing the results of pairwise expert comparisons of the levels of the 7-point scale. For example, the main diagonal of such a matrix always contains the value 1 (highlighted in bold italics in the matrix), since the comparison result of any level of the rating scale with the same level has equal importance, expressed as the number 1.

It is convenient to consider the algorithm for filling the matrix using the example of the first row, the elements of which record the degree of preference (by the severity of symptom expression) of level "7" over the following levels ("6", "5",..., "1"). The result of the preference for rating "7" over rating "6" was assessed by the experts with the number 2, indicating an intermediate value between equal importance (1) and slight preference (3). Next, the result of the preference for rating "7" over rating "5" was assessed by the experts with the number 3, indicating a slight preference. Finally, the result of the last comparison: the preference for rating "7" over rating "1" was assessed by the experts with the number 9, indicating an absolute preference.

The elements of the following rows of the upper triangular part of the matrix (above the main diagonal of the matrix) are filled similarly and will always have values greater than 1, as the compared ranks are ordered by the severity of the symptom. The lower part of the matrix (below the main diagonal) is symmetrically filled with reciprocal values (less than 1) since they correspond to the results of reverse comparisons. Therefore, AHP pairwise comparison matrices are called reciprocal symmetric, where elements symmetric relative to the main diagonal satisfy the relationship:

a (i, j) = 1 / a (j, i),

where a(i, j)  is an element of the pairwise comparison matrix A (Table 2) with indices i and j.

We present the main theoretical AHP relationships for finding the weights of the rating scale assessments [2; 10-12; 15; 28; 29]. Let w(i) denote the numerical normalized weights of the i-th rating assessment, where I = 1,..., 7 (the normalization of weights means satisfying the condition: ∑ w(i) = 1).

Then the scale of numerical intensities (Int) of the rating scale assessments is obtained based on the values w(i) by the formula:

Int (i) = w(i) / w(7), (1)

If we denote by W = (w (7), w (6), w (1)) T the column vector of normalized weights of the rating scale assessments (T indicates the matrix operation "transposition," which converts a row into a column), then the main AHP equation is:

A*W = λmax*W, (2),

where A is the pairwise comparison matrix (Table 2), and λmax​ is an important numerical parameter in AHP theory, which, along with the column vector W, needs to be found from the nonlinear matrix equation (2). The exact solution (i.e., the values of λmax​ and W) of the nonlinear equation (2), considering the normalization of weights w(i), can be obtained using the MS Excel "Solver" add-in, requiring a certain level of proficiency with this tool.

We will present the exact solution of equation (2) below, but now we will outline a simple algorithm based on elementary algebraic operations to find an approximate value (accurate to 0.01) of the elements of the vector W based on the elements of the matrix A (see, for example, [11; 12; 15; 28]).

The algorithm consists of 4 stages:

1) Find the row products (p) of the elements of matrix A (e.g., for the first row A from Table 2, we get: p(1) = 11340, for the second row, p(2) = 2835, etc.);

2) Extract the 7th root from each obtained product (the root exponent coincides with the size of the matrix A). As a result, we get for the first row q (1) = 3.795; for the second q (2) = 3.113, etc.;

3) Find the sum of the obtained roots. S = q (1) +q (2) +…+q (7). We get S = 10,796;

4) Find the weights w(i) by the formula: w(i) = q(i) / S.

Table 3 shows the approximate values of weights w(i) and corresponding intensities Int(i) for the 7-point scale obtained using this algorithm.

Table 3. Approximate values of weights and corresponding intensities for the levels of the 7-point scale

Table 4 shows the exact values of weights w(i) obtained when solving equation (2) using MS Excel and the corresponding intensities Int(i) for the levels of the 7-point scale.

Table 4. Exact values of weights and corresponding intensities for the levels of the 7-point scale

As can be seen from the results in Tables 3 and 4, the approximate and exact values practically coincide (the difference in values is no more than 3-4%). This result is explained by the good consistency of the expert information represented by the pairwise comparison matrix A (Table 2).

In AHP, the consistency of expert data presented in matrix A is determined by the parameter λmax​ based on the consistency ratio (CR) value using the formula:

CR = RICI. ​

In this formula: CI = (λmax – n) / (n – 1) – Consistency Index (n – size of matrix A), and RI – Random Index, which is taken from the corresponding table of random indices; in particular, RI = 1.35 for n = 7 (see, for example, [2; 15; 29]).

For CR = 0 – the matrix A is perfectly consistent. For 0 < CR ≤ 0.1– it is considered that the matrix is well consistent (i.e., in this case, the CI only slightly matches the random RI – no more than 10%).

For the considered example (Table 2), the value λmax = 7.50 and the corresponding value CR = 0.06 (indicating good consistency of matrix A).

In practical research, the first step after forming the matrix A is to find the value λmax (approximate or exact) and assess the corresponding value CR.

If the expert matrix A does not have good consistency, it is necessary to eliminate the causes of inconsistency in expert preferences and achieve good consistency of matrix A. This can be done through various means, by reaching a consensus among the experts or by involving more experienced experts (see, for example, [15]). Additionally, there are AHP modifications that significantly reduce the amount of expert work while obtaining perfectly consistent pairwise comparison matrices, for which the calculation of exact elements of the vector W requires only elementary algebraic operations, as in the above algorithm [12; 13].

We will now provide the solution to the initial example (Table 1), based on the numerical intensity values for the 7-point rating scale assessments (Table 4).

Denoting the arithmetic mean of intensity values for patient P1 as S1, and for P2 as S2, and using the intensity values of the ranking levels from Table 4, we obtain to the accuracy of 0.01:

S1 = (8 * Int (6) + 3 * Int (4)) / 11 = 0,68; S2 = (8 * Int (5) + 3 * Int (6)) / 11 = 0,51

Considering the standard error of the mean calculation, we get:

S1 = 0,68 ± 0,07; S2 = 0,51 ± 0,06

Thus, the numerical solution to the considered example, as well as the correct ranking solution, provides a consistent clinical result, namely: the condition of patient P1 should be considered more severe.

3. Remarks on the Rasch Measurement System and Criticism of Psychometric Tools

As an alternative to psychometric approaches, which practice a "numerical" view of ranking data and use transformation procedures based on normal distribution, G. Rasch [26; 27] developed a probabilistic metric approach to measuring latent psychosocial properties. The English name for this approach is Rasch Measurement, and in domestic literature (see, for example, [1]), it is referred to as the Rasch Measurement System (RMS).

The classical RMS model was developed for tasks with dichotomous items of uniformly increasing difficulty, allowing for the assessment of the test structure and the measurement of the level of expression of the latent trait in respondents.

RMS can be used in studies of practically any personal trait, provided its statistical stability and clinically significant observability using a system of psychosocial indicators. RMS includes the technology of test development as well as mathematical and software tools for assessing the difficulty measure of tasks and the level of preparedness of respondents.

3.1. Difficulties in Using RMS

1. The practical implementation of RMS is associated with fairly large samples (hundreds of respondents) and corresponding volumes of statistical information, the processing of which requires specialized mathematical and software tools.

2. Statistical modeling based on RMS uses aggregated data, which represents information about personality characteristics at the population level. However, the results of processing statistical data are difficult to interpret at the individual level, which is essential for psychosocial applications.

Emerging problems in the development and assessment of test characteristics based on RMS (e.g., the well-known Rasch paradox within the classical RMS model – the risk of removing the best items from the test) require the use of iterative expert procedures to solve them.

3.2. On Criticism of RMS

Since part of the criticism of psychometric tools (e.g., in [17]) was directed at RMS, let us first consider an important example of the practical use of the classical RMS as a model for the reasons behind the criticism of psychometric tools.

The developers of the well-known Unified State Exam (USE) system use RMS algorithms (see, for example, [9]) to convert "raw" scores into test scores and appropriately assess the examinees' knowledge. The classical RMS model is used here, in which success in solving test tasks depends only on two variables: the examinee's knowledge level and the task's difficulty, which are to be assessed based on exam results.

However, it is known that the success of any human activity significantly depends on motivation. Modern assessments of learning outcomes and corresponding cognitive processes show that motivation affects success several times more than individual abilities (e.g., [16]).

Furthermore, the USE test (e.g., in mathematics, both basic and advanced versions) includes both simple tasks (arithmetic, planimetry) and more complex ones (derivative tasks, the ability to build mathematical models, stereometry). As a result, the following student behavior strategies are observed in practice (for both basic and advanced versions), which can be divided into roughly equal groups: 1) The first group needs mathematics for continuing engineering-technical and mathematical education; hence, they aim to get the maximum number of points; 2) The second group aims to get a diploma and pursue humanitarian education; hence, they have a minimum strategy.

Moreover, USE organizers try to statistically compensate for these violations using exam results nationwide (the concept of statistical modeling). However, it is obviously impossible to consider different regional and socio-economic conditions affecting the uneven preparation of students under the current form of the USE. This also evidently leads to a violation of the applicability of RMS and a flood of critical remarks about the USE results.

The example considered shows that to evaluate a specific psychometric tool, it is necessary to consider the basic concepts of the method that define its applicability and to properly understand the main mathematical and statistical procedures used to present and process data within the method.

In this context, let us present specific, significant errors made in the criticism of RMS in [17].

1) On page 5 of this work, one can read (a direct quote): "The logistic transformation is needed to approximate the score distribution to normal, as the Rasch model assumes that the latent characteristic (trait, property) being studied is normally distributed in the population." This quote shows that its author does not understand the basic principles of RMS and the conditions for their implementation.

Indeed, it should first be noted that the logistic transformation (function) in RMS refers to the functional relationship between the probability of a correct answer to a test item, the examinee's preparation, and the task's difficulty and is not related to any normal distribution of scores in the population. Moreover, G. Rasch in his works [26; 27] opposed the use of the normal Gaussian distribution for data related to human activities, which has received substantial confirmation nowadays (see, for example, [18]).

As for the logistic transformation itself, G. Rasch [26; 27], based on the general properties of the model for estimating the probability of the i-th examinee's response to the j-th test item (P (i, j)), proposed a functional relationship in the form of a logistic function:

P (i, j) = exp (bi – tj) / (1+ exp (bi – tj)), (3),

where bi​ is the knowledge level of the i-th examinee, and tj is the difficulty level of the j-th task. The logistic function (3) has no relation to the normal distribution of scores in the population.

2) Referring again to [17], on page 6, we find an elementary error in using the logarithmic function:

Ln (m1 / m2) = Ln (m2 – m1).

That is, the author "invents" a new algebraic formula, assuming that the logarithm of the ratio is equal to the logarithm of the difference (moreover, swapping the arguments in the difference!), instead of the correct formula: the logarithm of the ratio is equal to the difference of the logarithms.

This error could be dismissed as a typo, but three lines later on the same page the error is repeated:

Ln (b1 / b2) = Ln (b2 – b1).

Using this formula with b1 = b2, we can receive Ln (1) = Ln (0), or 0 = - ∞!

These errors (the conceptual one on page 5 and the elementary algebraic ones on page 6) raise justified doubts about the author’s [17] understanding not only of RMS concepts but also of elementary algebraic operations constituting the RMS apparatus, so the criticism of RMS presented in the indicated work should be recognized as untenable.

3.3. On Criticism of AHP

There are critical remarks in the literature concerning the concepts and apparatus of AHP. Such remarks, and sometimes logical and analytical "counterexamples" proving the alleged insolvency of AHP concepts and apparatus, are considered in [10; 15; 29]. These works show that the authors of the critical remarks interpret AHP concepts very superficially (in particular, they do not understand the fundamental difference between the descriptive and normative approaches of AHP and the conditions for their application) and also make elementary errors in using the AHP apparatus. The most common errors not only in articles but also in textbooks on AHP are described in [10].

Conclusions

1. The decisive advantage of the AHP method in the problem of measuring and evaluating latent characteristics is a clear representation of the structure of the multifactorial problem, the elements of the problem, and the interdependencies between them based on the hierarchical model, as well as the numerical expression of expert judgments based on the fundamental AHP scale.
2. The methods of obtaining information from an expert within the AHP framework correspond to psychologically comfortable conditions, meeting the psychophysical capabilities of a person to process information. A very important element of the AHP apparatus is the ability to assess the consistency of expert judgments using consistency indices (CI and RI), which allow for judging the quality of expert information. At the same time, data processing based on AHP algorithms does not require specialized mathematical and software tools; only standard MS Excel tools are sufficient.
3. Psychometric tools such as AHP and RMS allow for accurate numerical assessments of a person's latent characteristics. Criticism of these tools is associated with a superficial understanding of the basic concepts of psychometric measurements and their areas of application, as well as with elementary errors in using the psychometric tools' apparatus.

1. Avanesov V.S. Metricheskaja sistema Georga Rasha – Rasch Measurement (RM) [The metric system of Georg Rasch-Rasch Measurement (RM)]. Pedagogicheskie izmerenija [Educational measurements], 2010, no. 2, pp. 57–80. Available at: https://cyberleninka.ru/article/n/metricheskaya-sistema-georga-rasha-rasch-measurement-rm. (In Russ.).
2. Andreichikov A.V., Andreichikova O.N. Nauka i iskusstvo prinyatiya reshenii. Kn. 2: Prinyatie reshenii v usloviyakh neopredelennosti: ot metoda analiza ierarkhii do nechetkikh modelei. Uchebnik [The science and art of decision-making. Book 2: Decision-making in conditions of uncertainty: from the analytic hierarchy process to fuzzy models]. Moscow: Publ. LENAND, 2021. 800 p. (In Russ.).
3. Assanovich M.A. Sravnitel'naja ocenka porogovyh kriteriev vyrazhennosti depressii, razrabotannyh na osnove klassicheskoj teorii testov i modeli Rasha [Comparative Evaluation of Cut-Off Criteria of Depression Severity Developed on the Basis of Classical Test Theory and Rasch Model]. Vestnik JuUrGU. Serija «Psihologija» [Bulletin of the South Ural State University. Ser. Psychology], 2017. Vol. 10, no. 1, pp. 26–34. DOI:10.14529/psy170103 (In Russ.).
4. Assanovich M.V., Karpiuk V.A. Psihometricheskie svojstva i diagnosticheskie kriterii shkaly ocenki negativnyh simptomov (SANS – Scale for the assessment of negative symptoms) pri shizofrenii [Psychometric properties and diagnostic criteria of Scale for the assessment of negative symptoms (SANS) in schizophrenia]. Obozrenie psihiatrii i medicinskoj psihologii imeni V.M. Behtereva [V.M. Bekhterev review of psychiatry and medical psychology], 2021. Vol. 55, no. 1, pp. 33–41. DOI:10.31363/2313-7053-2021-1-33-41 (In Russ.).
5. Bruk B.N., Burkov V.N. Metody ekspertnykh otsenok v zadachakh uporyadocheniya ob"ektov [Methods of expert assessments in the problems of object ordering]. Izvestiya AN SSSR. Tekhnicheskaya kibernetika [News of the USSR Academy of Sciences. Technical cybernetics], 1972, no. 3, pp. 29–39. (In Russ.).
6. Zolotareva A.A. Psihometricheskie svojstva russkojazychnoj versii Shkaly vosprinimaemogo stressa (versii PSS-4, 10, 14) [Psychometric Properties of the Russian Version of the Perceived Stress Scale (PSS-4, 10, 14)]. Klinicheskaja i special'naja psihologija [Clinical Psychology and Special Education], 2023. Vol. 12, no. 1, pp. 18–42. DOI:10.17759/cpse.2023120102 (In Russ.).
7. Kukshina A.A., Kotelnikova A.V., Rassulova M.A., Dajlidovich V.S. Issledovanie psihometricheskih svojstv «Gospital'noj shkaly trevogi i depressii» (HADS), rekomendovannoj dlja vrachej obshhesomaticheskoj praktiki, na vyborke pacientov s narusheniem dvigatel'nyh funkcij [Investigation of the Psychometric Properties of the Hospital Anxiety and Depression Scale (HADS) Recommended for General Medical Practitioners, on a Sample of Patients with Impaired Motor Functions]. Klinicheskaja i special'naja psihologija [Clinical Psychology and Special Education], 2023. Vol. 12, no 2, pp. 1–24. DOI:10.17759/cpse.2023120201 (In Russ.).
8. Matematicheskie metody v psihologii [Mathematical methods in psychology]. Kupcov M.I., Filipova E.E., Slobodskaya I.N., Pavlova S.A., Koryachko A.V., Zharkikh A.A. Moscow: Publ. Goryachaya liniya–Telekom, 2017. 156 p. (In Russ.).
9. Malinetzky G.G., Podlazov A.V. EGJe kak katalizator krizisa Rossijskogo obrazovanija [USE as a Catalyst for the Crisis of Russian Education]. Vestnik MGU. Serija 20: Pedagogicheskoe obrazovanie [Bulletin of the Moscow State University. Series 20: Pedagogical education], 2011, no. 3, pp. 18–59. Available at: https://cyberleninka.ru/article/n/ege-kak-katalizator-krizisa-rossiyskogo-obrazovaniya (In Russ.).
10. Mitikhin V.G. K voprosu reshenija mnogokriterial'nyh zadach na osnove metoda analiza ierarhij [On the issue of multi-criteria decision problems based on the Analytic Hierarchy Process]. Cloud of Science [Cloud of Science], 2015. Vol. 2, no. 4, pp. 519–529. Available at: http://cloudofscience.ru (In Russ.).
11. Mitikhin V.G., Alieva L.M., Enikolopov S.N. Primenenie metoda analiza ierarhij dlja obrabotki dannyh issledovanija telesnogo obraza «Ja» u bol'nyh psihicheskimi rasstrojstvami [An analytic hierarchy process of body image in mental disorders]. Psihiatrija [Psychiatry], 2015, no. 1(65), pp. 29–32. DOI:10.30629/2618-6667-2015-65-29-32 (In Russ.).
12. Mitikhin V.G., Solokhina T.A., Kuzminova M.V., Tiumenkova G.V., Limankin O.V., Babin S.M. Jeffektivnost' psihosocial'noj reabilitacii: innovacionnyj metod ocenki [Effectiveness of psychosocial rehabilitation: innovative method of results assessment ]. Psihiatrija [Psychiatry], 2022. Vol. 20, no. 2, pp. 51–59. DOI:10.30629/2618-6667-2022-20-2-51-59 (In Russ.).
13. Nogin V.D. Uproshchennyi variant metoda analiza ierarkhii na osnove nelineinoi svertki kriteriev [A simplified version of the analytic hierarchy process based on nonlinear criteria convolution]. Zhurnal vychislitel'noi matematiki i matematicheskoi fiziki [Computational Mathematics and Mathematical Physics], 2004. Vol. 44, no. 7, pp. 1261–1270. (In Russ.).
14. Orlov A.I. Prikladnaya statistika [Applied statistics]. Moscow: Publ. Ehkzamen [Exam], 2006. 656 p. (In Russ.).
15. Saaty T.L. Prinyatie reshenii pri zavisimostyakh i obratnykh svyazyakh: Analiticheskie seti / Per. s angl. [Decision making with dependence and feedback: The Analytic Network Process]. Moscow: Publ. LKI, 2008. 360 p. (In Russ.).
16. Solodova E.A. Novye modeli v sisteme obrazovaniya. Sinergeticheskii podkhod [New models in the education system. A synergistic approach]. Moscow: Publ. URSS, 2016. 344 p. (In Russ.).
17. Tyumeneva Y.A. Chto dajut chisla dlja izmerenija v psihologii? Obzor kritiki v adres netestiruemyh dopushhenij psihometriki i vozmozhnye al'ternativy «izmeritel'noj» psihologii [What do numbers provide for measurement in psychology? A Review of Critiques of the untested assumptions of psychometrics and possible alternatives to "measurement" in psychology]. Psikhologicheskie Issledovaniya [Psychological research], 2022. Vol. 15, no. 84, pp. 1–30. DOI:10.54359/ps.v15i84.1198 (In Russ.).
18. Khaitun S.D. Kolichestvennyi analiz sotsial'nykh yavlenii. Problemy i perspektivy [Quantitative analysis of social phenomena. Problems and prospects]. Moscow: Publ. LIBROKOM, 2014. 280 p. (In Russ.).
19. Shalaginova K.S., Dekina E.V., Klochneva D.A. Jeskapizm i videoigrovaja addikcija u podrostkov [Escapism and Video Game Addiction in Adolescents]. Psikhologo-pedagogicheskie issledovaniya = Psychological-Educational Studies, 2023. Vol. 15, no. 4, pp. 97–110. DOI:10.17759/psyedu.2023150406 (In Russ.).
20. Bech P. Clinical psychometrics. Chichester, West Sussex: Wiley-Blackwell, 2012. 216 p.
21. Bond T.G., Fox C.M. Applying the Rasch Model: Fundamental Measurement in the Human Sciences. N.Y.: LEA, 2015. 406 p. DOI:10.4324/9781315814698
22. Kim S., Kim H., Suh H.S. Priorities in the Prevention Strategies for Medication Error Using the Analytical Hierarchy Process Method. Healthcare, 2022. Vol. 10, no. 3, pp. 512–523. DOI:10.3390/healthcare10030512
23. Kopylov A.T., Stepanov A.A., Butkova T.V., Malsagova K.A., Zakharova N.V., Kostyuk G.P., Elmuratov A.U., Kaysheva A.L. Consolidation of metabolomic, proteomic, and GWAS data in connective model of schizophrenia. Scientific Reports, 2023. Vol. 13, no. 1, pp. 21–39. DOI:10.1038/s41598-023-29117-7
24. Michell J. Measurement in psychology: Critical history of a methodological concept. Journal of Applied Measurement, 1999. Vol. 2, no. 1, pp. 96–100.
25. Pfanzagl J. Theory of measurement. In cooperation with Baumann V. and Huber H. 2nd revised edition. Wurzburg–Wien: Physica-Verlag, 1971. 235 p.
26. Rasch G. Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Danmarks Paedagogiske Institut, 1960. 184 p.
27. Rasch G. Probabilistic Models for Some Intelligence and Attainment Tests. With a Foreword and Afteword by B.D. Wright. Chicago & London, 1980. 199 p.
28. Saaty T.L. The Analytic Hierarchy Process. New York: McGraw Hill, 1980. 287 p.
29. Saaty T.L. On the Measurement of Intangibles. A Principal Eigenvector Approach to Relative Measurement Derived from Paired Comparisons. Notices of the American Mathematical Society, 2013. Vol. 60, no. 2, pp. 192–208. DOI:10.1090/noti944
30. Schmidt K., Aumann I., Hollander I., Damm K., Graf J.-M. Applying the Analytic Hierarchy Process in healthcare research: A systematic literature review and evaluation of reporting. BMC Medical Informatics and Decision Making, 2015. Vol. 15, no. 1, pp. 112–139. DOI:10.1186/s12911-015-0234-7
31. Uher J. Psychometrics is not measurement: Unraveling a fundamental misconception in quantitative psychology and the complex network of its underlying fallacies. Journal of Theoretical and Philosophical Psychology, 2020. Vol. 41, no. 1, pp. 58–84. DOI:10.1037/teo0000176