New approaches for assessing the activities of operators of complex technical systems

By now, a certain amount of results related to selection of abnormal exercise implementations has been accumulated [1—2, 7—15, 18, 29—31, 35, 37—38]. The vast majorities of them consider aircraft trajectories only and do not take into account other exercise parameters. The techniques obtained have the following limitations, which essentially restrict their practical applications, especially in case of analyzing flight data:

— Laborious manual preliminary data processing is always required;

— No rigorous mathematical criteria for selecting abnormal exercises;

— No capabilities to reveal the causes of incorrect exercise implementations including flight parameters clarifying them;

— Inefficient numerical optimization methods were selected;

— Different time scales for flight parameters under study, which are not acceptable for correct data analysis, are in use;

— Unsuccessful choices for metrics to compare exercises;

— Very significant computing resources are in need for calculations that results from both time series representation of the data to be processed and selection of time-consuming algorithms for solving relevant optimization problems;

— The EM-algorithm involved in data processing is not usually applicable for the data of high dimension;

— The regression models involved in data processing cannot be applied effectively for all possible flight parameters of interest;

— Calculated “averaged” structures (“centroids”) which are in use for flight classification cannot be generally considered as samples those might be implemented during a real flight;

— So called “negative scaling” that yields unrealistic interpretation for flight analysis is frequently resulted from data processing;

— Unacceptably large samples of similar exercises are required for qualitative analysis;

— Clustering flight trajectories instead of flights themselves;

— No really successful attempts to reduce the amount of flight data to be processed that results from their time series representation.

Certain application prospects are available for the approach implemented with the aid of the self-organizing maps (SOM) [5, 10]. However its real capabilities are limited essentially by non-determinism of SOM clustering and an inefficient way of aligning analyzed time series in the time domain, with structural stability of the obtained solutions being questionable.

As a result, there are only few examples of useful practical applications of these techniques by now (usually landing trajectories are under study). It should be noted that one of the principal weaknesses of the techniques in question is applying traditional metrics for comparing flight fragments since these metrics failed to solve the problem under study as well as related problems.

Since pilots are mentioned and because one of the most popular, but the least effective method for simple assessing the piloting quality is to check whether certain parameters are in the given critical value ranges for the specified flight modes, comments on this topic are unavoidable. Validity of such a technique is best clarified by the following analogy: it looks like an attempt to assess the quality of car driving via the number of collisions with a fence along the road, with all the other information about the car movement being ignored. It is obvious that the effectiveness and relevance of such evaluations do not hold water.

All the limitations presented above are overcome with the aid of the techniques [27—28] based on wavelet representation of operator activity parameters and comparisons of eigenvalue trajectories associated with observed exercises in the likelihood metric, which practical application is under consideration in this paper. Presented below are the techniques for skill assessments associated with relevant activity data obtained with the aid of both experiments and mathematical analysis of their results. As a result of its application, classes of skills should be determined using activity parameters and operator actions revealed during exercises.

To evaluate the data under study an activity record database is required, in which patterns of training data representing exercise implementations by different operator crews are collected. A pattern in question is a representation of a certain activity fragment to be analyzed, which is referred to as an exercise, via the set of parameters describing this exercise. These patterns are to be related to one of the recognizable skill classes of trainees.

Collected training data should include exercise parameters as well as relevant expert as- sessment/supervision comments from different sources including various types of training simulators, virtual reality systems and real operator work. The expert comments in use should reveal weak points of OCTS performance using information about typical mistakes in terms of activity parameters and advices to a local operator instructor how to correct these weaknesses. Certain attributes related to recognizable classes can be stored in the database both for cluster centers of patterns belonging to the classes under consideration and for each exercise pattern to solve application problems of interest.

The general assumption for the approaches under consideration is that the activities implemented in different styles and quality as well as exercises of different types can be discriminated in the multidimensional space formed with the aid of the wavelet coefficient metric or the likelihood metric of eigenvalue trajectories for activity parameters transforms. This statement is proved by computer experiments based on relevant empirical data. The general method that results from this conclusion is the pattern selection.

The techniques presented provide essential advantages over manual data analysis since they greatly reduce the combinatorial enumeration of the options considered.

The approaches in question can be used to support education activity and instructor assessment processes. They should be also useful as tools for comparing different training means and syllabus. They differ significantly from the probabilistic methods applied for system control, predictive diagnostics of technical failures, condition monitoring and operator activity support [16].

Basic approach: analysis of activity parameters represented via the discrete
wavelet transform of their time series

It is assumed that operator’s activity is represented by a set of time series describing the dynamics of technical system parameters as well as, if possible, the operator’s state. Both principal steps and connections of the analysis performed according to the proposed approach are presented in Figure 1, comments to which are placed below.

Among the purposes of this analysis are:

— Support of the outcome grading for current activity by means of its comparing with the activity patterns collected beforehand in the corresponding record database;

— Recognition of abnormal activity and detection of the parameters characterizing operator mistakes to reveal the sources of abnormality.

As preliminary processing, selections of time intervals for exercise comparison and data normalization are carried out. Selected are time series subsets corresponding to common time intervals, which are suitable for comparison of the same type exercises to be analyzed. Prior to the following computations, time series representing the history of performance of exercise completion are brought to a single scale, where the maximum is one and the minimum is zero.

Where the data are broken down by certain exercises represented by several measured parameters, the redundant information contained in the given time series is eliminated using the Principal Components Analysis [34, 39]. To do so, matrices of mutual correlations of time series values are computed, the algebraic problem of eigenvalues is solved and it is figured out to what extent it is possible to decrease the dimension of eigen subspace of the researched parameters so that this would contain a sufficiently representative part of variability of the observed parameters. For each of the selected eigendirections of this subspace (principal components), for one of the highest component loads a representative is elected from registered parameters (transition to the basis of principal components is impractical due to uncertain substantial interpretation of principal components and, in a number of applied tasks, due to no precise synchronization of researched processes for different exercises in time). The purpose of this stage is to find out only relatively independent characteristics replacing groups of significantly dependent representatives with only one characteristic parameter to avoid distortions stipulating to combined effect of strongly dependent characteristics in the subsequent phases.

Following to the basic approach, transition to integral characteristics for time intervals by means of discrete wavelet transform is performed. Time series representing the training processes under study are replaced with series of wavelet coefficients obtained as a result of the Multiresolution Analysis [33]. In this case, the original processes as functions of time are replaced with the integral characteristics of the time intervals, which are associated with these functions domain. In addition, significant saving (for about an order of magnitude) in the number of coefficients necessary for a correct representation of these process becomes available. Due to the rules for assigning wavelet coefficients to time series fragments, which are in use in multiresolution analysis, problems associated iWth the need to precisely synchronize processes relating to various same-type training exercises in tim? hca^^ been cancelled since the most significant coefficients relating to relatively long time intervals are almost insensitive to moderate time shifts. Length of the wavelet representations used during subsequent analysis can be significantly (approximately an order of magnitude) smaller than the length of the corresponding original time series, with no loss in the estimations accuracy.

For exercises under study, one needs either to compute the matrix of mutual distances between wavelet representations of source processes for different operator activities or to use the alternative approach creating similar matrices in the likelihood metric of eigenvalue trajectories associated with observed activity fragments (this way is presented in the subsequent Section). Dimensions of such matrices are equal to the sample sizes of analyzed exercises. Such matrices of mutual distances for all considered parameters are added to form the total matrix of mutual distances between the exercises under study. Estimation of relative contributions of activity parameters in the elements of the mutual distances matrices is available for recognizing abnormal exercises in the scaling spaces and detection of the parameters characterizing operator mistakes to reveal the sources of abnormality.

Multidimensional Scaling [3, 36] to analyze the mutual allocation of operator exercises within a space with acceptable dimensions is available after computing the matrix of mutual distances. The computed allocation of exercises under study in the resulting space of the Multidimensional Scaling is further used to define distances between the exercises to make diagnostic decisions. The dimensions of the scaling space are defined based on the condition of sufficient differentiation of exercise samples relating to different recognizable classes. The aim of this step is subsequent discrimination between the exercise types and normal/abnormal activity implementations in a scaling space.

The Cluster Analysis of patters in the obtained scaling space is performed to reveal clusters representing various types of exercises and operator skill classes. Obtained results provide possibility for creating certain classification rules to separate different scale levels of trial quality assessment in a scaling space. Wherein cluster differences for an exercise type can be explained by the exercise implementation resulted from individual skills. In particular, both Cluster Analysis and Multidimensional Scaling steps are necessary to reasonably select clusters of abnormal exercises.

Probabilistic models represented by Markov random processes with discrete states and continuous time [6, 18—25, 32] for each pattern cluster are created using the identification procedure to represent probabilistic dynamics for each operator skill class to forecast probabilistic class behavior. This step is implemented in two ways: via distribution of probabilities of being in model states and via dynamics of mathematical expectations for each independent parameter determined with the aid of the Principal Components Analysis (correspondingly, these parameters are considered approximately as independent ones).

Computations of distances to pattern cluster centers or to the nearest patterns are based on results of performing a sequence of test exercises. If the pattern sample size is fairly large, distances are defined to centers of cluster patterns. These are computed based on multidimensional scaling data obtained earlier. If pattern samples are small, the nearest pattern is defined, which definition may be done in two ways: either immediately through the computing of the pattern being nearest in the Euclidian metric of wavelet representation or through the identifying of the pattern in the resulting space of multidimensional scaling, with such pattern being nearest in the Euclidian metric.

Probabilistic assessments of operator skill class recognition using sample distribution functions of distances to cluster centers are defined using sample distribution functions F.(X) of Euclidean distances X to cluster centers of patterns belonging to the relevant recognizable operator skill class i e {0, ..., z} in a multidimensional scaling space. Computed values ofpt = 1 - Fi(ri), where r is the Euclidean distance of ith cluster in the space of multidimensional scaling, are interpreted as probabilistic assessments of belonging to the given classes. Their distribution among classes {pi}z=0 characterizes the reliability of the obtained classification. In fact, the given approach implements the idea of the Linear Discriminant Analysis in an generalized form. But unlike the latter there are no restrictions on observed data distributions.

After an operator completes a certain test exercise, in order to collect a meaningful data set representing his activity accumulated by some time check point, the following results are to be determined with the aid of the given database:

— The nearest pattern, following to which the relevant attributes are ascribed to the operator under assessment;

— Distances to cluster centers of patterns belonging to recognizable classes together with the sample distribution functions of distances to centers of such clusters.

Probabilistic assessments of operator skill class attribution using Bayesian likelihood estimations are calculated basing on identified probabilistic models.

Forecasting behavior of the selected parameters for the recognized operator skill class cluster with the aid of the created probabilistic models is carried out by both distribution of probabilities of being in model states and dynamics of mathematical expectations for each independent parameter to estimate probability of hitting into dangerous situations associated with the certain intervals of the parameters under consideration.

So, three ways of skill class assessments are available:

—    direct comparison of current exercises with the activity database patterns in the wavelet representation metric associated with observed exercises, which is considered as a basic technique, as well as on

—    probabilistic assessment of skill class recognition via the Generalized Discriminant Analysis using sample distribution functions of exercise distances to cluster centers in a scaling space, which is considered as a supportive technique, and

— Bayesian likelihood estimation (selecting a skill class with the aid of probabilistic profile of staying in activity parameter ranges), which is also considered as a supportive technique.

Wherein, the actions, which an expert is responsible for, are:

— Assessment to select clusters of abnormal exercises;

— Analysis of parameter’s balance to select relevant parameters to be under study;

— Identification and interpretation of abnormalities and mistakes.

The approach in question can be applied even if the exercise sample sizes are small since both the pattern selection and calculation of the parameter’s contributions to the mutual distances in pairwise exercises comparison are available in this case.

The Intelligent System for Flight Analysis (ISFA) implementing the developed techniques with the aid of the LabVIEW graphical programming system provides the required calculations [27—28]. ISFA has been officially registered at the Russian Patent Agency (“ROSPATENT”) [26], with the following list of computational macro operations being available:

— Viewing characteristics of the activities represented by the exercises, including characteristic parameters and relevant meaningful comments;

— Eliminating of redundant information by means of the Principal Components Analysis;

— Assessments of a skill class by selecting the closest activity pattern with the aid of the matrix of mutual distances between wavelet representations of time series for collected activities (1st way of skill class assessment);

— Getting the matrix of successful transitions to select a next training activity;

— Clustering collected activities by the K-Means Algorithm;

— Assessments of a skill class using sample distribution functions of distances to cluster centers in a scaling space (2nd way of skill class assessment);

— Identifying probabilistic profile of a selected parameter for a selected cluster in terms of the Markov process with time-dependent transition flow rates between the states. This process implies solving the inverse problem for parameters of a set of ordinary differential equations. It uses observed solution fragments for a certain number of time points, with transition flow rates itself being assumed to be functions of time. Detailed description of a numerical method developed for the abovementioned identification is given in previously published works [22, 32];

— Assessments of a skill class using the Bayesian likelihood estimations. Visualization in form of 3 dimensional surface plots should be used to select characteristic parameters which are potentially responsible for abnormal activity implementation (3rd way of skill class assessment).

Alternative approach: analysis of activity parameters via complex
eigenvalue trajectories of their time series transform

Let us consider a set of M time-dependent operator’s activity parameters which are represented by the set of time series:

where values of index  correspond to discrete time points.

Let the r dynamics be represented by the following matrix equation:

where F. is time-dependent matrix function of dimension M. Since this matrix function is of interest for analyzing dynamics of the activity parameters under study, it should be identified, with empirical data being the only source for such estimation.

Despite the fact that matrix F. depends on time in the general case, it can be approximated by the corresponding stationary matrix during a sufficiently short period including K time points and starting at point i0, where K > M:

If this approximation does not fit empirical data, the time step used in forming the time series in question can be reduced several times to obtain the desired fitness.

If one considers K equations

each row of matrix F_i0 can be identified as a pseudosolution of the corresponding l-th matrix equation that is overdetermined in the general case: