WO2006106408A2 - System and method for estimation of human activity and mood states based on rapid eye movements and galvanic skin response - Google Patents

System and method for estimation of human activity and mood states based on rapid eye movements and galvanic skin response Download PDF

Info

Publication number
WO2006106408A2
WO2006106408A2 PCT/IB2006/000782 IB2006000782W WO2006106408A2 WO 2006106408 A2 WO2006106408 A2 WO 2006106408A2 IB 2006000782 W IB2006000782 W IB 2006000782W WO 2006106408 A2 WO2006106408 A2 WO 2006106408A2
Authority
WO
WIPO (PCT)
Prior art keywords
activity
isi
complexity
mood states
gsr
Prior art date
Application number
PCT/IB2006/000782
Other languages
French (fr)
Other versions
WO2006106408A3 (en
WO2006106408B1 (en
Inventor
Serguei Bedziouk
Anatoly N. Kostin
Yurij Ya Golikov
Original Assignee
Icon Orbital Technologies Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Icon Orbital Technologies Corporation filed Critical Icon Orbital Technologies Corporation
Publication of WO2006106408A2 publication Critical patent/WO2006106408A2/en
Publication of WO2006106408A3 publication Critical patent/WO2006106408A3/en
Publication of WO2006106408B1 publication Critical patent/WO2006106408B1/en

Links

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/05Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves 
    • A61B5/053Measuring electrical impedance or conductance of a portion of the body
    • A61B5/0531Measuring skin impedance
    • A61B5/0533Measuring galvanic skin response
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/16Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/165Evaluating the state of mind, e.g. depression, anxiety
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/24Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316Modalities, i.e. specific diagnostic methods
    • A61B5/398Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]

Definitions

  • the present invention relates generally to mental workload (or human activity) measurement and eye movement rBssarch. More particularly, the present Invention relates to human activity measurement, such as for operator training* and mood state estimation based on rapid eye movement
  • Different methods of eya movement analysis can be used for eye movement measurement, such as: electrooculogramic, electromagnetic, photo-electrical with infra-red source, photo-optical with mirror sensor, video or cine-camera means, etc.
  • Such methods can analyze view paths, eye fixation on different external objects, and saccadic movements due to a change of fixation on objects. This can reveal the main stages of the activity, structure of perceptual processes of information searching, moments of attention and difficulties, which then link with reasoning processes.
  • Use of these methods permits estimation of human activity when a subject is working with different options of indicators and interfaces, researching a nature of information processes, attention and reasoning.
  • Methods of GSR typically measure resistance, conductivity or potential difference of the human skin.
  • GSR Reactions of Activation are connected with regulation of human mood states when there is a fast change of GSR amplitude. However, the GSR Reactions of Relaxation can relate to a slow change of GSR amplitude. If measurement of skin resistance (or potential difference) is used, GSR Reaction of Activation Is associated with a fast decrease of the skin parameter. Using Measurement of skin resistance, GSR Reaction of Relaxation is associated with a slow increase of the skin parameter.
  • GSR Reaction of Activation Is associated with a fast Increase of the skin conductivity and QSR Reaction of Relaxation is associated with a slow decrease of the skin conductivity.
  • the use of GSR method permits analysis of various human mood states that can be influenced by extreme factors such as stress, weariness, illness, emotions, physical efforts, etc.
  • Some known methods estimate human activity by comparing time Intervals between saccadic eyes movements (Inter Saccadic Intervals - ISI) with certain ranges (ta ⁇ ons) of ISI which characterize regulation levels of human activity having different contents and complexity.
  • the method Is based on the psych ⁇ -physiDl ⁇ gica! evidence for the relationships between ranges of ISI changing and different levels of regulation processes.
  • As criteria of an estimation of activity both the current and total workload are used.
  • the method permits an estimation of human activity In various control systems, such as on simulators. It also provides the ability to estimate a complexity of human controlled procedures and the human proficiency level.
  • the present invention provides a method of separating regulation processes relating to human activity and mood states, In response to measurement of Inter-saccadic intervals In eye movement and reactions of activation In galvanic skin response.
  • the present invention provides a method of estimating human activity including measuring tnter-saccadic intervals in eye movement, and removing a contribution due to mood states in response to measurement of reactions of activation in galvanic skin response.
  • FIG. 1 is a flowchart representing program procedures of a digital computer according to an embodiment of the present invention.
  • FIG. 2 is a diagram of ISI duration for an estimation of dynamics of activity and mood states.
  • the present Invention provides a method and system for estimation of human activity and mood states based on registration of Inter Saccadic Intervals (ISI) In eye movements and Reactions of Activation of Galvanic Skin Response (GSR).
  • ISI Inter Saccadic Intervals
  • GSR Galvanic Skin Response
  • a method of estimation ⁇ f human activity by means of mental workload measurement is employed, including comparisons of registered ISI movements of eyes with the borders of ranges of ISI which characterize time scales of regulation processes at different levels having various contents and complexity.
  • Embodiments of the present Invention additionally provide an estimation of regulation of mood states by means of measurement of parameters of Reactions of Activation (RA) of GSR, comparison of ISI in eye movements and duration of RA GSR as a result of which separation In time of processes of regulation of activity and regulation of mood states is achieved.
  • RA Reactions of Activation
  • ISI duration for different levels of regulation process change In a certain range because of complexity of regulation processes (defined by their psychological substance and qualitative originality) is limited on the concrete level.
  • ISI range for levels of regulation process are equal for different people because it is based on common regularities of mental activity.
  • ISI Increase In scope with the Increasing level of brain's work load (from passive interaction to specific movement coordination, programmed task execution, individual psychosomatic changes and corrections for external influences and factors).
  • ISI ranges increase with growth of regulation levels (such as from direct
  • Total values of ISI ranges for all five regulation process levels In a preferred embodiment (from 0.03 to more then 30 seconds) can form a united or universal scale of human activity complexity. Borders of ranges on this scale can represent universal constants for regulation process levels.
  • ISI ranges can be validated during experimental investigations by means of a special method of qualitative psychological analysis of human activity and mood states.
  • This analysis can be based on data gathered from survey, interview, comments, thinking aloud.
  • embodiments of the present invention function as a prediction machine to determine the different levels of regulation processes in human activity and mood states. These levels can be defined on the basis of comparing registered ISI In eye movements and duration ⁇ f RA GSR with borders of ISI ranges. In this way, embodiments of the present invention make use of naturally occurring psycho-physiological processes that underlie the regulation processes of different levels (from direct interaction to mediate coordination, program-purpose organization, personality-normative changing and world outlook corrections) and characterize mental nature and complexity of regulation as human activity and as mood states.
  • Table 1 describes parameters of ISI taxon ⁇ , also described as ISI levels or ranges. These levels or ranges are used with presently preferred embodiments of the present invention, through other levels or ranges can alternatively be used.
  • Regulation processes of human activity at the second level Mediate Coordination - Include orientation In the external situation and logical coordination of activity on the basis of space-temporary Images, and processes of mood state regulation. Examples include: diagnostics of functioning of separate systems of an organism; the control and estimation of a degree of activity of behaviour; weariness and an emotional condition (excitation or depression); overcoming of weariness, physical discomfort, stress, emotional reactions and moods (in mimicry, gestures, speech).
  • Ievsl - Program-purpose Organization - regulation processes of human activity include searching and defining purposes and programs of activity, test and evaluation of Its total particularities. Examples include: organization and planning of activity as a whole; regulation processes of mood states understanding of state of health; an opportunity to preserve or change of an emotional condition and mood; a degree of activity of behaviour, such as In formation of the purposes and programs of restoration ⁇ r maintenance of capacity to work, mood, state of health.
  • Corrections - are connected with the addition and changing of system of professional knowledge and belief, and processes of mood states regulation. Examples include changing and transformation of system of knowledge and belief about human opportunities and his reserves, resources, about restoration or maintenance of capacity to work, mood, state of health.
  • ISl and RA GSR can reflect regulation processes of activity and mood states. Their joint measurement can overcome restrictions of methods of estimation of human activity based solely on taxonomy of ISI in eye movements and, also permit the separation of the processes of regulation of human activity and mood states.
  • Imperfection of control system Interface defects in the interface of computer programs, control panels, workplaces of operators, Information on displays and control handles, interfaces of computer programs. Therefore, an estimation of a degree of erg ⁇ nomic efficiency of a human-machine Interface and software usability can be carried out by means of comparison of total complexity and workload of activity for different options of the interlace and the software.
  • the movement of eyes Is registered by a common electroocubgrarnic method (EOG).
  • EOG electroocubgrarnic method
  • the GSR parameter is a skin resistance which can be measured by an electric amplifier. This can be done by way of 2 electrodes located on fingers of a human hand between which a weak electric current Is passed (about 5 mA).
  • the skin resistance Is relatad to the emission of perspiration and may vary over a wide range from 10 kOhm up to 2 mOhr ⁇ depending on mood states and the locations of GSR sensors on a human body.
  • Embodiments of the present Invention can be Implemented in software, or in hardware, and typically In a combination thereof.
  • a digital computer routine 10 inputs EOG data from a digital output of an EOG amplifier Into a memory buffer ordered by time-code.
  • a digital computer routine 11 categorizes the eye sight state as saccade or fixation, eye blink or slow movement (drift) based on the received EOG data.
  • Saccadic jumps may be voluntary or involuntary. Both of these saccades move the eyes of the viewer between fixations, or return them on the fixation if eyes drift from it. Saccades are sudden and rapid ballistic movements lasting about 10 to 120 milliseconds and traversing from less than 1 to 40 degrees. The peak velocity for a 10 degree saccade can reach 450 degrees per second; the peak velocity for a 20 degree saccade can range from 290 to 400 degrees per second.
  • fixation between two saccades is a comparatively brief period of relative stability which allows the portion of the scene within the fovea! vision to be studied.
  • fixation period Is typically from 300 to 600 milliseconds.
  • Small jittery motions occur during a fixation. These motions comprise slight drifts with amplitudes less than one degree, followed by microsaccadlc corrections.
  • Blinks usually last from 20 to 200 milliseconds and do not coincide with saccades. (00503).
  • more complex regulation processes exist such as when a human imagines some logic picture, makes an important responsible decision, or thinks about an unusual idea.
  • the routine 11 uses a velocity method for saccade detection.
  • the eye movement velocity is computed from the EOG data by first smoothing with a low* pass digital filter to remove noise and then numerically differentiating the filter output
  • the computed velocity is compared to an empirically determined threshold level (e.g.5 degrees per second) separating saccadic movements from fixations and drifts. If the computed velocity Is above the threshold, it Is determined as saccadic, whereas IfIt is less than the threshold, it ts considered as fixation ⁇ r drift;.
  • a special scheme ⁇ f analysis is used for blink detection.
  • routine 12 determines the start and end of the saccade, computes the Interval between this saccade and previous one - Inter Saccadic Interval (ISI) - and stores these parameter values In memory. Routine 13 then realizes ISI taxonomy by comparing with the borders of ISI ranges.
  • ISI Inter Saccadic Interval
  • ISI duration increases In accordance with the complexity of regulation processes on different levels, and that It can change from 0.03 to more than 30 seconds.
  • ISI duration can change within a certain range ortaxon because of complexity of regulation processes (defined by their mental nature and qualitative originality). This change can be limited on the concrete level.
  • This ISI range Is common for different people because It is based on universal regularities of thinking activity.
  • ISI ranges increase with growth of regulation process levels (from direct: interaction to mediate coordination, program-purpose organization, personality- normative changing and world outlook corrections) in accordance with complication of their mental nature.
  • Total values of ISI taxons for all five regulation process levels can form a united or unified scale of human activity complexity. Range borders on this scale represent universal constants for every regulation processes level (see Table 1).
  • a routine 26 computes this diagram after comparison of ISI and RA GSR in real time and displays it on the computer monitor.
  • a computer routine 15 inputs GSR data from the measuring equipment into a memory buff sr ordered by time-code.
  • GSR signals tt is possible to distinguish between Reactions of Activation and Reactions of the Relaxation. It is known that regulation of mood states occurs with respect to human activation when an increase of pulses in the nervous endings in the top layers of skin is observed, thereby increasing intensity of perspiration emissions in perspiration channels. This can lead to fast reduction In an electric resistance value of skin, which can be represented by a pulse of some seconds in duration and defined as Reaction of Activation (RA).
  • RA Reaction of Activation
  • Reactions of Activation of GSR can reflect activation psycho-physiological , emotional and mental (cognitive) processes In each of these levels.
  • So 1 at the first level - Direct Interaction - Reaction of Activation GSR can be connected t ⁇ occurrence ⁇ f internal physiological sensations, maintenance of physical efforts, accuracy and speed of movements.
  • the second level - Mediated Coordination - they can reflect change of activity of behaviour and an emotional state, overcoming of weariness and physical discomfort, stress.
  • these reactions can be connected to activation of mental processes for understanding of state of health, change of an emotional state and mood, formation of the purposes and programs of restoration or maintenance of capacity to work, mood, state of health.
  • the fourth level - Personality-normative changing - Reactions of Activation of GSR can reflect mental processes for formation and changing of personal norms of restoration or maintenance of capacity to work, mood, state of health.
  • the fifth level - World DUtlook corrections - the given reactions can be connected to the most complex mental processes for changing and transformation of system of knowledge and belief about human reserves, resources, opportunities of restoration or maintenance of capacity to worki mood, statB of health.
  • routine 16 Is determined In response to a decrease In value of electroskln resistance, and the end - on Its Increase (it considers a significant change of resistance more than 1% from its current value). Then routine 17 checks a condition of presence of GSR Reaction of Activation; If this reaction Is absent, there is a return to Input a new portion of GSR data in routine 15. In a case where GSR Reactions of Activation are found, routine 18 determines the moments of its beginning and ending, and saves these values In computer memory.
  • a computer routine 19 compares the time of the beginning and the ending of the determined ISI (received routine 13), with the presence of the given time interval of GSR Reaction of Activation. An ISI is considered coincident with RA GSR If Reactions of Activation get between saccades, forming given ISI. Further, routine 20 checks a condition Df coincidence, or correspondence, of ISI and GSR Reactions of Activation. [0064] If this coincidence is absent, routine 21 calculates values of the current complexity and workload of activity. If coincidence exists, routine 23 compares taxon numbers of ISI and GSR RA. In case of coincidence, routine 24 calculates values of the current complexity of both workload and mood states, and in case of discrepancy there is a return to procedure 21.
  • Criteria of the current complexity and workload of activity and mood states represent an estimation of activity and mood states by values of ISI of different regulation levels.
  • a problem In such estimation lies in the comparison ⁇ f the qualitative specificity of ISI 1 concerning different taxons.
  • quality of some regulation processes change when a maximum or upper border of a corresponding taxon is exceeded. Therefore, quality of different regulation levels can be taken into account by using weight factors representing the relations of the maximal border, or limit, of the certain tax ⁇ n to the maximal border of the first taxo ⁇ .
  • Routine 22 summarizes the received value of the current complexity and workload of activity with respect to the data received earlier. Routine 25 carries out similar operation for the current criteria of mood states.
  • routine 26 which represents registered ISI on a diagram of ISl duration.
  • This diagram can have the form of a sequence of columns with different widths and heights.
  • the width of a column can be based on, or related to, ISI duration value; the height of a column can also be based on ISI duration value, but in the other scale.
  • FIG. 2 a diagram of ISI duration Is provided in which the abscissa axis represents the activity time (in seconds), the right ordinate axis represents the ISI duration (also in seconds), and the left axis represents the taxon's numbers.
  • This diagram can also have the form of succession of columns of various widths and heights.
  • the column width can be based on the value of ISI duration; the column height can be based on the ISI duration also, but on a large scale. Fuzziness between taxons borders Is shown in FIG.2 as horizontal gray zones.
  • Routine 27 checks the condition of whether the criteria of the current complexity workload of activity and mood states should be presented t ⁇ the computer monitor as feedback to the human. If such a presentation or display is desired, there Is a transition to procedure 28 which presents these values to the monitor.
  • routine 29 which checks or determines whether controlled system modes should be Increased. If such an Increase Is desired, there Is a transition to routine 30 which checks whether the normative (threshold) value of criteria of the current complexity of activity and mood states has been exceeded. If the value has been exceeded, routine 31 carries out delivery of a command to increase controlled system modes. [0073] If the threshold value has not been exceeded, there Is a transition to routine 32 which checks whether a command relating to, or requesting, the termination of measurements has arrived from the computer keyboard. If such a command has not arrived, the method returns to a new cycle of measurements in routines 10 and 15.
  • routine 33 calculates total criteria - total complexity of both workload activity and mood states.
  • Total complexity of activity or mood states represent total values of corresponding valu ⁇ s of the current complexity of activity and mood states from routines 22 and 25, divided by, or over, the total number of ISI:
  • Total workload of activity or mo ⁇ d states represents total values of corresponding values of current workload of activity and mood states from routines 22 and 25:
  • W total workload activity or mood states
  • W current workload of activity or mood states for i ISI, N -total number of ISI.
  • routine 33 After calculation of total values of these criteria in routine 33, subsequent routines are Intended for achievement of the separate purposes of the Invention relating to estimation of a human proficiency level, complexity of human controlled procedures, influence of extreme fiactors, degree of erg ⁇ nomic perfection of the human-machine interface. Routines for each of these purposes are carried out Independently and separately from each other based on the presence of a corresponding condition. After completion of each of these routines, the program reaches termination.
  • routine 34 checks whether an estimation of a human proficiency level Is desired. If such an estimation is desired, routine 35 determines this parameter by means of comparison of total complexity and workload of activity from routine 33 with their normative values received based D ⁇ the data for experienced operators on simulators or In real conditions. This information can be inputted into the program separately, either in advance or on demand.
  • Routine 36 checks whether an estimation of complexity of human controlled procedures Is desired. If such an estimation Is desired, routine 37 estimates this complexity by means of addition of the received value of total complexity of activity to a file of values of, or related to, this criterion. These values can be received for the group of experienced operators, whose activity is considered as a reference, and can be inputted into the program either in advance or on demand. The received data Is subsequently averaged. [0079] Routine 38 checks whether an estimation of Influence of external extreme factors on activity and mood states Is desired. If such an estimation is desired, routine 39 determines a degree of the specified influence by means of comparison of the received values of total complexity and workload of activity and regulation of mood states with their values for the normal conditions. The information relating to normal conditions can be
  • Routine 40 checks whether an estimation of a degree ergo ⁇ omic effectiveness of the human-machine interface or software usability is desired. If such an estimation Is desired, routine 41 determines this parameter by means of comparison of the received values of total complexity and workload of activity with the similar data received for other options of the Interface.
  • Embodiments of the present Invention can provide one or more of the following advantages and/or benefits:
  • Reactions of Activation of GSR can separate In time and adequately estimate processes of human activity regulation. These processes can be connected with realization of the tasks, and processes of regulation of mood states, connected with Influence of external, Including extreme factors (stress, weariness, emotional reactions, physical efforts, etc.).
  • a method for estimation of human activity and mood states can determine a plurality of levels (e.g. five levels) of regulation processes, each having different contents and complexity.
  • a method for estimation of human activity and mood states can show dynamics of regulation processes as ISI sequence, relating to different levels and which taxons coincide and do not coincide with taxons of corresponding RA GSR duration. This can be provided as a diagram of ISI online, such as on a display means.
  • a method for estimation of human activity and mood states can estimate a current and total complexity and workload of activity automatically and objectively.
  • a method for estimation of human activity and mood states according to an embodiment of the present invention can estimate a current and total complexity and workload of regulation of mood states automatically and objectively.
  • a method for estimation of human activity and mood states according to an embodiment of the present invention can estimate a currant complexity and workload of activity and regulation of mood states online.
  • a method for estimation of human activity and mood states can evaluate the human proficiency level on simulators and In real conditions automatically and objectively.
  • a method for estimation of human activity and mood states can estimate a complexity of human controlled procedures automatically and objectively.
  • a method for estimation of human activity and mood states according tD an embodiment of the present invention can estimate a Influence of extreme factors on activity and mood states automatically and objectively.
  • a method for estimation of human activity and mood states can estimate a degree of erg ⁇ omic effectiveness of a human-machine interface automatically and objectively.
  • a method for estimation of human activity and mood states can estimate software usability automatically and objectively.
  • a method far estimation of human activity and mood states can provide feedback relating to the current complexity and workload of activity and regulation of mood states online.
  • a method for estimation of human activity and mood states can change human controlled system modes based on values of the current complexity and workload of human activity or regulation of mood states.
  • Some embodiments of the present invention are provided as a system or means for estimation of human activity and m ⁇ od states.
  • a system or means can include some of the following exemplary components:
  • [00104J) Means for showing human activity dynamics as a sequence of ISI, relating to different levels and which taxons coincide and do not coincide with taxons of corresponding RA GSR duration, as a diagram of ISI on-line.
  • This means can comprise a display means.
  • [00112] Means for computing software usability by comparing a total complexity and workload of human activity for different options of software.
  • Means for presentation on a computer monitor the value of the current workload of human activity and mood states on-line as feedback to a human subject [001143 ?• Means for changing human controlled system modes In response to the current workload of human activity or mood states regulation when they reach a threshold quantlty.
  • video or cine-camera means for measurement of eye movements are as follows: video or cine-camera means for measurement of eye movements; photo-electrical means with infra-red source for measurement of eye movements; photo-optical means with mirror sensor for measurement of eye movements; electromagnetic sensor means for measurement of eye movements; means of estimating the value of ISI of eye movements using measured signals from a human face; and/or means for GSR measurement based on a value of electric conductivity, resistance or a potential difference.
  • a method for an estimation of human activity and mood states can be implemented by means of measurement Inter Saccadic Intervals (ISI) In eye movements and Reactions of Activation (RA) of GSR.
  • ISI Inter Saccadic Intervals
  • RA Reactions of Activation
  • [00135] 18. estimating of software usability by means of comparison of total complexity and workload activity for different options of the software; [00136] 19. displaying such as on a computer monitor, of values of current complexity and workload of activity and regulation of mood states as a feedback signal; and/or [00137] 20. Increasing human controlled system modes when values of the current complexity and workload of activity or current workload of regulation mood states reach, exceed or fall below threshold values
  • a method for estimating human activity and mood states by means of measurement of time intervals between rapid (saccadic) movements of eyes (Inter Saccadic Intervals - ISl) and Reactions of Activation (RA) of a Galvanic Skin Response (GSR), which are used as indicators of different levels of human regulation processes.
  • the basis of the method Is the generality (commonality, universality) of mental regulation processes ⁇ f human activity and mood states expressed in uniform time scales of these processes at different levels. Different psychophysiological parameters can be used for an estimation of human activity and mood states accordingly.
  • the method can be used for human activity and mood state analysis in different scenarios and for different purposes, such as for an estimation of a human proficiency level on simulators and in real conditions; for an estimation of complexity ⁇ f human controlled procedures; for an estimation of Influence of external extreme factors on activity and mood states; for an estimation of usability of Interfaces of control systems and computer programs, etc.

Abstract

A method and system are provided for estimation of human activity and mood states based on registration of Inter Saccadic Intervals (ISl) In eye movements and Reactions of Activation of Galvanic Skin Response (GSR) . A method of a estimation of human activity by means of mental workload measurement Is employed, including compailsons of registered ISI movements of eyes with the borders of ranges of ISI which characterize time scales of regulation processes at dltlerent levels of various contents and complexity. Some embodiments additionally provide an estimation of regulation of mood states by means of measurement of parameters of Reactions of Activation (RA) of GSR, comparison of ISI In eye movements and duration of RA GSR as a result of which separation In time of processes of regulation of activity and regulation of mood states Is achieved.

Description

SYSTEM AND ME=THOD FOR ESTIMATION OF HUMAN ACTIVITY AND MOOD STATES BASED ON RAPID EYE MOVEMENTS AND GALVANIC SKIN RESPONSE
FIELD OF THE INVENTION
[0001] The present invention relates generally to mental workload (or human activity) measurement and eye movement rBssarch. More particularly, the present Invention relates to human activity measurement, such as for operator training* and mood state estimation based on rapid eye movement
BACKGROUND OF THE INVENTION
[0002] Eye movement analysis and Galvanic Skin Response (GSR) have traditionally been used for human activity estimation.
[0003] Different methods of eya movement analysis can be used for eye movement measurement, such as: electrooculogramic, electromagnetic, photo-electrical with infra-red source, photo-optical with mirror sensor, video or cine-camera means, etc. Such methods can analyze view paths, eye fixation on different external objects, and saccadic movements due to a change of fixation on objects. This can reveal the main stages of the activity, structure of perceptual processes of information searching, moments of attention and difficulties, which then link with reasoning processes. Use of these methods permits estimation of human activity when a subject is working with different options of indicators and interfaces, researching a nature of information processes, attention and reasoning. [0004] Methods of GSR typically measure resistance, conductivity or potential difference of the human skin. They permit the estimation of changes in human mood states. Amplitude and phase parameters of GSR can be analyzed, and Reactions of Activation and Relaxation of GSR can be determined. GSR Reactions of Activation are connected with regulation of human mood states when there is a fast change of GSR amplitude. However, the GSR Reactions of Relaxation can relate to a slow change of GSR amplitude. If measurement of skin resistance (or potential difference) is used, GSR Reaction of Activation Is associated with a fast decrease of the skin parameter. Using Measurement of skin resistance, GSR Reaction of Relaxation is associated with a slow increase of the skin parameter. Using measurement of skin conductivity, GSR Reaction of Activation Is associated with a fast Increase of the skin conductivity and QSR Reaction of Relaxation is associated with a slow decrease of the skin conductivity. The use of GSR method permits analysis of various human mood states that can be influenced by extreme factors such as stress, weariness, illness, emotions, physical efforts, etc.
[0005] Some known methods estimate human activity by comparing time Intervals between saccadic eyes movements (Inter Saccadic Intervals - ISI) with certain ranges (taκons) of ISI which characterize regulation levels of human activity having different contents and complexity. The method Is based on the psychα-physiDlαgica! evidence for the relationships between ranges of ISI changing and different levels of regulation processes. As criteria of an estimation of activity, both the current and total workload are used. The method permits an estimation of human activity In various control systems, such as on simulators. It also provides the ability to estimate a complexity of human controlled procedures and the human proficiency level.
[0006] It has been discovered that prior methods may not be able to separate the factors which influence human activity from the factors which influence human mood states. In terms of resolving tasks In an activity, the influence of external extreme factors, weariness, physical efforts, emotional reactions or illness can be overcome by means of common (universal) regulation processes. Thus there are problems connected not with the complexity Df the activity, but with complexity of mood states. Known methods are not directed towards the separation of regulation processes concerning activity and mood states. [0007] It Is, therefore, desirable to provide additional means reflecting specificity of mood states regulation In human activity analysis.
SUMMARY OF THE INVENTION
[0008] Jt Is an object of the present invention to obviate or mitigate at least one disadvantage of previous human activity analysis approaches, r.0009] In a first aspect, the present invention provides a method of separating regulation processes relating to human activity and mood states, In response to measurement of Inter-saccadic intervals In eye movement and reactions of activation In galvanic skin response. [0010] in another aspect, the present invention provides a method of estimating human activity including measuring tnter-saccadic intervals in eye movement, and removing a contribution due to mood states in response to measurement of reactions of activation in galvanic skin response.
[0011J Other aspects and features of the present invention will became apparent to those ordinarily skilled In the art upon review of the following description of specific embodiments of the Invention in conjunction with the accompanying figures.
BRIEF DESCRIPTION OFTHE DRAWINGS
[0012] Embodiments of the present invention will now be described, by way of example only, with reference to the attached Figures, wherein:
[0013] FIG. 1 is a flowchart representing program procedures of a digital computer according to an embodiment of the present invention; and
[0014] FIG. 2 is a diagram of ISI duration for an estimation of dynamics of activity and mood states.
DETAILED DESCRIPTION
[0015] Generally, the present Invention provides a method and system for estimation of human activity and mood states based on registration of Inter Saccadic Intervals (ISI) In eye movements and Reactions of Activation of Galvanic Skin Response (GSR). A method of estimation σf human activity by means of mental workload measurement is employed, including comparisons of registered ISI movements of eyes with the borders of ranges of ISI which characterize time scales of regulation processes at different levels having various contents and complexity. Embodiments of the present Invention additionally provide an estimation of regulation of mood states by means of measurement of parameters of Reactions of Activation (RA) of GSR, comparison of ISI in eye movements and duration of RA GSR as a result of which separation In time of processes of regulation of activity and regulation of mood states is achieved.
[0016] The concept behind embodiments of the present invention is based on a generality (universality) of processes of mental regulation of activity and regulation of mood states. These processes can be expressed In uniform time scales of processes of regulation at different levels, which allows comparison of psycho-physiological parameters used for an estimation of activity and mood states. There is also some psycho-physiological evidence of connection of ranges of ISI changing with different levels of regulation processes and the well-known connection of GSR Reactions of Activation with regulation of mood states.
[0017] There are certain factors or properties that contribute to the viability of embodiments of the present Invention. These can be summarized as follows;
[0018] 1. ISI duration for different levels of regulation process change In a certain range because of complexity of regulation processes (defined by their psychological substance and qualitative originality) is limited on the concrete level.
[0019] 2. ISI range for levels of regulation process are equal for different people because it is based on common regularities of mental activity.
[0020] 3. ISI Increase In scope with the Increasing level of brain's work load (from passive interaction to specific movement coordination, programmed task execution, individual psychosomatic changes and corrections for external influences and factors).
[0021] 4. ISI ranges increase with growth of regulation levels (such as from direct
Interaction to mediate coordination, program-purpose organization, personality-normative changing and world outlook corrections) in accordance with complication of their mental nature.
[0022] 5. Total values of ISI ranges for all five regulation process levels In a preferred embodiment (from 0.03 to more then 30 seconds) can form a united or universal scale of human activity complexity. Borders of ranges on this scale can represent universal constants for regulation process levels.
[0023] 6, ISI ranges can be validated during experimental investigations by means of a special method of qualitative psychological analysis of human activity and mood states.
This analysis can be based on data gathered from survey, interview, comments, thinking aloud.
[0024] 7. The comparison of registered ISI with the borders of their ranges permits an estimation of the regulation process level during this time Interval.
[0025] 8. Regulation of mood states correlates with the GSR Reactions of Activation expressed In fast decreasing of electric resistance of human skin. [0026] 9. Correlation of ISI eye movement and RA GSR, which depends on the overall extent of brain work load and mood states of the subject, allows the separation of the defined processes on the time scale.
[0027] 1 D. Comparison of ISI in eye movements and RA GSR is possible because of a universality of mental regulation processes of activity and mood states. This allows the separation of these processes ϊπ time.
[0028] 11. Successive changing of regulation processes levels on the different time intervals allows to show the dynamics of regulation of human activity and mood states. [0029] Conceptually, embodiments of the present invention function as a prediction machine to determine the different levels of regulation processes in human activity and mood states. These levels can be defined on the basis of comparing registered ISI In eye movements and duration αf RA GSR with borders of ISI ranges. In this way, embodiments of the present invention make use of naturally occurring psycho-physiological processes that underlie the regulation processes of different levels (from direct interaction to mediate coordination, program-purpose organization, personality-normative changing and world outlook corrections) and characterize mental nature and complexity of regulation as human activity and as mood states.
[0030] Table 1 describes parameters of ISI taxonε, also described as ISI levels or ranges. These levels or ranges are used with presently preferred embodiments of the present invention, through other levels or ranges can alternatively be used.
Figure imgf000007_0001
Table 1 [0031] At the first level -Direct Interaction - mental nature of human activity regulation processes Is concluded on the basis of Immediate sensible-practical contact of the human with reality. Examples Include: reaction to events; actualization of sensor-perceptional Images and processes of the perception and realization of motor actions; processes of mood state regulation - Jn distinction and an Identification of Internal physiological sensations (pain, heat, a cold, weight, etc.); maintenance of physical efforts; accuracy and speed of movements; and performance of sequence of movements or maintenance of their necessary rate.
[0032] Regulation processes of human activity at the second level - Mediate Coordination - Include orientation In the external situation and logical coordination of activity on the basis of space-temporary Images, and processes of mood state regulation. Examples Include: diagnostics of functioning of separate systems of an organism; the control and estimation of a degree of activity of behaviour; weariness and an emotional condition (excitation or depression); overcoming of weariness, physical discomfort, stress, emotional reactions and moods (in mimicry, gestures, speech).
[0033] At the third Ievsl - Program-purpose Organization - regulation processes of human activity Include searching and defining purposes and programs of activity, test and evaluation of Its total particularities. Examples include: organization and planning of activity as a whole; regulation processes of mood states understanding of state of health; an opportunity to preserve or change of an emotional condition and mood; a degree of activity of behaviour, such as In formation of the purposes and programs of restoration αr maintenance of capacity to work, mood, state of health.
[0034J At the fourth level -* Personality-normative Changing - regulation processes of human activity are connected with changing of the way of performing an activity that Is accepted by tha human. -Examples Include: criteria, values and norms of Its realization, and processes of mood states regulation - in formation and changing of norms and criteria of restoration or maintenance of capacity to work, mood, state of health. [0035] Regulation processes of human activity at the fifth level - World Outlook
Corrections - are connected with the addition and changing of system of professional knowledge and belief, and processes of mood states regulation. Examples Include: changing and transformation of system of knowledge and belief about human opportunities and his reserves, resources, about restoration or maintenance of capacity to work, mood, state of health.
[0036] As levels Increase, an Increasing complexity of regulation processes as human activity and as mood states can be observed. As a result, complexity of activity and complexity of mood states regulation can be estimated based on a degree of actualization of different levels of processes of regulation. The Workload can be estimated based DΠ duration of these processes. Thus quantitative criteria of the current and total complexity and workload of activity can be calculated based on: ISI data; which taxons do not coincide with taxons of RA GSR; quantitative criteria of the current and total complexity and workload regulation mood states; and on data ISI, Including which taxons coincide with taxons of RA
GSR.
[0037] Hence, ISl and RA GSR can reflect regulation processes of activity and mood states. Their joint measurement can overcome restrictions of methods of estimation of human activity based solely on taxonomy of ISI in eye movements and, also permit the separation of the processes of regulation of human activity and mood states.
[0038] It Is known that in the process of training a reduction of regulation levels occurs. For example, if high regulation levels are actualized at the beginning of training and
ISIs of high taxons occur, then In the final stage low levels and ISIs of low taxons occur. This leads to a reduction of values of the complexity and workload of human activity. Therefore the comparison of its data with respective normative values permits an estimation of a human proficiency level in simulators and real conditions.
[0039] An Increase of regulation levels and ISI taxon numbers occurs in response to an increase in complexity of human controlled procedures. To avoid Influence of an insufficient proficiency level, complexity of human controlled procedures can be estimated by means of calculation of an average value of total complexity of activity fora group of experienced operators, which activity should be considered as a reference.
[0040] Also it is known that higher regulation levels of activity and mood states actualization are typically required to overcome strong external factors. Thus, there are difficulties connected to occurrence of adverse mood states (weariness, excitation, depression, physical discomfort, stress) and corresponding deterioration of processes of activity. Therefore by means of comparison of total complexity and workload activity and regulation mood states with their values for normal conditions, It Is possible to estimate a degree of Influence of external conditions on activity and mood states. [0041] With respect to the complicating iactors of human activity, these can Include:
Imperfection of control system Interface, defects in the interface of computer programs, control panels, workplaces of operators, Information on displays and control handles, interfaces of computer programs. Therefore, an estimation of a degree of ergαnomic efficiency of a human-machine Interface and software usability can be carried out by means of comparison of total complexity and workload of activity for different options of the interlace and the software.
[0042] Calculation of values of the current complexity and workload of activity and regulation of mood states on-line permits the display σf such values on a computer monitor; this can advantageously be used to organize or provide a biofeedback. [0043] Complex situations of control (situation awareness) or influence of extreme conditions can occur when a human does not cαpβ with the decision of the arisen problems and cannot overcome adverse mood states. Controlled system modes can be increased (voluntarily, automatically, and/or In a compulsory manner) for maintenance of reliability and safety of control by technical systems. This Increasing of controlled system modes can be carried out In response tα achievement of threshold values of the current complexity and workload αf activity and mood states regulation.
[0044] In a presently preferred embodiment of the present Invention, the movement of eyes Is registered by a common electroocubgrarnic method (EOG). [0045] The GSR parameter is a skin resistance which can be measured by an electric amplifier. This can be done by way of 2 electrodes located on fingers of a human hand between which a weak electric current Is passed (about 5 mA). The skin resistance Is relatad to the emission of perspiration and may vary over a wide range from 10 kOhm up to 2 mOhrπ depending on mood states and the locations of GSR sensors on a human body. [0046] Embodiments of the present Invention can be Implemented in software, or in hardware, and typically In a combination thereof. A method according to an embodiment of the present invention will now be discussed. However, It is to be understood that this method can be performed by software according to an embodiment of the present invention, or stored DΠ a computer-readable medium or embodied in a carrier signal which, when executed, performs the steps of the method. Methods according to embodiments of the present invention can Include same, many, or all of the routines or steps described herein. [0047] Referring to FIG. 1, a digital computer routine 10 inputs EOG data from a digital output of an EOG amplifier Into a memory buffer ordered by time-code. A digital computer routine 11 categorizes the eye sight state as saccade or fixation, eye blink or slow movement (drift) based on the received EOG data.
EQ048] Saccadic jumps may be voluntary or involuntary. Both of these saccades move the eyes of the viewer between fixations, or return them on the fixation if eyes drift from it. Saccades are sudden and rapid ballistic movements lasting about 10 to 120 milliseconds and traversing from less than 1 to 40 degrees. The peak velocity for a 10 degree saccade can reach 450 degrees per second; the peak velocity for a 20 degree saccade can range from 290 to 400 degrees per second.
[0049] For sensor-perceptual processes the fixation between two saccades is a comparatively brief period of relative stability which allows the portion of the scene within the fovea! vision to be studied. In this case the fixation period Is typically from 300 to 600 milliseconds. Small jittery motions occur during a fixation. These motions comprise slight drifts with amplitudes less than one degree, followed by microsaccadlc corrections. Blinks usually last from 20 to 200 milliseconds and do not coincide with saccades. (00503 However, more complex regulation processes exist, such as when a human imagines some logic picture, makes an important responsible decision, or thinks about an unusual idea. In such a case, the subject may mentally remove himself from surrounding situation (situational awareness) and Immerse in his Inner world. In these cases human eyes can slowly move In some winding path with saccades depression and intervals between saccades (ISI) possibly being very long, for example lasting 30 seconds and more. [0051] The routine 11 uses a velocity method for saccade detection. In this method, the eye movement velocity is computed from the EOG data by first smoothing with a low* pass digital filter to remove noise and then numerically differentiating the filter output The computed velocity is compared to an empirically determined threshold level (e.g.5 degrees per second) separating saccadic movements from fixations and drifts. If the computed velocity Is above the threshold, it Is determined as saccadic, whereas IfIt is less than the threshold, it ts considered as fixation αr drift;. For blink detection, a special scheme αf analysis is used.
[0052] If a saccade Js found, routine 12 determines the start and end of the saccade, computes the Interval between this saccade and previous one - Inter Saccadic Interval (ISI) - and stores these parameter values In memory. Routine 13 then realizes ISI taxonomy by comparing with the borders of ISI ranges.
[0053] There Is experimental evidence that ISl duration increases In accordance with the complexity of regulation processes on different levels, and that It can change from 0.03 to more than 30 seconds. For every level of regulation process, ISI duration can change within a certain range ortaxon because of complexity of regulation processes (defined by their mental nature and qualitative originality). This change can be limited on the concrete level. This ISI range Is common for different people because It is based on universal regularities of thinking activity. Moreover, ISI ranges increase with growth of regulation process levels (from direct: interaction to mediate coordination, program-purpose organization, personality- normative changing and world outlook corrections) in accordance with complication of their mental nature. Total values of ISI taxons for all five regulation process levels (from 0.03 to more than 3D seconds) can form a united or unified scale of human activity complexity. Range borders on this scale represent universal constants for every regulation processes level (see Table 1).
[0054] The borders between neighbour taxons are fuzzy, overlap and are clearly not separable. The maximum (right) border previous taxon is more than the minimum (left) border of the next one. Therefore it Is Impossible to define the ISI from overlap borders ranges uniquely between neighbour taxαns. This can be taken Into account In the case of an estimation of accuracy αf used criteria.
[0055] The values of ISI ranges validated during experimental Investigations by means of qualitative psychological activity analysis with data of survey, interview, comments, thinking aloud and definition αf problemacities. In this analysis the problemacity Is a special notion for the description of uncertainties, ambiguities and difficulties in activity, appearing on different levels of regulation processes. Definition of the problemacity type on some temporary Interval of activity permits reference to the occurrence of regulation processes, or to, a certain level. [0056] A computer routine 14 realizes ISI taxonomy by comparing the registered ISI with the fixed borders of their five taxoπs. This permits estimation of the regulation process level during this time interval In accordance with the taκon's number. The successive changing of regulation process levels on the different time intervals permits display of the dynamics of human activity and regulation of mood states as the diagram αf ISI duration. A routine 26 computes this diagram after comparison of ISI and RA GSR in real time and displays it on the computer monitor.
[0057] A computer routine 15 inputs GSR data from the measuring equipment into a memory buff sr ordered by time-code.
[0058] In GSR signals tt is possible to distinguish between Reactions of Activation and Reactions of the Relaxation. It is known that regulation of mood states occurs with respect to human activation when an increase of pulses in the nervous endings in the top layers of skin is observed, thereby increasing intensity of perspiration emissions in perspiration channels. This can lead to fast reduction In an electric resistance value of skin, which can be represented by a pulse of some seconds in duration and defined as Reaction of Activation (RA).
[0059] A Reaction of Relaxation Is observed In relation to an absence of regulation of mood states In GSR. This is reflected In a slow Increase of skin electric resistance and can last up to several tens of seconds.
[0060] In connection with that regulation αf mood states (which can occur on each of five levels), Reactions of Activation of GSR can reflect activation psycho-physiological , emotional and mental (cognitive) processes In each of these levels. So1 at the first level - Direct Interaction - Reaction of Activation GSR can be connected tα occurrence αf internal physiological sensations, maintenance of physical efforts, accuracy and speed of movements. At the second level - Mediated Coordination - , they can reflect change of activity of behaviour and an emotional state, overcoming of weariness and physical discomfort, stress. At the third level -the Program-purposed organization -, these reactions can be connected to activation of mental processes for understanding of state of health, change of an emotional state and mood, formation of the purposes and programs of restoration or maintenance of capacity to work, mood, state of health. At the fourth level - Personality-normative changing -, Reactions of Activation of GSR can reflect mental processes for formation and changing of personal norms of restoration or maintenance of capacity to work, mood, state of health. At the fifth level - World DUtlook corrections -, the given reactions can be connected to the most complex mental processes for changing and transformation of system of knowledge and belief about human reserves, resources, opportunities of restoration or maintenance of capacity to worki mood, statB of health. [0061] A computer routine 16 on received GSR data separates fast Reactions of
Activation from slow Reactions of Relaxation,
[0062] The beginning GSR Reaction of Activation In routine 16 Is determined In response to a decrease In value of electroskln resistance, and the end - on Its Increase (it considers a significant change of resistance more than 1% from its current value). Then routine 17 checks a condition of presence of GSR Reaction of Activation; If this reaction Is absent, there is a return to Input a new portion of GSR data in routine 15. In a case where GSR Reactions of Activation are found, routine 18 determines the moments of its beginning and ending, and saves these values In computer memory.
[0063] A computer routine 19 compares the time of the beginning and the ending of the determined ISI (received routine 13), with the presence of the given time interval of GSR Reaction of Activation. An ISI is considered coincident with RA GSR If Reactions of Activation get between saccades, forming given ISI. Further, routine 20 checks a condition Df coincidence, or correspondence, of ISI and GSR Reactions of Activation. [0064] If this coincidence is absent, routine 21 calculates values of the current complexity and workload of activity. If coincidence exists, routine 23 compares taxon numbers of ISI and GSR RA. In case of coincidence, routine 24 calculates values of the current complexity of both workload and mood states, and in case of discrepancy there is a return to procedure 21.
[0065] Criteria of the current complexity and workload of activity and mood states represent an estimation of activity and mood states by values of ISI of different regulation levels. A problem In such estimation lies in the comparison αf the qualitative specificity of ISI1 concerning different taxons. However, quality of some regulation processes change when a maximum or upper border of a corresponding taxon is exceeded. Therefore, quality of different regulation levels can be taken into account by using weight factors representing the relations of the maximal border, or limit, of the certain taxαn to the maximal border of the first taxoπ.
[0066] On the basis of this discussion, criteria of the current complexity and workload of activity and mood states can be represented by the following equations:
where: Q - the current complexity of activity or mood states for registered ISI1 i -registered ISI number, Bt - the maximal border of taxon to which belongs registered ISI1 81 - the maximal border of the 1-st taxoπ of ISL
where: W| - the current workload of activity or mood states for registered ISI1 i -registered ISi number, Bi - the maximal border of taxon to which belongs registered ISI, Bi - the maximal border of the 1-st taxoπ of ISI, tl - duration of registered ISI. [0067] Thus, complexity Is a relative criterion of activity and rnood states, and workload is an absolute criterion (measured In terms of time). [0068] Routine 22 summarizes the received value of the current complexity and workload of activity with respect to the data received earlier. Routine 25 carries out similar operation for the current criteria of mood states.
[0069] After routines 22 or 25 there Is a transition to routine 26 which represents registered ISI on a diagram of ISl duration. This diagram can have the form of a sequence of columns with different widths and heights. The width of a column can be based on, or related to, ISI duration value; the height of a column can also be based on ISI duration value, but in the other scale. Referring to FIG. 2, a diagram of ISI duration Is provided in which the abscissa axis represents the activity time (in seconds), the right ordinate axis represents the ISI duration (also in seconds), and the left axis represents the taxon's numbers. This diagram can also have the form of succession of columns of various widths and heights. The column width can be based on the value of ISI duration; the column height can be based on the ISI duration also, but on a large scale. Fuzziness between taxons borders Is shown in FIG.2 as horizontal gray zones.
[0070]' If ISI does not coincide with RA GSR with respect to regulation of activity, the values can be displayed on a diagram as a first representation, such as dark grey color columns. If ISI coincide, with RA GSR with respect to regulation of mood states, the values can be displayed on a diagram as a second representation, such as light grey color columns. [00713 Routine 27 checks the condition of whether the criteria of the current complexity workload of activity and mood states should be presented tα the computer monitor as feedback to the human. If such a presentation or display is desired, there Is a transition to procedure 28 which presents these values to the monitor. [0072] Then, as well as in a case when feedback is not necessary, there is a transition to routine 29 which checks or determines whether controlled system modes should be Increased. If such an Increase Is desired, there Is a transition to routine 30 which checks whether the normative (threshold) value of criteria of the current complexity of activity and mood states has been exceeded. If the value has been exceeded, routine 31 carries out delivery of a command to increase controlled system modes. [0073] If the threshold value has not been exceeded, there Is a transition to routine 32 which checks whether a command relating to, or requesting, the termination of measurements has arrived from the computer keyboard. If such a command has not arrived, the method returns to a new cycle of measurements in routines 10 and 15. [0074] If such command has arrived, there Is a transition to routine 33. This routine calculates total criteria - total complexity of both workload activity and mood states. Total complexity of activity or mood states represent total values of corresponding valuεs of the current complexity of activity and mood states from routines 22 and 25, divided by, or over, the total number of ISI:
Figure imgf000016_0001
where; C — total complexity of activity or mood states, Cj - the current complexity of activity or mood states for I ISI1 N - total number of ISI. [0075] Total workload of activity or moαd states represents total values of corresponding values of current workload of activity and mood states from routines 22 and 25:
Figure imgf000017_0001
where: W - total workload activity or mood states, W) - current workload of activity or mood states for i ISI, N -total number of ISI.
[0076] After calculation of total values of these criteria in routine 33, subsequent routines are Intended for achievement of the separate purposes of the Invention relating to estimation of a human proficiency level, complexity of human controlled procedures, influence of extreme fiactors, degree of ergαnomic perfection of the human-machine interface. Routines for each of these purposes are carried out Independently and separately from each other based on the presence of a corresponding condition. After completion of each of these routines, the program reaches termination.
[0077] So, routine 34 checks whether an estimation of a human proficiency level Is desired. If such an estimation is desired, routine 35 determines this parameter by means of comparison of total complexity and workload of activity from routine 33 with their normative values received based DΠ the data for experienced operators on simulators or In real conditions. This information can be inputted into the program separately, either in advance or on demand.
[0078] Routine 36 checks whether an estimation of complexity of human controlled procedures Is desired. If such an estimation Is desired, routine 37 estimates this complexity by means of addition of the received value of total complexity of activity to a file of values of, or related to, this criterion. These values can be received for the group of experienced operators, whose activity is considered as a reference, and can be inputted into the program either in advance or on demand. The received data Is subsequently averaged. [0079] Routine 38 checks whether an estimation of Influence of external extreme factors on activity and mood states Is desired. If such an estimation is desired, routine 39 determines a degree of the specified influence by means of comparison of the received values of total complexity and workload of activity and regulation of mood states with their values for the normal conditions. The information relating to normal conditions can be
Inputted into the program either in advance or on demand.
[0080] Routine 40 checks whether an estimation of a degree ergoπomic effectiveness of the human-machine interface or software usability is desired. If such an estimation Is desired, routine 41 determines this parameter by means of comparison of the received values of total complexity and workload of activity with the similar data received for other options of the Interface.
[0081] Embodiments of the present Invention can provide one or more of the following advantages and/or benefits:
[0082] 1. A method for estimation of human activity and mood states according to an embodiment of the present Invention based on measurements ISI In movements of eyes and
Reactions of Activation of GSR, can separate In time and adequately estimate processes of human activity regulation. These processes can be connected with realization of the tasks, and processes of regulation of mood states, connected with Influence of external, Including extreme factors (stress, weariness, emotional reactions, physical efforts, etc.).
[0083] 2. A method for estimation of human activity and mood states according to an embodiment of the present Invention can determine a plurality of levels (e.g. five levels) of regulation processes, each having different contents and complexity.
[0084] 3. A method for estimation of human activity and mood states according to an embodiment of the present invention can show dynamics of regulation processes as ISI sequence, relating to different levels and which taxons coincide and do not coincide with taxons of corresponding RA GSR duration. This can be provided as a diagram of ISI online, such as on a display means.
[0085] 4. A method for estimation of human activity and mood states according to an embodiment of the present invention can estimate a current and total complexity and workload of activity automatically and objectively.
[0086] 5. A method for estimation of human activity and mood states according to an embodiment of the present invention can estimate a current and total complexity and workload of regulation of mood states automatically and objectively. [0087] 6. A method for estimation of human activity and mood states according to an embodiment of the present invention can estimate a currant complexity and workload of activity and regulation of mood states online.
[0088J 7. A method for estimation of human activity and mood states according to an embodiment of the present invention can evaluate the human proficiency level on simulators and In real conditions automatically and objectively.
[0089] S. A method for estimation of human activity and mood states according to an embodiment of the present invention can estimate a complexity of human controlled procedures automatically and objectively.
[0090] 9. A method for estimation of human activity and mood states according tD an embodiment of the present invention can estimate a Influence of extreme factors on activity and mood states automatically and objectively.
[0091 J 1 Q. A method for estimation of human activity and mood states according to an embodiment of the present invention can estimate a degree of ergαπomic effectiveness of a human-machine interface automatically and objectively.
[0092] 11 , A method for estimation of human activity and mood states according to an embodiment of the present invention can estimate software usability automatically and objectively.
[0093] 12. A method far estimation of human activity and mood states according to an embodiment of the present invention can provide feedback relating to the current complexity and workload of activity and regulation of mood states online.
[00943 13. A method for estimation of human activity and mood states according to an embodiment of the present invention can change human controlled system modes based on values of the current complexity and workload of human activity or regulation of mood states.
[0095] Some embodiments of the present invention are provided as a system or means for estimation of human activity and mαod states. Such a system or means can include some of the following exemplary components:
£0096] 1. Means for measurement of EOG signals, such as from a human face.
[0097] 2. Means for computing a value of ISI of eye movements using EOG signals. [0098] 3. Means for computing ISI taxonomy for determination of a level of human activity regulation for every ISI by comparing its value with the borders of ISI ranges. These ranges define the time scales of regulation processes as limits of ISI changing for respective levels of regulation of human activity with different substance and complexity.
[0099] 4. Means far measurement of GSR signals, such as from a human hand,
[00100] 5. Means for estimation of GSR Reactions of Activation.
[00101] 6. Means for computing parameters of GSR Reactions of Activation.
[00102] 7. Means for RA GSR taxonomy for definition of a level of regulation of mood states for every RA GSR by means of comparison of Its duration with the same borders of known ISI ranges (such as the five known ISI ranges).
[00103] 8. Means for comparison of ISI in eye movements and GSR Reactions of
Activation and comparison of their taxons.
[00104J 9. Means for showing human activity dynamics as a sequence of ISI, relating to different levels and which taxons coincide and do not coincide with taxons of corresponding RA GSR duration, as a diagram of ISI on-line. This means can comprise a display means.
[00105] 1 D. Means for computing the current and total complexity and workload of activity on ISI data, which taxons do not coincide with taxons of RA GSR.
[00106] 11. Means for computing the current and total complexity and workload of mood states regulation on ISl data, which taxons coincide with taxons of RA GSR.
[00107] Some particular systems or means according to embodiments Df the present invention can include some of the following exemplary, yet optional, components:
[00108] 1. Means for computing a human proficiency level in simulators and real conditions by comparing a total complexity and workload of human activity with Its normative values.
[00109] 2, Means for computing the complexity of human controlled procedures by computing an average value of total complexity of human activity for experienced operators
[00110] 3. Means for estimation of extreme factors that Influence activity and mood states by means of comparison of total complexity and workload of activity and regulation of mood states with their values for normal conditions. [00111] 4. Means for computing ergoπomlc effectiveness of an interface of a human- machine system by comparing of the total complexity and workload of human activity for different options of Interface.
[00112] 5. Means for computing software usability by comparing a total complexity and workload of human activity for different options of software.
[00113] 6. Means for presentation on a computer monitor the value of the current workload of human activity and mood states on-line as feedback to a human subject [001143 ?• Means for changing human controlled system modes In response to the current workload of human activity or mood states regulation when they reach a threshold quantlty.
[00115] While many elements have been described In general as a means for performing a particular task or function, some of these elements can be provided according to exemplary embodiments of the present Invention using particular elements. Such particular elements which can preferably be used are as follows: video or cine-camera means for measurement of eye movements; photo-electrical means with infra-red source for measurement of eye movements; photo-optical means with mirror sensor for measurement of eye movements; electromagnetic sensor means for measurement of eye movements; means of estimating the value of ISI of eye movements using measured signals from a human face; and/or means for GSR measurement based on a value of electric conductivity, resistance or a potential difference.
[00116] While some specific Implementations have been described, it is to be understood that various alternatives can be implemented according to other embodiments αf the present invention. Some examples of alternatives Include: placement of GSR sensors at other locations on a human body; a different structure (number and definition) of human activity and mood state regulation levels; different values of borders of ISI ranges (taxons) for levels of regulation of human activity and mood states; different criteria for the current and total complexity and workload αf human activity on data of ISI; and/or different criteria for the current and total complexity and workload of mood states on data of ISI. [00117] A method for an estimation of human activity and mood states according to an embodiment of the present Invention can be implemented by means of measurement Inter Saccadic Intervals (ISI) In eye movements and Reactions of Activation (RA) of GSR. Such a method can Include same or many of the following:
[00118] 1. measuring EOG signals, such as from a human face;
[001193 2. determining saccades In EOG signals and estimating saccade start times;
[00120] 3. estimating Inter Saccadic Interval value (duration) and storing such value for calculation of criteria of activity and mood states;
[00121] 4. determining a regulation level of activity and mood states by calculation of taxon number for registered ISI and by means of comparison of Its duration with borders of
(five) ISI ranges (taxons);
[00122] 5. measuring GSR signals, such as from human hand;
[00123] 6. determining a GSR Reaction of Activation;
[00124] 7. determining parameters of GSR Reaction of Activation;
[00125] 3. determining a regulation level of mood states by calculation of taxon number for registered GSR Reaction of Activation. This can be achieved by means of comparison of Its duration with borders of five ISI ranges;
[00126] 9. determining of which ISl coincide and do not coincide with RA GSR;
[00127] 10. comparing taxon numbers of ISI and RA GSR for ISI which coincide with
RA GSR;
[00128] 11. showing or displaying dynamics of regulation processes of activity and mood states as sequence of ISI. This can relate to different regulation levels and which taxons coincide or do not coincide with taxons of RA GSR;
[00129] 12. calculating a current and total complexity and workload of activity on data of ISI, which taxons do not coincide with taxons of RA GSR;
[00130] 13. calculating a current and total complexity and of regulation mood states on data of ISl, which taxons coincide with taxons of RA GSR;
[00131] 14. evaluating a human proficiency level on simulators and In real conditions by means of comparison of total complexity and workload of activity with their normative values;
[00132] 15. estimating a complexity of human controlled procedures by means of calculation of average value of total complexity of activity for experienced operators; [00133] 16. estimating a degree of influence of extreme factors on activity and mood states by means of comparison of total complexity and workload of activity and regulation mood states with their values for normal conditions;
[00134] 17. estimating a degree of ergonomlc efficiency of a human-machine Interface by means of comparison of totat complexity and workload of activity for different options of the Interface;
[00135] 18. estimating of software usability by means of comparison of total complexity and workload activity for different options of the software; [00136] 19. displaying such as on a computer monitor, of values of current complexity and workload of activity and regulation of mood states as a feedback signal; and/or [00137] 20. Increasing human controlled system modes when values of the current complexity and workload of activity or current workload of regulation mood states reach, exceed or fall below threshold values
[00138] In summary, a method is provided for estimating human activity and mood states by means of measurement of time intervals between rapid (saccadic) movements of eyes (Inter Saccadic Intervals - ISl) and Reactions of Activation (RA) of a Galvanic Skin Response (GSR), which are used as indicators of different levels of human regulation processes. The basis of the method Is the generality (commonality, universality) of mental regulation processes αf human activity and mood states expressed in uniform time scales of these processes at different levels. Different psychophysiological parameters can be used for an estimation of human activity and mood states accordingly. [00139] Methods are known for mental workload measurement, based on rapid eye movements Including registration of eye movements by means of EOG, ISI estimation and their comparison with ranges (taxons) of ISI which characterize time scales of regulation processes at different levels of the various contents and complexity. These processes can include the organization of sensomotoric and perception activity, comprehension of a situation and decision^maklng before formation of new professional norms and essentially new knowledge. Embodiments of the present invention additionally provide an estimation of change of mood states by means of measurement of duration of GSR Reactions of Activation (fast reductions In skin resistance, which connected with arousal and emotional reactions), and their comparison with the same ISI ranges (taxons). Further comparison of eye IS! movements and Reactions of Activation of GSR can be performed. As a result, separation in time of processes of human activity regulation connected with human tasks decision, and regulation of mood states connected with Influence αf external or extreme føctors (stress, weariness, emotional reactions, physical efforts, etc.) can be achieved. [00140] Human activity can be estimated automatically by means of calculation of quantitative criteria of complexity and workload on data ISI, which taxons do not coincide with taxons of GSR RA. Mood states can be estimated automatically by means of calculation of quantitative criteria of complexity and workload of mood states regulation on data ISI, which taxons coincide with taxons of RA GSR.
[00141J The method can be used for human activity and mood state analysis in different scenarios and for different purposes, such as for an estimation of a human proficiency level on simulators and in real conditions; for an estimation of complexity αf human controlled procedures; for an estimation of Influence of external extreme factors on activity and mood states; for an estimation of usability of Interfaces of control systems and computer programs, etc.
[00142] The above-described embodiments of the present Invention are Intended to be examples only. Alterations, modifications and variations may be effected to the particular embodiments by those of skill In the art without departing from the scope of the Invention, which is defined solely by the claims appended hereto.

Claims

WHAT IS CLAIMED IS:
Claim 1. A method of estimating human activity comprising the steps of: inputting EOG data from a digital output of an EOG amplifier into a memory buffer ordered by time-code; S categorizing an eye sight state as saccade or fixation, eye blink or slow movement (drift) based on the received EOG data; determining the start and end of the saccade thereby computing an interval between the saccade and a previous saccade and storing parameter values In memory; realizing ISI taxonomy by comparing with a borders of ISl ranges; realizing ISI taxonomy by comparing a registered ISI with the borders of at least five taxoπs 0 thereby permitting the estimation of the regulation process level during this time interval in accordance with the taxon's number; computing a diagram after comparison of ISI and RA GSR in real time and displaying it on a computer monitor; Inputting GSR data from a measuring equipment Into a memory buffer ordered by time-code; separating fast Reactions of Activation from slow Reactions of Relaxation in the received GSR data; checking a 5 condition of presence of GSR Reaction of Activation and If absent, input a new portion of GSR data, and when GSR Reactions of Activation are found, determining the moments of beginning and ending points, and saving these values In a computer memory; comparing the tfrne of the beginning and the ending of the determined ISI1 with the presence of the given time Interval of GSR Reaction of Activation; checking a condition of coincidence, or Q correspondence, of ISI and GSR Reactions of Activation and if coincidence is absent, calculating values of the current complexity and workload of activity, and If coincidence exists, comparing taxoπ numbers of ISl and GSR RA and calculating values of the current complexity of both workload and mood states. Claim 2. The method of claim 1 further comprising the step of: calculating current complexity 5 of activity using:
Figure imgf000025_0001
where: Q -the current complexity of activity or mood states for registered ISI, I -registered ISI number, Bj - the maximal border of taxon to which belongs registered JSJ, Bi - the maximal border of the 1-st taxoπ of JSI. Claim 3. The method of claim 2 further comprising the step of: calculating current workload of activity using:
1 B1 where: Wi - the current workload of activity or mood states for registered ISI, 1 -registered ISl number, Bi - the maximal border of taxon to which belongs registered ISI, Bi - the maximal border of the 1-st taxon of ISI, tl - duration of registered ISi.
Claim 4. The method of claim 3 further comprising the steps of: summarizing the received value of the current complexity and workload of activity with respect to stored data. Claim S. The method of claim 4 further comprising the steps of: summarising the current criteria of mood states with respect to stored data.
Claim 6. The method of claim 6 further comprising the steps of: determining whether the values of criteria of the current complexity workload of activity and mood states should be presented to the computer monitor as feedback to the human and if so, presenting these values to the monitor, and If not, determining If controlled system modes should be Increased, and if such an Increase is desired, checking whether the normative (threshold) value Df criteria of the current complexity of activity and mood states has been exceeded. Claim 7. The method of claim 6 further comprising the step of: establishing a command to Increase controlled system modes. Claim 8. The method of claim 6 further comprising the step of: checking if a command relating to, or requesting, the termination of measurements has arrived from the computer keyboard and if not proceeding with inputting further EOG data from the digital output of the EOG amplifier Into the memory buffer ordered by time-code; categorizing an eye sight state as saccade or fixation, eye blink or slow movement (drift) based on the received EOG data; determining the start and end of the saccade thereby computing an interval between the saccade and a previous saccade and storing parameter values in memory.
Claim 9. The method of claim 7 further comprising the step of: calculating total complexity of both workload activity and mood states, wherein total complexity of activity or mood states represent total values of corresponding values of the current complexity of activity and mood states divided by the total number of ISI using:
Figure imgf000027_0001
wlwe: C - total complexity of acrtlvJty or mood states, Q - the current complexity of activity or mood states for J ISI, N - total number of ISI.
Claim 10. The method of claim 9 further comprising the step of: calculating total workload of activity or mood states using:
Figure imgf000027_0002
where: W - total workload activity or mood states, W| - current workload of activity or mood states for i ISI, N -total number of ISI.
Claim 11. ThB method of claim 10 further comprising the step of: comparing total complexity and workload of activity with normative values based on data for experienced operators on simulators.
Claim 12. The method of claim 11 further comprising the step of: checking if an estimation of complexity of human controlled procedures Js desired and if so, estimating this complexity by adding a received value of total complexity σf activity to a file of values of this criterion, wherein, these values relate to the group of experienced operators considered a reference. Claim 13. The method of claim 12 further comprising the step of: checking If an estimation of influence of external extreme factors on activity and mood states Is desired and If so, determining a degree of the specified influence by means of comparison of the received values of total complexity and workload of activity and regulation of mood states with values for corresponding normal conditions.
Claim 14. The method of claim 13 further comprising the step of: checking if an estimation of a degree ergonomte effectiveness of the human-machine interface or software usability Is desired and If such an estimation is desired, determining this parameter by comparing received values of total complexity and workload of activity with the similar data otherwise received.
Claim 15. The method of claim 1 wherein the categorizing of eye sight state uses a velocity method for saccade detection in which the eye movement velocity is computed from the EOG data by first smoothing with a low-pass digital filter to remove noise and then numerically differentiating the filter output
Claim 16. The method of claim 2 wherein the computed velocity Is compared to an empirically determined threshold level separating saccadic movements from fixations and drifts.
Claim 17. A method for estimating human activity and mααd states Implemented by means of measuring Inter Saccadic Intervals (ISI) In eye movements and Reactions of Activation (RA) of GSR comprising the steps of: a) measuring EOG signals from a human face; b) determining saccades In EOG signals and estimating saccade start times; c) estimating Inter Saccadic Interval value (duration) and storing such value for calculation of criteria of activity and moσd states; d) determining a regulation level of activity and mood states by calculation of taxon number for registered ISI and by means of comparison of its duration with borders of (five) ISI ranges (taxons); e) measuring GSR signals from human hand; f) determining a GSR Reaction of Activation; g) determining parameters of GSR Reaction of Activation; h) determining a regulation level of mood states by calculation of taxon number for registered GSR Reaction of Activation. i) determining which ISI coincide and which ISl do not coincide with RA GSR; j) comparing taxon numbers of ISl and RA GSR for ISI which coincide with RA GSR; k) displaying dynamics of regulation processes of activity and mood states as a sequence of ISI. I) calculating a current and total complexity and workload of activity on data of ISI, which taxons do not coincide with taxoπs of RA GSR; m) calculating a current and total complexity of regulation of mood states on data of ISI, which taxons coincide with taxons of RA GSR; π) evaluating a human proficiency level on simulators and in real conditions by means of comparison of total complexity and workload of activity with their normative values; o) estimating a complexity of human controlled procedures by means of calculation of average value of total complexity of activity for experienced operators; p) estimating a degree of Influence of extreme factors on activity and mood states by means of comparison of total complexity and workload of activity and regulation mood states with their values for normal conditions; q) estimating a degree of ergonomic efficiency of a human-machine Interface by means of comparison of total complexity and workload of activity for different options of the interface; r) estimating of software usability by means of comparison of total complexity and workload activity for different options of the software; s) displaying values of current complexity and workload of activity and regulation of mood states as a feedback signal; and t) increasing human controlled system modes when values of the currant complexity and workload of activity or current workload of regulation mood states reach, exceed or fall below threshold values.
PCT/IB2006/000782 2005-04-04 2006-04-04 System and method for estimation of human activity and mood states based on rapid eye movements and galvanic skin response WO2006106408A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US66768005P 2005-04-04 2005-04-04
US60/667,680 2005-04-04

Publications (3)

Publication Number Publication Date
WO2006106408A2 true WO2006106408A2 (en) 2006-10-12
WO2006106408A3 WO2006106408A3 (en) 2006-11-23
WO2006106408B1 WO2006106408B1 (en) 2007-09-07

Family

ID=36942392

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2006/000782 WO2006106408A2 (en) 2005-04-04 2006-04-04 System and method for estimation of human activity and mood states based on rapid eye movements and galvanic skin response

Country Status (1)

Country Link
WO (1) WO2006106408A2 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015149606A1 (en) * 2014-04-01 2015-10-08 姚健欣 Emotion detection method and system based on human skin resistance changes
CN109884588A (en) * 2019-01-16 2019-06-14 北京大学 A kind of distance metric method and system of pulse train
CN113243917A (en) * 2021-05-18 2021-08-13 中国民用航空总局第二研究所 Fatigue detection method and device for civil aviation controller, electronic equipment and medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5295491A (en) * 1991-09-26 1994-03-22 Sam Technology, Inc. Non-invasive human neurocognitive performance capability testing method and system
US6520905B1 (en) * 1998-02-26 2003-02-18 Eastman Kodak Company Management of physiological and psychological state of an individual using images portable biosensor device
WO2006024129A1 (en) * 2004-09-03 2006-03-09 Canadian Space Agency System and method for mental workload measurement based on rapid eye movement

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5295491A (en) * 1991-09-26 1994-03-22 Sam Technology, Inc. Non-invasive human neurocognitive performance capability testing method and system
US6520905B1 (en) * 1998-02-26 2003-02-18 Eastman Kodak Company Management of physiological and psychological state of an individual using images portable biosensor device
WO2006024129A1 (en) * 2004-09-03 2006-03-09 Canadian Space Agency System and method for mental workload measurement based on rapid eye movement

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015149606A1 (en) * 2014-04-01 2015-10-08 姚健欣 Emotion detection method and system based on human skin resistance changes
CN109884588A (en) * 2019-01-16 2019-06-14 北京大学 A kind of distance metric method and system of pulse train
CN113243917A (en) * 2021-05-18 2021-08-13 中国民用航空总局第二研究所 Fatigue detection method and device for civil aviation controller, electronic equipment and medium
CN113243917B (en) * 2021-05-18 2023-05-12 中国民用航空总局第二研究所 Fatigue detection method and device for civil aviation controller, electronic equipment and medium

Also Published As

Publication number Publication date
WO2006106408A3 (en) 2006-11-23
WO2006106408B1 (en) 2007-09-07

Similar Documents

Publication Publication Date Title
US20230056506A1 (en) Systems and methods for assessing and improving sustained attention
EP1799105B1 (en) System and method for mental workload measurement based on rapid eye movement
US8790280B2 (en) Human state estimating device and method
US11083398B2 (en) Methods and systems for determining mental load
KR101854812B1 (en) Psychiatric symptoms rating scale system using multiple contents and bio-signal analysis
Fletcher et al. The effect of motor task precision on pupil diameter
CN109222888A (en) A method of psychological test reliability is judged based on eye movement technique
Bakker et al. Decisions in motion: passive body acceleration modulates hand choice
Rose et al. ERP correlates of associative learning
WO2006106408A2 (en) System and method for estimation of human activity and mood states based on rapid eye movements and galvanic skin response
US10827922B2 (en) Apparatus and method for objective visual acuity measurement using dynamic velocity threshold filter in optokinetic response processing
Somia et al. A computer analysis of reflex eyelid motion in normal subjects and in facial neuropathy
JPH04193255A (en) Method for analysis of eyeball control system
Scocchia et al. Differential effects of visual attention and working memory on binocular rivalry
CN115191018A (en) Evaluation of a person or system by measuring physiological data
CN108451494B (en) Method and system for detecting time-domain cardiac parameters using pupil response
Logan et al. From individual features to full faces: Combining aspects of face information
EP4312713A1 (en) Ocular system for diagnosing and monitoring mental health
Buchner Multimodal Feature Selection to Unobtrusively Model Trust, Workload, and Situation Awareness
JP3724524B2 (en) Eyeball control system information detection apparatus and eyeball control system analysis method
EP4312227A1 (en) Trail making test system
Coelho et al. Ergonomic design and evaluation of surveillance systems
Noah Modeling mental workload with eye tracking using latent profile analysis
Pokojski Transdisciplinary Assessment Matrix to Design Human-Machine Interaction
Bläsing et al. Influence of Increasing Task Complexity and Use of Informational Assistance Systems on Mental Workload. Brain Sci. 2021, 11, 102

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application
NENP Non-entry into the national phase

Ref country code: DE

WWW Wipo information: withdrawn in national office

Country of ref document: DE

NENP Non-entry into the national phase

Ref country code: RU

WWW Wipo information: withdrawn in national office

Country of ref document: RU

122 Ep: pct application non-entry in european phase

Ref document number: 06727422

Country of ref document: EP

Kind code of ref document: A2

WWW Wipo information: withdrawn in national office

Ref document number: 6727422

Country of ref document: EP