WO2004028362A1

WO2004028362A1 - Eeg-based fatigue detection

Info

Publication number: WO2004028362A1
Application number: PCT/AU2003/001248
Authority: WO
Inventors: Saroj Lal; Ashley Craig
Original assignee: University Of Technology, Sydney
Priority date: 2002-09-24
Filing date: 2003-09-24
Publication date: 2004-04-08
Also published as: AU2002951605A0

Abstract

The invention concerns a method and system for computing a state of fatigue whilst a user carries out a task. In a first step of the method, EEG data is sampled from a user when the user is performing a task. Frequency domain analysis is then performed on the sampled data to derive the magnitude of EEG in a plurality of frequency bands. The magnitude is then simultaneously computed in each of the bands prior to comparing the magnitude in each of the bands against pre-determined standards to determine a corresponding state of fatigue.

Description

"EEG-basβd Fatigue Detection"

Technical Field

The invention concerns a method and a system for computing a state of fatigue whilst a user carries out a task.

Background Art

Driver fatigue is a significant cause of traffic accidents, and is a persistent occupational hazard for professional or long-distance drivers who are involved in shift-work, Fatigue related accidents have potentially catastrophic personal consequences and are a substantial financial burden on the community. Cognitive skills are impaired by fatigue. An adverse effect of fatigue is a drivers' limited ability to assess their own level of alertness. This affects a drivers' ability to continue to drive safely. It is therefore desirable to develop countermeasures to driver fatigue.

Various approaches to monitor fatigue have been attempted. In one approach, Artaud¹, et al. described how the analysis of the regularity of a driver's breathing contributed to the prediction of deterioration in alertness. In another approach, Artaud¹, et al. reported on the prospect of videoϊng a driver's face. A further approach included the use of telemetric applications incorporated into a vehicle. These applications are aimed at supporting the driver during route guidance. However this type of system can distract the driver by presenting too much information and such systems are typically unpopular with the driving community. More recently, an individuals' electroencephalogram signal (EEG) has been found to be a predictive and reliable indicator of that individual's level of alertness. The EEG records the electrical activity generated in the brain of the individual and may be used to define which stage of alertness sleep the Individual is experiencing. Ninomija et al². developed a system which detects sleepy states of drivers using grouped EEG alpha waves to warn the driver of such a state. However, the error in their system has a reported order of magnitude of 25- 35%. in addition, the system is cumbersome as extra electrodes are required to monitor separate physiological signals. A further proposal described a system based on detecting grouped alpha waves. The system incorporates a convolution with weighting factors such as moving average methods. Such a system separates grouped alpha waves from various kinds of noise and detects low awakening levels as soon as grouped alpha waves appear.

The inventors have described the importance of using EEG as an indicator of fatigue to reduce fatigue related errors and accidents (Lai & Craig)^3,4'⁵.

Disclosure of invention

In a first aspect, the invention is a method for computing a state of fatigue whilst a user carries out a task, the method comprising the steps of : sampling EEG data from a user when the user is performing a task; performing frequency domain analysis of the sampled data to derive the magnitude of EEG in a plurality of frequency bands; computing the magnitude simultaneously in each of the bands; and comparing the magnitude in each of the bands against pre-determiπed standards to determine a corresponding state of fatigue.

The method may further alert the user as to their determined state of fatigue.

The predetermined standards may be determined according to the steps of: sampling EEG data obtained from a user, wherein the data is representative of when the user is in an alert state; performing frequency domain analysis of the sampled data to compute a mean and a standard deviation of the EEG magnitude simultaneously in each of a plurality of frequency bands; computing at least a first threshold coefficient for each band in terms of the mean and the standard deviation of the respective band; and computing a plurality of standards in terms of the EEG magnitude in each frequency band and the relation of each magnitude to the respective coefficient such that each standard is representative of a state of fatigue.

An advantage of at least one embodiment of the invention is that the determination of fatigue utilises detection of simultaneous changes that occur in a plurality of frequency bands rather than a single frequency band. A further advantage of at least one embodiment is that the determination of fatigue detection in three stages i.e. early, medium and extreme is based on brain activity changes.

The method may use an FFT to perform the frequency domain analysis, Sampling data representative of an alert state may be derived from a single user. Optionally, the sampling data may be derived from a sample of users who may perform similar tasks whereby an average is taken of the sample set. Either way, the sampling data representing an 'alert state' may be acquired 'on-line', whilst carrying out a task. Optionally, the data may be acquired 'off-line', and stored for future use. In obtaining such data, video data, or audio data, may be simultaneously acquired for confirmation that the, or each user is in an alert state.

The sampled data may be classified into four frequency bands comprising frequencies within the range of delta waves, theta waves, alpha waves and beta waves. Delta, theta, alpha and beta waves may be within the ranges of about 0 to 4 Hz, about 4 to 8 Hz, about 8 to 13 Hz, and about 13 to

20 Hz respectively.

A first and a second threshold coefficient may be assigned for each band, the first and second coefficients representing an upper bound and a lower bound respectively. Further coefficients may be defined.

Boolean logic may be applied in order to define the respective standards. Aside from an alert state, the states of fatigue may correspond, in increasing order, to a transition state, a transitional to post-transitional state and a post-transitional state. The EEG data may be obtained using a single or a multi channel physiological monitor. Data may be sampled at 256 Hz. The EEG magnitude may be computed as the sum of the values within each frequency band.

Optionally, the EEG magnitude may be computed as an average of each of the recording channels. Equipment indicators may alert a user as to their current state of fatigue, for example, a green indicator may indicate to a user that they are performing a task in an alert state. Yellow may indicate a transitional state, orange a transitional to post -transitional state, and red a post-transitional state.

In a second aspect, the invention is a system for computing a state of fatigue whilst a user carries out a task, the system comprising: sampling means for sampling EEG data from a user when the user is performing a task; analysing means to perform frequency domain analysis of the sampled data to derive the EEG magnitude in a plurality of frequency bands; computing means for classifying the spectrum and simultaneously computing the magnitude in each of the bands; and memory means to compare the magnitude in each of the bands against a pre-determined standard to determine a corresponding state of fatigue

The pre-determined standard may be determined according to the method described above.

The sampling data representative of an alert state may be derived from a single user. Alternatively, the sampling data representative of an alert state may be derived from a sample of users who perform similar tasks and whereby an average is taken of the sample set. The sampling data representing an 'alert state' may be acquired 'on-line', whilst the, or each user is carrying out a task. Alternatively, the sampling data may be acquired 'off-line' and stored for future use.

Whilst obtaining the sampling data, video data and/or audio data, may be simultaneously acquired for confirmation that the, or each user is in an alert state.

The sampled data may be classified into four frequency bands comprising frequencies within the range of delta waves, theta waves, alpha waves and beta waves. The delta, theta, alpha and beta waves may be within the ranges of 0 to about 4 Hz, about 4 to about 8 Hz, about 8 to about 13 Hz, and about 13 to about 20 Hz respectively.

At least a first and a second threshold coefficient may be assigned for each band, the first and second coefficients representing an upper bound and a lower bound respectively. Further coefficients may be defined.

Boolean logic may be applied in order to define the respective standards. Aside from an alert state, the states of .fatigue may correspond, in increasing order, to a transition state, a transitional to post-transitional state and a post-transitional state.

EEG data may be obtained using a single or a multi channel physiological monitor, The EEG magnitude is the sum of the values within each frequency band. The EEG magnitude may be computed as an average of the separate individual recording channels. Alternatively, the EEG magnitude may be computed as an average of a particular site on the brain, for example, but not limited to, the temporal, parietal, or central site.

The system may further include an alert means to alert the user as to their determined state of fatigue. For instance, auditory indicators may alert a user as to their current state of fatigue, The system may include a first indicator which indicates to a user that they are performing a task in an alert state, a second indicator which indicates a transitional state, a third indicator which indicates a transitional to post -transitional state, and a fourth indicator which indicates a post-transitional state. Each indicator may be identified by a different sound feedback.

Brief Description of Drawings

An example of the invention will now be described with reference to the accompanying drawing, in which:

Figure 1 , which illustrates steps to compute a state of fatigue whilst a user performs a task;

Figure 2 schematically illustrates a software panel for monitoring fatigue; and Figure 3 schematically illustrates data detection during computation of a state of fatigue.

Best Mode for Carrying Out the Invention

Figure 1 illustrates a sequence of steps 10 used to compute a state of fatigue whilst a user performs a task. The partial sequence of steps 12 indicates the steps required in order to determine a plurality of standards, each corresponding to a different state of fatigue. Four different fatigue states were identified, these were alert phase, transitional phase or early fatigue phase, the transitional-post transitional phase referred to as medium levels of fatigue, and the post-transitional phase in other words extreme levels of fatigue. In an alert state, a user is essentially non-fatigued.

Methodology

The user participates in a trial, and data is taken over a period of time that is representative of the user's alert state 14. This data is taken from the beginning of the trial before the user develops signs of fatigue and recorded on a spectrum analyser. Video footage of the user's face is used to confirm that the user shows signs of being in the alert state. This alert state data is referred to as 'baseline data'.

An FFT is performed and the data is categorised into a number of frequency bands, a delta band = 0 to 4 Hz, a theta band = 4 to 8 Hz, an alpha band = 8 to 13 Hz and a beta band - 13 to 20 Hz.

The magnitude for each second of data, in each of the bands, is calculated as the sum of the values in microvolts. From the baseline data, the mean and standard deviation of the magnitudes in each frequency band are calculated 18. The spectrum analyser has multiple recording channels and for each recording channel the following values are computed: D_m, Ds , T_m> T_sci,

A_m, Aβ_d, B_m, B__ where D, T, A and B represent the magnitude in the delta, theta, alpha and beta bands respectively, and subscript _m and subscript _. respectively represent the mean and standard deviation of those magnitudes.

Threshold coefficients are defined in each frequency band in terms of the mean and standard deviation of that band during the baseline period. For example: DT = dιXD_m + axDsd, is such that DT represents a threshold in the delta band and di and d₂ represent coefficients that define that threshold. Two thresholds are defined in this way for each frequency band in each channel, giving a set of thresholds in each channel: DTι, DT₂, TTι, TT₂, AT-,, AT₂, BTι, BT₂, 20.

Figure 2 illustrates a software controlled panel 50 into which the user is able to change the conditional and combinatorial logic. The right hand column specifies the frequency ranges for each of delta 52, theta 54, alpha 56 and beta 58. Boolean logic is used to define standards representing the four states of fatigue in terms of the instantaneous magnitude in each frequency band and the relation of those magnitudes to the thresholds 22. For example: (D & T) & A | B indicates that the state of fatigue is indicated only if the delta and theta and either the alpha or beta magnitude is within the range defined. The left hand column in figure 2 represents specified algorithms to detect specific states. Testing

The methodology was tested on EEG data collected from ten subjects. The subjects who were licensed truck drivers, were randomly recruited for the test. The drivers were between thirty three and fifty five years of age and all gave written consent for the test. To qualify for the test, the drivers had to have no medical contraindications such as severe concomitant disease, alcoholism, drug abuse, psychological or intellectual problems likely, to limit compliance. This was determined during the initial interview on a separate day prior to the test. The test was conducted in a temperature-controlled laboratory and each driver performed a standardised sensory motor driver simulator task. The driving task consisted of ten minutes of active driving to familiarise each driver, followed by a maximum of two continuous hours of driving with a speed less than eighty km/hr, till the respective driver showed physical signs of fatigue. Simultaneous EEG and EOG measures were obtained during the driving task. The EOG or electrooculogram detected the muscle movement of the subject's eyes due to movement of eye muscles, for example, from blink activity.

Nineteen channels of EEG were recorded according to the International 10-20 system which spans the entire brain. A monopolar montage was used, that is, EEG activity was recorded in relation to a linked-ear reference. Left eye EOG measurements were obtained with electrodes positioned above and below the eye with a ground electrode on the masseter. The EOG signal was used to identity blink artefacts in the EEG data as well as changes in blink types such as the small and slow blinks that characterise fatigue. Physical signs of fatigue were identified using a video image of the driver's face, linked in real time with the EEG and EOG measures. Specific facial features that were used to identify fatigue included changes in facial tone, blink rate, eye activity and mannerisms such as nodding and yawning. The video image, and the EOG were used to validate the EEG changes associated with fatigue. The driving task was concluded when the specific facial signs of fatigue appeared. Data Acquistion

The EEG and EOG data were acquired using a multi channel physiological monitor. An individual EEG data point was classified as an epoch; a basic unit for stored EEG data. Data was sampled at 256 Hz, 24. The total sample time was dependent on the subject and lasted until arousal from fatigue by a verbal interaction from a test investigator. An FFT was performed on the EEG data using a spectral analysis package, 26. The EEG was defined in terms of the pre-categorised frequency bands. For each band the average EEG magnitude measured in microvolts was computed as an average of the nineteen channels representative of the entire head of the subject, 28. The EEG of fatigue was classified into the first appearance of transitional phase, between awake and absence of alpha, the transitional-post transitional phase which has characteristics of both, and post transitional phase followed by self-arousals, 30.

For each phase, thirty successive EEG spectra were generated and averaged to form thirty second means to derive the EEG magnitude In the four EEG bands. During fatigue many 'microsleep' cycles were observed spanning transitional through to post-transitional phases followed by self arousal periods. The first complete cycle constituting the four fatigue phases were analysed for each driver. Validation of fatipue states

The four different fatigue phases were classified according to the simultaneous video analysis of facial features in the EOG measurements. Physical signs of fatigue were identified using a video image of each driver's face, linked in real time with the respective physiological measures. The video analysis served as an independent variable for fatigue assessment. The identification of fatigue from the video and EOG had excellent reliability, demonstrated by a high inter- observer and intra-observer agreement, 88% between three trained observers. On appearance of fatigue as classified from the video and EOG measures, thirty epochs that spanned the range of each of the alert and three fatigue phases were recorded to test the ability of the software to allocate each epoch into the correct phase.

As illustrated in Figure 3, a fatigue monitor in an off-line summary, 80, outputs EEG data in the four phases beginning with the alert state were categorised into four channels represented by colour panels, which were green , yellow, orange and red respectively. A colour scale indicated green, 82 as a 'safe' level i.e alert and red 88 as a 'dangerous level of fatigue = post- transitional phase. Yellow, 84 and orange, 86 denoted early = transitional phase and medium transitional-post transitional phase levels of fatigue, respectively. Figure 3 illustrate data collection from a singular channel only, constituting one side of the brain. Statistical analysis

The thirty epochs identified as representing each of the fatigue phases from the video and EOG measures were tested. The testing involved identifying the proportion of epochs that were in each fatigue phase and allocating the data to the respective colour panels. In an off-line analysis mode, the data could also be viewed graphically with a line indicating in which panel i.e. alert or one of the fatigue states, a particular epoch had been allocated. A repeated analysis of variance (ANOVA) was performed to identify if differences existed in the means of the four states detected by the software. A Scheffe test then identified where the differences existed in the comparison of the means. The significance level was set at p<0.05 for all analyses performed.

Results

The data was categorised into each state of fatigue. Twenty five percent of the total epochs were allocated in each of the four states according to the video and EOG analysis which acted as the control against which the allocation of the epochs were compared. The ability of the software to detect fatigue, validated by the video analysis of fatigue, was demonstrated by the fact that the software detected no false positives. A false positive was defined as detecting fatigue in the absence of facial/EEG signs of fatigue. Table 1 demonstrates the allocation by the software of the total number of epochs to each fatigue phase for each subject,

Table. 1 Detection of epochs in an alert or fatigue state (%)

The ANOVA showed that there was an overall difference In the comparison of the means of the four states, F-9.15, df=3, 27, p=0,0002. The post-hoc analysis found that the percentage of time the subjects were in the transitional-post transitional and post-transitional fatigue phases was significantly different to the alert phase, p=0.003 and p=0.0009, respectively, A larger proportion of epochs were detected in the first fatigue state, that is, the transitional phase to fatigue, compared to the other two fatigue phases p<0.01. The number of epochs detected in the transitional to post-transitional and post- transitional phases was not significantly different. The video and EOG analysis had identified subjects as being in the alert phase for an average of 40% of the time, in the transitional phase for 25% of the time, in the transitional to post- transitional for 20% and in the post transitional state for 15% of the total study time. The percentage error of the algorithm detecting fatigue compared with video/EOG allocation of fatigue in the whole data set was as follows: alert=1%, transitiona 9.2%, transitional to post-transitional-1 .5% and post-transitional: 2.7%. The largest difference in the two methods of detection was observed for the transitional and transitional to post-transitional phases with error rates in the order of ten. Data channels were output to the user to indicate their status of fatigue.

The results of testing the software found that the ten drivers were in a fatigue state for at least sixty percent of the total time they spent driving in the simulator. The software was shown to be capable of detecting the three stages of fatigue reliably, and these were validated by video and EOG monitoring. Although one embodiment of the invention has been discussed, it should be appreciated that such an embodiment is only one of the many utilising the principles of the invention. For instance whilst the embodiment - has been described in relation to truck driver fatigue, the invention equally applies to industries whereby a user performs a prolonged or repetitive task such as the aerospace industry, military, railway industry, mining, medical profession or where the user is required to undergo prolonged periods of concentration such as is required by air traffic controllers.

It will be appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the Invention as shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. The present embodiments are, therefore, to be considered In all respects as illustrative and not restrictive.

1. Artaud, Planque, Lavergne, Cara, de Lepine, Tarriere, & Gueguen 1994. 2. Ninomija, Funada, Yazu, Ide, & Daimon 1993. 3. Lal SKL and Craig A. A critical review of the psychophysiology of driver fatigue. Biological Psychology. 2001; 55: 73-194.

4. Lal SKL and Craig A. Electroencephalography activity associated with driver fatigue: implications for a fatigue countermeasure device. 2001 Journal of Psychophysiology; 15: 83-189.

5. Lal S.K.L. and Craig A. (2002) Driver Fatigue; electroencephalography and psychological assessment. Psychophysiology, 39, 1-9.

Claims

CLAIMS:

1. A method for computing a state of fatigue whilst a user carries out a task, the method comprising the steps of : sampling EEG data from a user when the user is performing a task; performing frequency domain analysis of the sampled data to derive the magnitude of EEG in a plurality of frequency bands; computing the magnitude simultaneously in each of the bands; and comparing the magnitude in each of the bands against pre-determined standards to determine a corresponding state of fatigue.

2. The method according to claim 1 , further comprising the step of alerting the user as to their determined state of fatigue.

3. The method according to claim 1 or claim 2, wherein the predetermined standards are determined according to the steps of : sampling EEG data obtained from a user, wherein the data is representative of when the user is in an alert state; performing frequency domain analysis of the sampled data to compute a mean and a standard deviation of the EEG magnitude simultaneously in each of a plurality of frequency bands; computing at least a first threshold coefficient for each band in terms of the mean and the standard deviation of the respective band; and computing a plurality of standards in terms of the EEG magnitude in each frequency band and the relation of each magnitude to the respective coefficient such that each standard is representative of a state of fatigue.

4. The method according to claim 3, wherein the sampling data representative of an alert state is derived from a single user.

5. The method according to claim 3, wherein the sampling data representative of an alert state is derived from a sample set of users who perform similar tasks and whereby an average is taken of the sample set.

6. The method according to any one of the preceding claims 3 to 5, wherein the sampling data representing an 'alert state' is acquired 'on-line', whilst the, or each user is carrying out a task.

7. The method according to any one of the preceding claims 3 to 5, wherein the sampling data is acquired 'off-line' and stored for future use.

8. The method according to any one of the preceding claims 3 to 7, wherein 5 whilst obtaining the sampled data, video data and/or audio data, are simultaneously acquired for confirmation that the, or each user is in an alert state.

9. The method according to any one of the preceding claims 3 to 8, wherein 10 the sampled data is classified Into four frequency bands comprising frequencies within the range of delta waves, theta waves, alpha waves and beta waves.

10. The method according to claim 9, wherein the delta, theta, alpha and beta waves are within the ranges of 0 to 4 Hz, 4 to 8 Hz, 8 to 13 Hz, and 13 to is 20 Hz respectively.

11. The method according to any one of the preceding claims 3 to 10, wherein at least a first and a second threshold coefficient are assigned for each band, the first and second coefficients representing an upper bound and a

20 lower bound respectively.

12. The method according to claim 11, wherein Boolean logic is applied in order to define the respective standards.

25 13. The method according to any one of the preceding claims 3 to 12, wherein aside from an alert state, the states of fatigue corresponds, in increasing order, to a transition state, a transitional to post-transitional state and a post-transitional state.

30 14. The method according to any one of the preceding claims, wherein EEG data is obtained using a single or a multi channel physiological monitor.

5. The method according to claim 14, wherein data is sampled at 256 Hz.

16. The method according to any one of the preceding claims, wherein the EEG magnitude is computed as the sum of the values within each frequency band.

17. The method according to any one of the preceding claims 1 to 15, wherein the EEG magnitude is computed as an average of each of the separate individual recording channels.

18. The method according to any one of the preceding claims, wherein an FFT performs the frequency domain analysis.

19. A system for computing a state of fatigue whilst a user carries out a task, the system comprising: sampling means for sampling EEG data from a user when the user is performing a task; analysing means to perform frequency domain analysis of the sampled data to derive the EEG magnitude in a plurality of frequency bands; computing means for classifying the spectrum and simultaneously computing the magnitude in each of the bands; and memory means to compare the magnitude in each of the bands against a pre-determined standard to determine a corresponding state of fatigue.

20. The system according to claim 19, further operable to determine the predetermined standards, wherein the sampling means is operable to sample EEG data obtained from a user, wherein the data is representative of when the user is in an alert state; the analysing means is operable to perform frequency domain analysis of the sampled data to compute a mean and a standard deviation of the EEG magnitude simultaneously in each of a plurality of frequency bands; and the computing means is operable to compute at least a first threshold coefficient for each band in terms of the mean and the standard deviation of the respective band and operable to compute a plurality of standards in terms of the EEG magnitude in each frequency band and the relation of each magnitude to the respective coefficient such that each standard is representative of a state of fatigue.

21. The system according to claim 20, wherein the sampling data representative of an alert state is derived from a single user.

22. The system according to claim 20, wherein the sampling data 5 representative of an alert state is derived from a sample of users who perform similar tasks and whereby art average is taken of the sample set

23. The system according to any one of the preceding claims 20 to 22, wherein the sampling data representing an 'alert state' is acquired 'on-line', 0 whilst the, or each user is carrying out a task.

24. The system according to any one of the preceding claims 20 to 22, wherein the sampling data is acquired 'off-line' and stored for future use.

is 25. The system according to any one of the preceding claims 20 to 24, wherein whilst obtaining the sampling data, video data and/or audio data, are simultaneously acquired for confirmation that the, or each user is in an alert state.

20 26. The system according to any one of the preceding claims 20 to 25, herein the sampled data is classified into four frequency bands comprising frequencies within the range of delta waves, theta waves, alpha waves and beta waves.

25 27. The system according to claim 26, wherein the delta, theta, alpha and beta waves are within the ranges of 0 to 4 Hz, 4 to 8 Hz, 8 to 13 Hz, and 13 to 20 Hz respectively,

28. The system according to any one of the preceding claims 20 to 27, 30 wherein at least a first and a second threshold coefficient are assigned for each band, the first and second coefficients representing an upper bound and a lower bound respectively.

29. The system according to claim 28, wherein Boolean logic is applied in 35 order to define the respective standards. θ

30. The system according to any one of the preceding claims 20 to 29, wherein aside from an alert state, the states of fatigue corresponds, in increasing order, to a transition state, a transitional to post-transftional state and a post-transitional state.

5

31. The system according to any one of the preceding claims 19 to 30, wherein EEG data is obtained using a single or a multi channel physiological monitor.

10 32, The system according to any one of the preceding claims 19 to 30, wherein the EEG magnitude is computed as the sum of the values within each frequency band.

33. The system according to any one of the preceding claims 19 to 32, is wherein the EEG magnitude is computed as an average of each of the separate individual recording channels.

34. The system according to any one of the preceding claims 19 to 32, wherein the EEG magnitude is computed as an average of a particular site on

20 the brain.

35. The system according to any one of the preceding claims 19 to 34, wherein an FFT performs the frequency domain analysis.

25 36. The system according to any one of the preceding claims 19 to 35, further comprising an alert means to alert the user as to their determined state of fatigue,

37. The system according to claim 36, wherein auditory indicators alert a 30 user as to their current state of fatigue.

38. The system according to claim 37, wherein a first indicator indicates to a user that they are performing a task in an alert state, a second indicator indicates a transitional state, a third indicator indicates a transitional to post -

35 transitional state, and a fourth indicator indicates a post-transitional state.