DE102014206626A1 - Fatigue detection using data glasses (HMD) - Google Patents

Abstract

Disclosed is a method for detecting the tiredness and / or falling asleep of a wearer of data glasses, in particular the driver of a vehicle; wherein the data glasses comprise motion sensors; the method comprising: providing measurements of the motion sensors of the data glasses; Detecting a movement of the data glasses, which indicates the tiredness or the falling asleep of the wearer of the data glasses, based on the measurements; In response to recognition: Issuing an indication that the wearer's fatigue or falling asleep was detected.

Description

The invention relates to a method for detecting tiredness and / or sleeping, in particular falling asleep, a wearer of data glasses.

Nowadays, data glasses, also known as head-mounted displays (HMD) are known, with the aid of which the wearer of the data glasses information can be displayed. The data glasses are worn like ordinary glasses, which is used as a visual aid on the head. Compared to ordinary eyeglasses, however, the data glasses include electronic storage and processing units and a display that is located near the user's eyes when wearing the data glasses. The display can include two partial displays, one for each eye. On the display, information may be displayed to the user in the form of text, graphics or mixtures thereof. In particular, the display may be semitransparent, that is configured in such a way that the wearer can also recognize the surroundings behind the display. Particularly preferably, the information is displayed to the wearer contact-analog, which is sometimes referred to as augmented reality. In this case, the carrier of the data glasses, the information is displayed at a location that is oriented to the location of an object in the area, so for example, adjacent to the object or this superimposed. Data glasses can also be used in vehicles, in particular to display contact-analog information.

There are data goggles have become known, which also each include a camera that makes shots in the direction of the wearer of the data glasses, see for example WO 2013/012914 , Data glasses also often include motion sensors integrated with the frame of the data glasses. These are typically acceleration and / or rotation rate sensors.

Known are driver assistance systems that can detect the tiredness or microsleep of the driver. For this purpose, for example, carried out on analyzes of the applied torque from the driver on the steering wheel, as in EP 2117870 B1 described, steering rest phases observed, as in WO 2006131254 A1 and US 2009021356 A described or using camera-based methods as in WO 2006108372 A1 described, closed eyelids detected and the driver warned to prevent accidents. Have become known in the document EP 2237237 B1 Furthermore, systems that are worn on the head in eyeglass-like form and observe the eyes of a user via cameras, for the purpose of detecting tiredness.

The methods of the prior art for detecting the fatigue or microsleep of a vehicle driver are often prone to error in practice. Often, the signals given by the driver can not be detected accurately enough, e.g. due to the low resolution of a camera and the distance of the camera to the driver. Furthermore, conclusions from the steering wheel movements require knowledge of the road and the individual steering behavior. Reports from users of such prior art systems often describe unnecessary warnings or missed alerts despite a dangerous situation. In addition, such systems often require additional hardware, such as cameras and controllers in the vehicle, which adds to energy consumption and vehicle weight, as well as incurs additional costs.

Against this background, the object of the invention is to provide a method which enables a robust and alternative possibility of fatigue and sleep detection, in particular sleep detection, with the aid of data glasses.

The object is solved by the subject matters of the independent claims. Advantageous developments are defined in the dependent claims.

One aspect relates to a method for detecting the tiredness and / or sleeping of a wearer of data glasses, in particular the driver of a vehicle; wherein the data glasses comprise motion sensors. The method comprises: providing measurements of the motion sensors of the data glasses; Detecting a movement of the data glasses indicative of fatigue or sleep, in particular falling asleep, of the wearer of the data glasses (also referred to herein as falling asleep) based on the measurements; In response to the detection: Issuing an indication that the tiredness or sleeping of the wearer of the data glasses has been detected. The measurements are typically in the form of measurement data. Unless the detection of the sleep-eye movement of the data glasses is performed by the data glasses, the measurement data for providing is previously transmitted from the data glasses to a remote device (for example a computing device of the vehicle). The method can be carried out both completely by a data goggles, as well as in combination with electronic processing units and / or output units of a vehicle.

In this way, the movement of the data glasses that corresponds to the movement of the head of the wearer of the data glasses, the fatigue or falling asleep (especially a microsleep) of the wearer is closed. The Evaluation of the sensor data can be carried out continuously by a computer program that is executed by the data glasses or a computing unit of the vehicle. If the evaluation is to be performed by a computing unit of the vehicle, this is in electronic and preferably wireless communication with the data glasses. For this purpose, a wireless transmission according to the WiFi- or Bluetooth standard can be used.

This application is especially useful for the use of data glasses in a vehicle. Outside of a vehicle, the system is also available, but with much less specific value, since in general no immediate danger situation results from falling asleep.

The detection of a movement may include: detecting a predetermined movement pattern, the movement pattern representing in particular typical head movements during tiredness and when sleeping. The movement pattern may correspond, for example, to a forward nod or a forward slump of the user's head. Likewise, prolonged immobility can correspond to the movement pattern. The prolonged immobility can be caused by the fact that the user has his head ajar against a headrest. It can be provided that the measurement of the movement is examined for agreement with one of a plurality of predetermined movement patterns. For example, the measured motion may be compared with motion patterns for forward nod and immobility.

The predetermined movement patterns and the associated sensor measurements can be determined using machine learning algorithms, in particular neural or Bayesian networks. These methods are known in the art.

In a preferred implementation, the method further comprises: providing measurements of the movement of the vehicle in which the data glasses are located for the same period of time for which measurements of the motion sensors of the data glasses are provided; Wherein the detection of a movement of the data glasses, the measurements of the movement of the vehicle are taken into account.

It is therefore proposed to consider the influence of the movement of the vehicle when detecting the movement of the data glasses. In fact, it is the case that a driver of a vehicle is also exposed to the movements of the vehicle, and these are also measured by the sensors of the data glasses, so there is a superposition of the signals. When detecting the movement of fatigue or sleeping, these overlapping movements must be taken into account and, in principle, subtracted from the measured movements of the data glasses.

As an alternative to subtracting the movement of the vehicle from the measured movement of the data glasses, the measured movement of the data glasses can also be represented by superposition of (weighted) basic movements. As basic movements, one or two sets of basic movements may be used, the basic movements of the first set being respectively orthogonal to the basic movements of the second set. Orthogonal herein means that the multiplication and summation of two different fundamental motions results in substantially zero. This technique is used in communications engineering in code division multiplexing, where, for example, orthogonal cosine and sine functions are used.

A set of ground moves can include one, two or more ground moves. The first set of fundamental movements (sometimes called signature) can be determined in such a way that the movements of the vehicle can be described in an advantageous manner (that is, as accurately as possible) by means of the superimposition (and weighting) of the basic movements contained. A basic movement of the first set may be, for example, a sine function. Typically, a basic movement of the first set is derived from the driving behavior of the vehicle while driving. The second set of basic movements can be determined in such a way that the movement of the head or of the data glasses relative to the vehicle can be described in an advantageous manner (ie as accurately as possible) by means of the superimposition (and weighting) of the basic movements contained. A basic movement of the second set may, for example, be a cosine function. Typically, a fundamental movement of the second set is derived from typical head movements.

The movement of the data glasses relative to the vehicle is then determined in operation by mapping the measured movement of the data glasses to the second set of base movements (herein "second set" does not mean that there must be a "first set", the second set also be the only set of basic movements). The mapping can be achieved by multiplying and summing the measured motion of the data glasses with the basic movements of the second set, respectively. Alternatively, for each fundamental movement of the first set, its inverse (eg -1 · sinusoidal function) can be determined, and the measured movement of the data goggles to the inverse of the Basic movements are mapped. Again alternatively, the measured motion may be mapped onto the first set of ground motions and this mapping then subtracted from the measured motion to conclude the head movement relative to the vehicle.

The basic movements of the first sentence can be adapted to different driving situations. The driving situation can be determined on the basis of the traffic, the route, etc., or can result from the observation of the vehicle movement (by vehicle sensors) over sliding time windows. The respectively valid first set of basic movements can be determined by the vehicle and possibly sent to the data glasses.

The determination of the relative movement of the data glasses to the vehicle by means of basic movements offers the advantage over the subtraction of the measured vehicle movement from the measured data spectacle movement that it is less dependent on randomness of the causality between head movement and vehicle movement. For example, it may happen that the head movement of the component induced by the vehicle movement leads or lags the head movement.

The measurements of the movement of the vehicle are typically provided by sensors of the vehicle, in particular wheel and steering wheel sensors, and by a satellite navigation unit (based on GPS, GLONASS and / or Galileo) of the vehicle. The measurements of the movement of the vehicle are transmitted either to the data glasses; or the movement data of the data glasses are transmitted to the vehicle. The aim is to combine both the measurements of the data glasses and the measurements of the sensors of the vehicle on a computing unit, so that the actual movement of the data glasses in relation to the vehicle can be calculated and fatigue or sleep patterns can be detected. The data transmission between the vehicle and the data glasses can take place, for example, wirelessly according to the WiFi standard or according to the Bluetooth standard.

When detecting a falling asleep movement of the data glasses can also be considered whether a touch of the steering wheel of the vehicle, in which the data glasses is located, could no longer be detected for a predetermined period of time. Here, therefore, an additional criterion is used to detect the falling asleep or fatigue of the driver. Only if both the movement of the data glasses, as well as the non-contact of the steering wheel on the tiredness or falling asleep the driver, a hint is issued.

In a preferred implementation, the data glasses further comprise a camera adapted to take pictures of the surroundings of the data glasses; The method comprises: detecting the interior of a vehicle in which the data glasses are located in the recordings of the camera; Determining the movement of the data glasses relative to the vehicle based on the images of the interior in the images of the camera; Wherein the detection of a movement also takes into account the particular movement of the data glasses on the basis of the images of the camera. Herein, it is proposed to determine the movement of the data glasses not only on the basis of the motion sensors of the data glasses, but also on the basis of a motion detection using camera shots. The camera is immovably connected to the data glasses and thus carries out the same movements as the data glasses. The images of an object will be shifted according to the movement of the data glasses. From the change in the image of the objects in the images of the camera can be inferred the movement of the data glasses. This method is particularly suitable for use in the interior of a vehicle, since its appearance is already known and can be pre-stored. For this purpose, the areas that represent the interior are first recognized in the images of the camera. By comparing the position of features of the interior, in particular the dashboard, in successive recordings or with pre-stored information (in particular features) to the interior, the movement of the data glasses can be determined. An advantage in the motion determination of the data glasses with the aid of camera shots is that the movements of the vehicle have only reduced or no influence on the motion determination, since the reference for the motion determination is the interior of the vehicle. Methods for determining the pose are in the applications filed by the same Applicant on the same day, with Applicants' file numbers EM 26576 and EM 28283 described.

The issued indication may be an optical, acoustic or haptic indication issued by the data glasses and / or the vehicle. For example, the data glasses may display a representation on the display or the vehicle may display a representation on a display such as a head-up display. However, it seems more appropriate to use an audible indication that is emitted by earphones of the data glasses or via the loudspeakers of the vehicle. Furthermore, the data glasses or the steering wheel of the vehicle may vibrate.

In another aspect, a vehicle includes receiving means for electronic Communication and electronic computing means and output means; wherein the vehicle is adapted to carry out a method according to any one of the preceding claims. In this aspect, the vehicle thus recognizes the movement that indicates tiredness or falling asleep. The necessary sensor data of the data glasses are received via a preferably wireless communication.

Another aspect relates to data goggles with electronic computing means and motion sensors and output means, such as a display; wherein the data glasses are adapted to carry out a method according to any one of the preceding claims. In this aspect, the detection of the sleep motion is thus performed by the data glasses. This also gives the hint. The data glasses preferably also include means for haptic or acoustic output.

BRIEF DESCRIPTION OF THE DRAWINGS

1a and 1b explain schematically the operation of a system according to an embodiment.

2 schematically illustrates a flowchart of a method according to an embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

1a and 1b explain schematically the operation of a system according to an embodiment. The system includes data glasses 3 by a user 1 is worn (in 1 the user's head is shown in a side view). The data glasses 3 lies on the ear 2 of the user. The data glasses 3 includes an electronic sensor and computing unit 4 , This unit 4 houses rate of rotation and acceleration sensors as well as a module for wireless communication according to the Bluetooth standard. During operation, the yaw rate and accelerometers continuously measure (for example at regular intervals of 0.2 s, 0.1 s or 50 ms) the movement of the data goggles 3 , These measurements (or compressed versions of them) will send the unit 4 the data glasses wirelessly to a control unit 5 of the vehicle in which the data glasses are located.

A control unit 5 receives in operation the data glasses 3 transmitted measurement data on movement of the data glasses (step S1 in FIG 2 ). Also receives the control unit 5 continuously measuring data for steering angle impact as well as acceleration of the vehicle (step S2 in 2 ). The control unit calculates from these data 5 the acceleration and yaw rate of the vehicle. It can be provided that the control unit 5 Data gaze and / or vehicle measurements are buffered to provide data for the same measurement times in the following analysis.

In a pre-processing the control unit calculates 5 based on both received data streams, the relative movement of the data glasses 3 to the vehicle (step S3 in FIG 2 ). In this case, the accelerations and yaw rates of the vehicle calculated at the same time are subtracted from the measured accelerations and yaw rates of the data glasses. Alternatively or additionally calculating the relative movement, the movement data of the vehicle can also be used to determine time ranges in which no analysis of the movement data of the data glasses with respect to a falling asleep movement is undertaken. These time ranges may be, for example, times when strong accelerations of the vehicle have occurred.

After preprocessing, the control unit performs 5 an investigation program that examines the incoming measurements to see if they correspond to one of several motion patterns that represents a sleep-in motion. These movement patterns are predefined and in the control unit 5 saved. In 1b is (opposite the position in 1a ) schematically illustrates a pitching of the user's head, as typically occurs in instantaneous sleep. Once the control unit 5 detects a falling asleep (step S5 in FIG 2 ) gives the control unit 5 via a connected speaker 6 , which may also be part of an audio system of the vehicle, emits an audible warning signal, as in 1b schematically represented by sound waves (step S6 in 2 ).

The use of data goggles in the vehicle makes it more reliable to detect the driver's tiredness or falling asleep. It can also save weight, energy and costs associated with fixed systems such as ECUs, sensors and cameras.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• WO 2013/012914 [0003]
• EP 2117870 B1 [0004]
• WO 2006131254 A1 [0004]
• US 2009021356 A [0004]
• WO 2006108372 A1 [0004]
• EP 2237237 B1 [0004]
• EM 26576 [0022]
• EM 28283 [0022]

Claims (16)

 A method for detecting fatigue and / or sleep, in particular falling asleep, a wearer of data glasses, in particular the driver of a vehicle; wherein the data glasses comprise motion sensors; the method comprising: Providing measurements of the motion sensors of the data glasses; Detecting a movement of the data glasses that indicates the weariness or sleep of the wearer of the data glasses based on the measurements; In response to the detection: Issuing an indication that the tiredness or sleeping of the wearer of the data glasses has been detected.  The method of claim 1, wherein detecting movement comprises: Recognizing a predetermined movement pattern, wherein the movement pattern in particular represents typical head movements during tiredness and when sleeping.  The method of claim 1, wherein detecting movement comprises: Recognizing one of a plurality of predetermined movement patterns, wherein the movement patterns in each case represent in particular typical head movements during tiredness and during sleep.  The method of claim 2 or 3, wherein the or the predetermined movement pattern using machine learning algorithms, in particular neural or Bayesian networks, are determined.  Method according to one of claims 2 to 4, wherein the or a movement pattern of the one or more movement pattern representing a nodding of the head and / or the or a movement pattern of the one or more movement pattern represents an inactivity of the head over a predetermined period.  The method of any one of the preceding claims, wherein the method further comprises: Providing measurements of the movement of the vehicle in which the data glasses are located for the same period of time for which measurements of the motion sensors of the data glasses are provided; Wherein the detection of a movement of the data glasses, the measurements of the movement of the vehicle are taken into account.  Method according to claim 6, wherein the measurements of the movement of the vehicle are provided by sensors of the vehicle, in particular of wheel and steering wheel sensors as well as of a satellite navigation unit of the vehicle.  Method according to one of claims 6 or 7, wherein the measurements of the movement of the vehicle are transmitted to the data glasses; or wherein the measurements of the motion sensors of the data glasses are transmitted to the vehicle.  Method according to one of the preceding claims, wherein when detecting a movement of the data glasses is also considered whether a touch of the steering wheel of the vehicle in which the data glasses is located, could no longer be detected for a predetermined period of time.  The method of any one of the preceding claims, wherein the data glasses further comprise a camera; The method comprises: Detecting the interior of a vehicle in which the data glasses are located in the images of the camera; Determining the movement of the data glasses relative to the vehicle based on the images of the interior in the images of the camera; Wherein the detection of a movement also takes into account the particular movement of the data glasses on the basis of the images of the camera.  Method according to one of the preceding claims, wherein the reference is an optical, acoustic or haptic hint, which is output from the data glasses and / or the vehicle.  The method of any one of the preceding claims, wherein detecting the movement comprises: Determining the movement of the data groove relative to the vehicle using a map of the measured motion of the data glasses, respectively, to one or more fundamental movements of a set of ground movements.  The method of claim 12, wherein the set of ground moves comprises ground moves determined based on typical vehicle motions or head movements.  Method according to one of claims 12 or 13, wherein the set of basic movements is temporally variable and is determined in particular on the basis of the driving situation.  A vehicle comprising receiving means for electronic communication and electronic calculating means and output means; wherein the vehicle is adapted to carry out a method according to any one of the preceding claims. Data glasses comprising receiving means for electronic communication, electronic calculating means and motion sensors and output means, in particular a display; wherein the data glasses are adapted to carry out a method according to any one of the preceding claims.

Images