WO2001063576A2

WO2001063576A2 - Monitoring system

Info

Publication number: WO2001063576A2
Application number: PCT/GB2001/000776
Authority: WO
Inventors: Neil Thacker; Patrick Courtney
Original assignee: The Victoria University Of Manchester
Priority date: 2000-02-23
Filing date: 2001-02-23
Publication date: 2001-08-30
Also published as: WO2001063576A3; GB0004142D0; AU3754101A

Abstract

A monitoring system for monitoring the behaviour of an object, the monitoring system having at least one image sensor and being operative to extract variables from image sequences input from the one or more image sensor, wherein the variables are interrelated such that certain combinations of values of variables occur frequently and other combinations of values of variables do not occur during normal behaviour of the object, the values of variables and relationships between the variables are recorded by the monitoring system as a reference set of data, and after the values of the variables and the relationships between the variables have been recorded for a predetermined training time, an event is triggered by the system when an unusual combination of values of variables occurs.

Description

MONITORING SYSTEM

The present invention relates to a monitoring system.

Systems which monitor a space to determine the presence or absence of a person or object are well known. For example, an automatic door is provided with an infra-red sensor (or other suitable sensor) with a limited field of view, which causes the door to open when a person is detected within the field of view of the sensor.

It is known to use remote systems to monitor a person, for an elderly person living at home, in order to ensure that that person has not suffered an injury. In one known system, described in US Patent No. 5441047, cameras are provided in a patient's home to allow a healthcare worker to monitor the behaviour of the patient at regular intervals without visiting the patient's home.

US Patent No. 4524243 describes a personal alarm system comprising a clock having a reset switch which must be depressed at predetermined intervals in order to prevent an alarm being triggered. The clock is located in a patient's home, and the alarm is connected to a central monitoring station. In the event that the predetermined interval elapses without the reset switch having been depressed, the alarm is triggered at the central monitoring station thereby alerting staff at the central monitoring station to a potential problem.

It is known to provide a self-activated alarm which may be worn by an elderly person, for example around the neck, the alarm being remotely connected to a telephone and via the telephone to a central monitoring station. The person activates the alarm in the event that assistance is required, if for example they have fallen and are unable to reach a telephone.

All of the above described systems require some form of intervention either to trigger an alarm or to prevent an alarm from being triggered. The system described in US Patent No. 5441047 requires the regular attention of a healthcare worker in order to monitor a subject, and is therefore expensive to implement. The system described in US Patent No. 4524243 requires a subject to reset a clock, which may unduly restrict a subject's behaviour, and may give rise to false alarms when the subject inadvertently omits to reset the clock. The self-activated alarm suffers from the disadvantage that it must be operated by the subject, which may not be possible in some circumstances, for example if the subject has been knocked unconscious during a fall.

A home health monitoring system is described in Medical Engineering & Physics 20 (1998), pages 573-579. This system includes a series of sensors, for example temperature sensors located on a bed, electrocardiogram (ECG) sensors located in a bath, and a body weight sensor located at a toilet. The sensors are used to detect information relating to a subject's temperature, heartbeat, and weight, without the need for visits from a healthcare worker. The system is intended to gather data over a period of time, to allow the early diagnosis of medical problems. The system collects data in a passive manner, and is not configured to raise an alarm in the event that the patient is in difficulty, for example if the subject has fallen and is unable to reach a telephone.

JP 10-257204 describes a monitoring system in which the number of transitions per day between rooms of a person within a home is measured. An alarm is raised when the number of transitions falls below a threshold. The threshold may be determined on the basis of the number of transitions recorded over several preceding of days. A disadvantage of JP 10-257204 is that it monitors only one variable. This single variable is unlikely to be sufficient to provide adequate monitoring of the person.

It is an object of the present invention to provide a monitoring system which overcomes or mitigates at least some of the above disadvantages.

According to a first aspect of the invention there is provided a monitoring system for monitoring the behaviour of an object, the monitoring system having at least one image sensor and being operative to extract variables from image sequences input from the one or more image sensor, wherein the variables are interrelated such that certain combinations of values of variables occur frequently and other combinations of values of variables do not occur during normal behaviour of the object, the values of variables and relationships between the variables are recorded by the monitoring system as a reference set of data, and after the values of the variables and the relationships between the variables have been recorded for a predetermined training time, an event is triggered by the system when an unusual combination of values of variables occurs.

The object is assumed to behave normally during the training time. Where the object is known to have behaved abnormally values of variables recorded as a result of the abnormal behaviour should be deleted from the reference set of data.

The monitoring system is advantageous because it is capable of automatically detecting abnormal behaviour of an object. The combinations of values of variables which describe normal behaviour of the object are accumulated over time to provide a reference set. Any subsequent combination of values of variables which falls outside of that reference set is indicative of abnormal behaviour.

Suitably, the unusual combination of values of variables comprises a combination that does not fall within the reference set of data.

Suitably, the values of variables and relationships between the variables are not added to the reference set of data after the predetermined training time has ended.

Suitably, the values of variables and relationships between the variables continue to be added to the reference set of data after the predetermined training time has ended, thereby continually updating the reference set of data. Suitably, the reference set of data may be modified directly by excluding a specific range of values of a variable from the reference set of data, or including a specific range of values of a variable in the reference set of data.

Suitably, the reference set of data may be modified directly by excluding a predetermined combination of values of variables from the reference set of data, or by including a predetermined combination of values of variables in the reference set of data.

Suitably, the system triggers an alarm when the system triggers an alarm when the unusual combination of values of variables occurs.

Suitably, subsequent to the alarm being triggered it is determined whether the alarm was a false alarm, and if the alarm was a false alarm the combination of values of variables that triggered the alarm is added to a memory.

Suitably, a subsequent combination of values of variables that will trigger an alarm is compared with combinations of values of variables held in the memory in order to determine whether the combination corresponds to a false alarm.

Suitably, if the combination of values of variables lies within a predetermined range, the system reviews preceding values of at least one variable to determine whether that variable has exceeded a threshold value.

Suitably, the system triggers an alarm when the combination of values of variables lies outside the reference set and the at least one variable has exceeded the threshold value.

Suitably, the at least one variable is acceleration of the object.

Suitably, time is one of the properties represented by a variable. Suitably, the position of the object is one of the properties represented by one or more variables.

Suitably, the pose of the object is one of the properties represented by one or more variables.

Suitably, periodicity of the behaviour of the object is one of the properties represented by one or more variables.

Suitably, the system includes means for determining the number of objects within the field of view of a sensor.

Suitably, the number of objects is one of the properties represented by one or more variables.

According to a second aspect of the invention there is provided a method of monitoring the behaviour of an object, the method comprising inputting a sequence of images, extracting variables from the sequence of images, wherein the variables are interrelated such that certain combinations of values of variables occur frequently and other combinations of values of variables do not occur during normal behaviour of the object, recording the values of variables and relationships between the variables as a reference set of data, and after the values of the variables and the relationships between the variables have been recorded for a predetermined training time, triggering an event when an unusual combination of values of variables occurs.

A specific embodiment of the invention will now be described with reference to the accompanying figure, which shows schematically a monitoring system according to the invention.

The embodiment of the invention described below relates to the monitoring of an elderly person (the subject) in his or her home. Referring to Figure 1, the system includes a sensor 1, and several monitoring devices 2-5 which accumulate values describing independent variables, for example position, movement, pose, etc of the subject on the basis of information received from the sensor 1. Values of variables are gathered from the monitoring devices 2-5 over a period of time, and are correlated to provide a reference set of data. Following construction of the reference set of data, subsequent values of variables are compared with the reference set of data. Correlated data which falls outside of the reference set of data will indicate unusual behaviour, and may trigger an event, for example the sounding of an alarm.

Referring in detail to Figure 1, a thermal imaging camera 1 located in a room (for example a living room) provides input information. The input information passes to a multiple motion monitor 2, and from there to a layout monitor 3 and a periodicity monitor 4. A vertical acceleration monitor 5 is connected to the layout monitor 3.

The input received at the multiple motion monitor 2 from the thermal imaging camera 1 comprises one or more 'blobs', which are thermal images of people located in the living room. The multiple motion monitor 2 determines whether the thermal image represents a single person or more than one person, using grey level segmentation and tracking. Position information is used to extract a sub-region of the image received from the thermal imaging camera 1, and the sub-region is then classified to extract information regarding the pose of each person. The number of people located in the room is output from the multiple motion monitor 2 to a state comparator 7 (the number of people may of course be zero).

Output from the multiple motion monitor 2 relating to the pose and position of people located in the living room passes to the layout monitor 3. The layout monitor 3 is also provided with situation specific reference data from a memory 6. This data may include locations where sitting is normal, for example a chair, and locations where the subject may occasionally lie down, for example on a settee. The layout monitor 3 determines whether the subject is moving, and if the subject is not moving, whether the pose and position of the stationary subject is acceptable. The layout monitor outputs a series of flags to the state comparator 7, indicating whether the subject is sitting, lying, the location of the subject, etc. One flag of particular importance is the 'potential fall configuration' flag, which will be output for example if the subject is lying on the floor.

The triggering of an alarm in response to the 'potential fall configuration' flag is delayed until data from a vertical acceleration monitor 5 has been reviewed. Thus for example, where the subject has been lying on the floor for more than 30 seconds, the layout monitor 3 outputs the 'potential fall configuration' flag to the vertical acceleration monitor 5, which reviews recent data relating to vertical acceleration. If the subject accelerated faster than a threshold value, this indicates that the subject has fallen, and the vertical acceleration monitor 5 will output a 'fall' flag to the state comparator 7 which will then trigger and alarm. The acceleration is monitored using a filter technique.

A periodicity monitor 4 receives output relating to the position and pose of the subject from the multiple motion monitor 2. The periodicity monitor 4 is configured to detect temporal patterns in the motion of the subject, and determines whether the subject is moving in a periodic manner that indicates distress, for example pacing backwards and forwards. The periodicity monitor 4 outputs an 'abnormal behaviour' flag to the state comparator 7 when such periodic movement is detected.

The state comparator 7 generates a body of data representing normal behaviour of the subject being monitored (for example during a 'learning phase'). Each variable monitored by the system, for example pose, position, periodicity, etc. is correlated with the other variables. The normal behaviour of the subject is thus represented by a multidimensional body of correlated variables (the reference set of data). For example, the subject will often sit on a chair. Thus, over time the position of the subject (on the chair) will be correlated with the pose of the subject (sitting) so that this combination of position and pose is part of the reference set of data which indicates normal behaviour of the subject. In a further example, the subject may walk behind a piece of furniture, for example a chair, whereupon part of the subject will be occluded from view by the camera 1. The change of shape (effectively the pose) of the subject will be correlated with the location of the subject (i.e. the position at which the table is located).

A further variable, time, is input to the state comparator 7. This variable is also incorporated into the multidimensional body of correlated variables (i.e. the reference set of data). In the example given, the reference set of data will correlate the position of the subject (on the chair) with the pose of the subject together with the time at which that combination occurs. For example, the subject may often sit in the chair between 7 am and 11 pm, but never between 11 pm and 7 am, and this will influence the formation of the reference set of data. If the system determines that the subject is sitting in a chair at 3 am, then this combination of position, pose and time will fall outside of the reference set of data, and the state comparator 7 will take appropriate action, for example the triggering of an alarm.

The reference set of data may be constructed during a 'learning phase', for example two weeks following installation of the system. At the end of a learning phase, a technician may adjust the reference set of data by retaining data that is considered to be pertinent, and discarding data that is not required. For example, if a person being monitored by the system fell to the floor during the learning phase, the combination of data associated with this fall may be removed from the reference set of data in order to prevent the system from accepting a future fall as being within the parameters of normal behaviour.

Situation-specific reference data may be input into the system, via the memory 6, by the technician during installation of the system. This may be done by imposing limits upon certain variables within the reference set of data. The situation-specific data may for example define certain regions of a room being monitored as being off limits, such that entry into those regions will cause the state comparator 7 to trigger an alarm.

Habit-specific temporal data may be input into the system via a second memory 8. For example, somebody may be close to a window between 10am and 11am on a Tuesday, since this is when a window cleaner visits. The set of data relating to a window cleaner may be added to the reference set of data via different methods. In a first method a technician cleans the window (or asks somebody else to clean it), whereupon the system will record the set of data that are associated with window cleaning. The technician may then link that set of data with a particular time of the week, for example Tuesday mornings. A window cleaner will thus fall within the reference set of data if he cleans the window on a Tuesday morning, but will fall outside of the reference set of data if he cleans the window at any other time, thereby causing the state comparator 7 to trigger an appropriate action. A second method via which information regarding the window cleaner may be added to the reference set of data is simply via the learning phase of the monitoring system. If the window is cleaned every Tuesday morning, then over time the combination of the set of data relating to window cleaning and the time at which the window is cleaned will be added automatically to the reference set of data. A limitation of this second method is that it may take a long time to define the reference set of data.

The reference set of data may be fixed once the learning phase has been completed and a technician has adjusted the data (if adjustment is required). Where this is done, subsequent events will not cause the reference set of data to be modified. Alternatively, the reference set of data may be updated indefinitely, such that the learning phase is effectively unending.

The system may occasionally generate false alarms. This is likely to happen whenever a person being monitored behaves in an unusual manner. For example, a person being monitored may occasionally get out of bed in the middle of the night to observe astral events such as lunar eclipses, comets, etc. If this did not happen during the learning phase, then the combination of values of variables corresponding to the person being adjacent to a window in the middle of the night will cause the state comparator 7 to trigger an alarm. Once it has been determined by a technician that the alarm was false, in the sense that the person being monitored was not in distress, the combination of values of variables corresponding to the person being adjacent to a window in the middle of the night may be added to a set of false alarm data. The system may then be configured to check a subsequent combination of values of variables against the set of false alarm data before raising an alarm. In this case, the system will determine that the combination of values of variables is a false alarm and will take no action.

The monitoring system may be configured to take account of conditions external to a room being monitored. For example, in a retirement home, a monitoring system configured to monitor a resident's room may be linked to a fire alarm. The monitoring system may be configured such that the state comparator 7 will not trigger an alarm if a fire alarm occurs in the retirement home, unless specific conditions occur. For example, the state comparator will not trigger an alarm on account of a person getting out of bed to evacuate the retirement home in the middle of the night, but will trigger an alarm if that person falls whilst attempting to leave their room during the evacuation.

The described embodiment of the invention may be used to monitor the behaviour of a subject in a defined space, and is particularly useful where that space includes other objects, for example furniture. The described embodiment of the invention may be used to monitor the behaviour of an animal in a zoo cage, or the behaviour of an inmate in a prison cell.

The invention may be used to monitor a space to determine the presence and behaviour of unknown parties, for example a car park may be monitored for car thieves, or the windows of a house may be monitored for burglars, etc.

Where the invention is used in a car park, the state comparator 7 will generate a reference set which will include a correlation between the presence of a person adjacent a car and the time spent by a person in that position. For example, it may be typical for a person to spend 10 seconds unlocking his or her car before entering the car and thereby disappearing from the view of a detector (the detector may be a thermal imager). The monitoring system may thus determine by monitoring the car park over a period of time that it is extremely unlikely that anyone will spend more than 60 seconds in that position. Once a person has been located adjacent to a car for more than 60 seconds, the state comparator will take appropriate action, for example triggering an alarm, or drawing the attention of a security guard.

Claims

1. A monitoring system for monitoring the behaviour of an object, the monitoring system having at least one image sensor and being operative to extract variables from image sequences input from the one or more image sensor, wherein the variables are interrelated such that certain combinations of values of variables occur frequently and other combinations of values of variables do not occur during normal behaviour of the object, the values of variables and relationships between the variables are recorded by the monitoring system as a reference set of data, and after the values of the variables and the relationships between the variables have been recorded for a predetermined training time, an event is triggered by the system when an unusual combination of values of variables occurs.

2. A monitoring system according to claim 1, wherein the unusual combination of values of variables comprises a combination that does not fall within the reference set of data.

3. A monitoring system according to claim 1 or 2, wherein the values of variables and relationships between the variables are not added to the reference set of data after the predetermined training time has ended.

4. A monitoring system according to claim 1 or 2, wherein the values of variables and relationships between the variables continue to be added to the reference set of data after the predetermined training time has ended, thereby continually updating the reference set of data.

5. A monitoring system according to any preceding claim, wherein the reference set of data may be modified directly by excluding a specific range of values of a variable from the reference set of data, or including a specific range of values of a variable in the reference set of data.

6. A monitoring system according to any preceding claim, wherein the reference set of data may be modified directly by excluding a predetermined combination of values of variables from the reference set of data, or by including a predetermined combination of values of variables in the reference set of data.

7. A monitoring system according to any preceding claim, wherein the system triggers an alarm when the unusual combination of values of variables occurs.

8. A monitoring system according to claim 7, wherein subsequent to the alarm being triggered it is determined whether the alarm was a false alarm, and if the alarm was a false alarm the combination of values of variables that triggered the alarm is added to a memory.

9. A monitoring system according to claim 8, wherein a subsequent combination of values of variables that will trigger an alarm is compared with combinations of values of variables held in the memory in order to determine whether the combination corresponds to a false alarm.

10. A monitoring system according to any preceding claim, wherein if the combination of values of variables lies within a predetermined range, the system reviews preceding values of at least one variable to determine whether that variable has exceeded a threshold value.

11. A monitoring system according to claim 10, wherein the system triggers an alarm when the combination of values of variables lies outside the reference set and the at least one variable has exceeded the threshold value.

12. A monitoring system according to claim 10 or claim 11, wherein the at least one variable is acceleration of the object.

13. A monitoring system according to any preceding claim, wherein time is one of the properties represented by a variable.

14. A monitoring system according to any preceding claim, wherein the position of the object is one of the properties represented by one or more variables.

15. A monitoring system according to any preceding claim, wherein the pose of the object is one of the properties represented by one or more variables.

16. A monitoring system according to any preceding claim, wherein periodicity of the behaviour of the object is one of the properties represented by one or more variables.

17. A monitoring system according to any preceding claim, wherein the system includes means for determining the number of objects within the field of view of a sensor.

18. A monitoring system according to any preceding claim, wherein the number of objects is one of the properties represented by one or more variables.

19. A method of monitoring the behaviour of an object, the method comprising inputting a sequence of images, extracting variables from the sequence of images, wherein the variables are interrelated such that certain combinations of values of variables occur frequently and other combinations of values of variables do not occur during normal behaviour of the object, recording the values of variables and relationships between the variables as a reference set of data, and after the values of the variables and the relationships between the variables have been recorded for a predetermined training time, triggering an event when an unusual combination of values of variables occurs.

20. A monitoring system substantially as hereinbefore described with reference to the accompanying figure.

1. A method of monitoring substantially as hereinbefore described with reference the accompanying figure.