WO2004054223A1

WO2004054223A1 - Arranging motion detection in mobile station

Info

Publication number: WO2004054223A1
Application number: PCT/FI2003/000945
Authority: WO
Inventors: Juhani Kallio; Tapani Salmi; Jani Helin; Tapani HUHDANMÄKI; Mika Kuusisto
Original assignee: Patria Ailon Oy
Priority date: 2002-12-12
Filing date: 2003-12-11
Publication date: 2004-06-24
Also published as: FI115277B; AU2003285381A1; FI20022193A0; FI20022193A

Abstract

The invention relates to arranging motion detection in a mobile station having a camera connected functionally thereto. The camera is controlled to define pixel values of monitored pixels at a minimum of two different time instants to form at least two images. Differences in the images are defined and on the basis of the differences in the images, a motion value is generated. The motion value refers generally to a comparison value formed by any method on the basis of differences in the image information of compared images. The motion value is compared with a predefined threshold value, and the mobile station initiates at least one function associated with monitoring if the motion value attains the threshold value.

Description

ARRANGING MOTION DETECTION IN MOBILE STATION

FIELD OF THE INVENTION

[0001] The invention relates to controlling a mobile station equipped with a camera and especially to arranging motion detection in a mobile station having a camera connected functionally to it.

BACKGROUND OF THE INVENTION

[0002] The significance of images has increased in telecommunications devices. Using a mobile station, it has been possible to transmit images produced with digital cameras for instance through electronic mail to other telecommunications devices. Standards that are more suitable than before for transmitting data have been further developed. Especially packet-switched data transmission, such as the GPRS (General Packet Radio Service) service, improves data transmission through a mobile network. A service called MMS (Multimedia Messaging Service) has also been developed for transmitting messages containing images to and from mobile stations. Mobile stations with an integrated camera are now also available. One device that combines the functionalities of mobile communication and a camera is a wireless detection camera that can be set to take pictures at certain time intervals and to transmit them using MMS messages to a specific mobile station or electronic mail address. The camera also has a motion detector, and a picture is taken and transmitted when the motion detector detects movement. This type of camera is easy to install to monitor a summer cottage, for instance, and to transmit a picture of a person breaking in to the cottage. A problem with known devices that combine the functionalities of mobile communication and a camera is that motion detection is implemented by infrared technology. Infrared radiation- based detection is quite unreliable; even certain clothing can prevent the detection of movement, if heat output required by the infrared sensor does not radiate through the clothing. Motion detection is also not possible through glass.

BRIEF DESCRIPTION OF THE INVENTION

[0003] It is an object of the present invention to develop a method and an apparatus implementing the method so as to improve the functioning of motion detection in a mobile environment. The object of the invention is achieved by a method, mobile station and computer program product that are characterized by what is stated in the independent claims. Preferred embodiments of the invention are disclosed in the dependent claims.

[0004] According to the invention, a mobile station having a camera functionally connected to it monitors motion on the basis of a digital analysis of differences in images produced with the camera. The camera is controlled to define the pixel values of monitored pixels at a minimum of two different time instants to form at least two images. The differences in the images are determined and a motion value is generated on the basis of the differences. The motion value refers generally to a comparison value formed by any method on the basis of differences in the image information of compared images. The motion value is compared with a predefined threshold value and at least one function associated with monitoring is initiated in the mobile station, if the motion value attains the threshold value. The mobile station has a sensitivity control function, on the basis of which the threshold value can be set.

[0005] The arrangement of the invention provides the advantage that motion detection can be made more reliable than before. The solution of the invention differs completely from earlier devices that combine the functionalities of a camera and mobile station, because in the solution, the camera can repeatedly define image information, from which movement is directly analyzed. This way, no separate detection sensors are needed to trigger the taking of the image. By digital analysis of consecutive images, it is possible to detect movement very accurately within the monitored area. With the mobile station of the invention, it is very easy to arrange temporary monitoring, because the monitoring can be set independently and wirelessly with the mobile station that is carried along in any case. With the motion detection of the invention that is based on image information generated by a camera, it is also possible to achieve a shorter reaction time when movement is detected. Because the detection is not based on thermal radiation, targets not radiating heat are also detected, movement can be monitored even through glass, i.e. the camera can be protected from weather, it is possible to detect if the camera is covered or moved, and errors are not generated by thermal flows (for example when an electrical heater is switched on). Because the detection does not use the curtain technique generally used in infrared sensors, which requires a moving target to pass through a plane or line segment formed by one or more devices, the monitored area does not have sectors or areas, from which motion detection is not possible, and movement is detected as quickly near and far. Sensi- tivity control makes it possible to take into account different environments and the quality of the monitored movement, and consequently further improve the reliability of motion detection in various environments. With sensitivity control, it is possible to set a threshold value so that a certain type of movement does not yet trigger motion detection and a possible alarm, for instance the movement of a pet does not exceed the threshold.

[0006] According to one embodiment of the invention, the user is shown a sensitivity control window, which shows at least a motion value updated continuously on the basis of the comparison of differences in the monitoring values of consecutive images, and the threshold value. The user is given the option of setting said threshold value and the threshold value is then set on the basis of the input from the user. This way, sensitivity control aids in taking into consideration different environments and the quality of the monitored movement, and improves the reliability of motion detection.

BRIEF DESCRIPTION OF THE FIGURES

[0007] The invention will now be described in greater detail by means of preferred embodiments and with reference to the attached drawings, in which

Figure 1 illustrates a mobile station with a camera connected thereto,

Figure 2 illustrates a method according to a preferred embodiment of the invention, and

Figure 3 illustrates a method for motion detection according to a preferred embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

[0008] Figure 1 illustrates a mobile station MS having a camera functionality CA connected to it. The mobile station refers generally to a device comprising the means necessary for implementing wireless data transmission. Examples of devices of this type are devices capable of data transmission with a PLMN network, such as a GSM/GPRS network or a third-generation network (e.g. 3GPP system). It should be noted that the invention could also be applied to mobile stations equipped with a camera, in which the camera is not integrated to the mobile station, but a camera producing image information is connected by wire or wirelessly to the mobile station. [0009] The mobile station MS contains memory MEM, a user interface Ul, input/output means I/O for arranging external data transmission, and a central processing unit CPU comprising one or more processors. The memory MEM has a non-volatile section for storing applications controlling the central processing unit CPU and other necessary information and a volatile section for use in temporary data processing. The mobile station MS illustrated in Figure 2 also comprises a camera CA that can use any known digital imaging technique and convert light into electric charges in an image sensor, define RGB values for pixels and convert them into digital format. It should be noted that the camera CA might be capable of recording both still images and video images. The central processing unit CPU performs the processing, such as compression, and storage into the memory MEM of digital image information.

[0010] According to a preferred embodiment of the present invention, the mobile station MS runs a monitoring application MA that detects movement in the area monitored by the camera CA on the basis of the image information received from the camera CA and initiates a function, such as alarm, associated with the monitoring application MA. Computer program codes executed in the central processing unit CPU can cause the mobile station MS to run the monitoring application MA, a few embodiments of which are illustrated in Figures 2 and 3. The functions of the monitoring application MA can be distributed into several separate software components that communicate with each other. The computer program can be stored in any memory means, such as the hard disc of a PC or a CD-ROM disc, from which it can be loaded into the memory MEM of a device MS running it. The computer program can also be loaded from the network by using a TCP/IP protocol stack, for instance. It is also possible to use hardware solutions or a combination of hardware and software solutions to implement the means of the invention.

[0011] Figure 2 illustrates a preferred method for the arrangement of the monitoring application MA. The monitoring application MA can be activated when the mobile station MS is placed in the correct location for monitoring. According to a preferred embodiment of the invention, the display of the mobile station MS can be utilized to find the correct monitoring area, i.e. the image is updated according to the position of the mobile station MS. This embodiment facilitates targeting in comparison with the prior-art solutions. Monitoring can be started by a timer at a set time or monitoring can be set to start after a time period specified in the settings and it may continue during a set time or until it is stopped. According to one embodiment, monitoring can also be started or interrupted from another terminal for instance by using a short message SMS (short message service). When the mobile station MS activates monitoring, the camera CA is controlled 201 to produce images at specified intervals, i.e. to define the pixel values of the pixels set for monitoring and to transmit them in digital format for the monitoring application MA to process. The camera CA is preferably controlled to use only part of the pixels used in a typical image. The image can be smaller or the number of pixels in the images can be rastered to be smaller, whereby the image area does not decrease. Alternatively, the monitoring application limits the number of compared pixels. This embodiment limiting the number of pixels makes it possible to limit the need for processing capacity and thus also the consumption of battery capacity, which is important in mobile stations. Motion detection also becomes faster when only part of the available pixels is compared. Images are captured using the frame rate set in the mobile station MS. The frame rate can preferably be selected in the settings of the monitoring application MA, for instance from the three alternatives: 0.5, 1 and 10 seconds. A sufficient frame rate is determined for instance on the basis of the speed of the monitored movement. Before activating the monitoring application MA, the user can test the adequacy of the selected frame rate and set it to suit the context.

[0012] On the basis of the image information received 202 from the camera CA, the differences in consecutive images are defined 203, and on the basis of the differences, a motion value is generated 204 to illustrate the magnitude of the change between images. The differences in the images can be derived directly from uncompressed image information (pixel values) received from the camera CA, or from compressed image information. After this, the motion value is compared 205 with a predefined threshold value to check whether the motion value attains the threshold value.

[0013] If the motion value does not attain 205 the threshold value, monitoring is continued and the routine moves to step 202 to wait for the pixel values of the next image that can be compared with the earlier image in step 203. If on the basis of the comparison 205, the motion value attains the threshold value, at least one function associated with monitoring is started 206 in the mobile station MS. In step 206, one or more of the functions described in the following can be performed. [0014] On the basis of the movement, the camera CA can be controlled to take a final image, for which a typically higher number of pixels is used than for just motion detection. In such a case, the pixel values of pixels according to the image size and quality selected in the settings are preferably received from the camera CA and the image is typically compressed to JPEG format, for example. When motion is detected, images can be taken according to the settings. If the mobile station MS is arranged to record images frequently enough, several images are available on the cause of the movement, and a moving image can be generated. Alternatively, the pixel values defined for monitoring are used (the image is stored and/or transmitted), whereby the image is even more up-to-date.

[0015] The captured image can in step 206 be stored into the memory MEM of the mobile station MS. A certain limit for the number of stored images can be defined in the settings or images can be stored until the memory space is full.

[0016] The captured image can be transmitted to another telecommunications device by using a data transmission service supported by the mobile station MS and the mobile network serving the mobile station MS. The image can be transmitted for instance using an MMS service or by electronic mail. The mobile station MS can be connected to the network and transmit the image immediately after it is taken. When using a circuit-switched service, a call is set up frorn the mobile station MS to a specific number, to which the image is transmitted. The address, such as the IP address, of the other telecommunications device can be set in the mobile station MS, and the image is then addressed thereto. Instead of transmitting the image through the mobile network, it can also be transmitted to another device in another manner, for instance by using a short-range radio connection. The image can be transmitted according to the connection settings of the monitoring application MA for instance to another terminal of the user or to a device of the monitoring system.

[0017] Based on detecting motion and/or taking an image, it is possible to transmit a notification to another telecommunications device. As with transmitting an image, the notification, too, can be transmitted using any service provided by the mobile station MS and the mobile network associated with it. An especially efficient manner is to transmit the notification as a short message SMS, which is transmitted to the specified number immediately. If the number cannot be reached, the short message is stored in the mobile network and transmitted to the number as soon as the terminal identified by the number connects to the network. On the basis of the received notification, the user or an employee of a security company, for instance, very quickly receives information on detected movement in the monitored area.

[0018] In step 206, a local alarm can be given, in which case a strong sound alarm can be repeated in the mobile station and/or use other effects available in the mobile station MS, such as blinking the display.

[0019] According to one embodiment of the invention, the sensitivity of motion detection can be adjusted by changing the threshold value. The user can set a specific threshold value or select one of the preset defaults. There may be several default values, and they can be set so as to make motion detection sufficiently sensitive. The user interface Ul can display a sensitivity control window, which shows the measured motion value and the set threshold value in real time. On the basis of this sensitivity control window, the user can easily set a correct motion detection level, i.e. threshold value, for different spaces and situations. For instance, the flickering of fluorescent lamps causes changes comparable to motion in the image information, so the threshold value must be high enough not to cause erroneous motion detection due to such noise. The window can display not only the current motion value but also the minimum and maximum values during the monitored time as bars and numerical values to further facilitate the setting of the correct threshold value. The values can be scaled for instance within 0 to 100 for ease of use. One alternative to implement the sensitivity control window is to draw a time-wise descriptor showing the motion value. The control of sensitivity (threshold value) can also be done automatically in accordance with an embodiment by defining a sufficient threshold value on the basis of the motion value determined during the control time. This embodiment further facilitates the use of the device. The threshold value is preferably adjusted before the start of monitoring so as to make sure that the threshold value is sufficient. The threshold value can then be set in such a manner for example that the movement of a pet does not trigger the execution of step 206. After the user has set the threshold value (in step 201 ), the monitoring application MA preferably waits until a predefined delay has elapsed before starting to monitor and controlling the camera to take images. Several threshold preselection alternatives can also be set for the monitoring application MA, such as sensitive, normal, rough. The user can di- rectly select from the preselection, which makes it yet easier to start to use the monitoring application MA.

[0020] The mobile station MS preferably also has a darkness function that observes the effect of brightness. In connection with motion detection, it is also possible to calculate from the RGB values of the pixels a brightness value representing the brightness of the image, which is used when deciding on the light sensitivity setting, such as day or night setting, for the camera CA. The brightness value can be calculated as an average of the RGB values of the pixels in the image area. Three degrees of brightness, dark, normal and bright, may for instance be defined for the sensitivity setting of the camera CA, with the middle one serving as the hysteresis area. The camera CA is set in the night setting, when the brightness value is within the dark area for a predefined time, and back to day setting, when the degree of brightness is within the bright area for a predefined time. This way, the mobile station can also control the camera CA during monitoring on the basis of the image information.

[0021] According to one preferred embodiment, the mobile station MS maintains historical information, a log, on detections during a monitoring period. Time instants when movement has been detected, and other information, such as the maximum and minimum motion values, can be stored in the log.

[0022] Figure 3 illustrates a motion detection method that can also be applied to the monitoring application illustrated above in Figure 2. When the pixel values of a new image are received 301 from the camera CA, pixel value pairs An;Bn and An+1 ;Bn+1 of the new image, called image B, and a previous image, image A, are defined 302. n and n+1 show consecutive pixels, so pixel value pairs An;Bn and An+1 ;Bn+1 show the same consecutive pixels of two consecutive images. The differences in the pixel value pairs are determined 303, after which it is checked 304, whether the pixel value pairs contain a difference that exceeds a first threshold value. The differences in the pixel value pairs can be summed and the summed value compared with the first threshold value. According to one preferred embodiment, the difference found in the pixel pairs is compared with the first threshold value one pixel pair at a time, and both pixel pairs must have a difference exceeding the first threshold value to be able to move on from step 304 to step 305. A discrepancy in a single pixel can be ignored in this embodiment. The first threshold value is preferably set as default to the monitoring application MA in such a manner that the difference in the pixels needs to be sufficiently large.

[0023] If the pixel value pairs have a difference that exceeds the first threshold value, the value of a counter maintained in the method is increased 305. If a large enough difference does not exist, the routine moves directly to step 306, where it is checked, whether all pixel value pairs to be monitored have been processed. If all pixel values have not been processed, the routine returns to step 302 to define the next pixel value pairs (An+1 ;Bn+1 and An+2;Bn+2).

[0024] When, on the basis of the check in step 306, all the pixel value pairs set to be monitored have been processed, the final counter value is obtained, which represents the amount of detected movement in the image pair. It is then possible to move to step 307, in which the final counter value is processed. The counter value is processed to provide the desired sensitivity with respect to the amount of movement. For this weighting, a mathematical weighting function can be used, or a weighted value corresponding to the counter reading can be fetched from a stored weighting table. With weighting, it is possible to emphasize a change caused by a small movement in a value representing the amount of movement (motion value). One way is to apply a logarithmic curve. As earlier mentioned, a calibration window depicting motion can be shown to the user, whereby a counter value processed to improve clearness can be scaled within 0 to 100, for instance (within which range the threshold value of the motion value is also set). The processed counter value is checked 308 to see whether it attains a preset second threshold value. If the processed counter value does not exceed the second threshold value, no essential movement has taken place between the examined consecutive images, in which case the counter is set to zero, and the routine can move to process the pixel values of a new received image in step 301. If the processed counter value attains the second threshold value, the function associated with monitoring and already illustrated above in Figure 2 is started 309.

[0025] It should be noted that step 307 is not necessary, and it is possible to directly compare the obtained counter value with the set second threshold value in step 308. When comparing Figure 3 with Figure 2, the processed counter value corresponds to the motion value of Figure 2, and the second threshold value used in step 308 corresponds to the threshold value for motion detection used in step 205 of Figure 2. The motion value is determined on the basis of the differences in the pixel value pairs (the counter is incremented and the obtained counter value is processed).

[0026] This method provides the advantage that it is possible to implement a sufficiently reliable and quickly reacting motion detection in devices with a limited processing capacity. The number of pixel values monitored for motion detection can be adapted according to the available processing capacity, including also the battery capacity. The motion detection method of Figure 3 used pixel pairs for comparison; in the comparison, it is possible to use any combination of adjacent pixels or to compare individual pixels directly. By comparing pixel pairs, it is, however, possible to filter random pixel noise.

[0027] According to one preferred embodiment of the invention, the monitoring application MA monitors movement, and if an essential change takes place in the movement, a function associated with monitoring is activated. This embodiment can be implemented utilizing the embodiments already described in connection with Figures 2 and 3, because a reduction in movement can equally well be detected on the basis of the differences in the images. When the difference exceeds the threshold value, the movement is considered to have reduced or stopped, and for instance a notification is transmitted to another telecommunications device.

[0028] According to yet another preferred embodiment of the invention, the monitoring application MA of the mobile station MS can also notify that no movement has been detected. MA is then preferably set to monitor movement during a period of time selected by the user, and if no movement is detected, MA transmits for instance a notification to another telecommunications device.

[0029] The following describes a few exemplary situations, in which the solution of the invention can be used. The user can set his or her mobile station MS to monitor the garage at 11 p.m., after which the mobile station MS transmits a short message SMS on any detected movement to a second mobile station of the user. The mobile station MS can be positioned towards the door of a hotel room and to trigger a loud alarm, if there is movement at the door. The mobile station MS can be left home in a suitable place when leaving home. The monitoring application MA of the mobile station MS can be activated with a short message, after which the mobile station transmits an image, if motion is detected. Other exemplary situations to which the solution can be applied are access control and nature watch. According to a preferred em- bodiment, user profiles with suitable threshold values, for instance, can be defined for these various situations. The user can select directly on the basis of the context a user profile, the settings of which are set when the monitoring of the monitoring application MA is started.

[0030] It is obvious to a person skilled in the art that while the technology advances, the basic idea of the invention can be implemented in many different ways. The invention and its embodiments are thus not restricted to the examples described above, but can vary within the scope of the claims.

Claims

1. A method for arranging motion detection in a mobile station having a camera connected functionally thereto, which method comprises controlling the camera to define pixel values of monitored pixels at a minimum of two different time instants to form at least two images, defining differences in the images, generating a motion value on the basis of the differences in the images, comparing the motion value with a predefined threshold value, and initiating in the mobile station at least one function associated with monitoring in response to the motion value attaining said threshold value, characterized by setting said threshold value by means of a sensitivity control function.

2. A method as claimed in claim ^characterized by defining at least two pixel value pairs from the images, with one

^• pixel value pair comprising the pixel values of corresponding pixels in the two compared images, comparing the differences in the pixel values in each pixel value pair, incrementing a counter value in response to the compared pixel value pairs exceeding a predefined first threshold value, weighting the counter value in response to having processed all pixel value pairs formed of the monitored pixels, checking whether the weighted counter value exceeds a predefined second threshold value, and initiating in the mobile station at least one function associated with monitoring in response to the weighted counter value attaining the predefined second threshold value.

3. A method as claimed in claim 1 or 2, characterized in that said function comprises taking an image, whereby the camera is controlled to define the values of the pixels set for capturing the image.

4. A method as claimed in any one of the preceding claims, characterized in that said function comprises storing an image, whereby at least one image is stored in the memory of the mobile station.

5. A method as claimed in any one of the preceding claims, characterized in that said function comprises transmitting an image, whereby the image is transmitted to at least one predefined telecommunications device in response to the storing of the image.

6. A method as claimed in any one of the preceding claims, characterized in that said function comprises transmitting a notification, whereby the notification is transmitted to at least one telecommunications device in response to the storing of an image.

7. A method as claimed in any one of the preceding claims, characterized by displaying to the user a sensitivity control window, which illustrates at least a motion value updated continuously on the basis of the comparison of differences in the monitoring values of consecutive images, and the threshold value, offering the user the option of setting said threshold value, and setting said threshold value on the basis of the input from the user.

8. A mobile station having a camera functionally connected thereto, characterized in that the mobile station is arranged to control the camera to define pixel values of monitored pixels at a minimum of two different time instants to form at least two images, the mobile station is arranged to define differences in images on the basis of the pixel values received from the camera, the mobile station is arranged to generate a motion value on the basis of the differences in the images, the mobile station is arranged to compare the motion value with a predefined threshold value, the mobile station is arranged to initiate at least one function associated with monitoring in response to the motion value attaining the predefined threshold value, and the mobile station also comprises a sensitivity control function for setting said threshold value.

9. A mobile station as claimed in claim 8, c h a r a c t e r i z e d in that said function is at least one of the following: sound alarm, taking a new image, storing a image, transmitting a image to at least one telecommunications device, or transmitting a notification to at least one telecommunications device.

10. A mobile station as claimed in claim 8 or 9, c h a r a c t e r i z e d in that the mobile station is arranged to display to the user a sensitivity control window that shows at least a motion value updated continuously on the basis of the comparison of differences in the monitoring values of consecutive images, and the threshold value, the mobile station is arranged to offer the user the option of setting said threshold value, and the mobile station is arranged to set said threshold value on the basis of the input from the user.

11. A computer program product for controlling a mobile station, c h a r a c t e r i z e d in that the computer program product comprises: a computer program code portion for controlling the mobile station to control a camera connected functionally thereto to define pixel values of monitored pixels at a minimum of two different time instants to form at least two images, a computer program code portion for controlling the mobile station to define differences in the images on the basis of pixel values received from the camera, a computer program code portion for controlling the mobile station to generate a motion value on the basis of the differences in the images, a computer program code portion for controlling the mobile station to compare the motion value with a predefined threshold value, a computer program code portion for controlling the mobile station to initiate at least one function associated with monitoring in response to the motion value attaining the predefined threshold value, and a computer program code portion for controlling the mobile station to implement a sensitivity control function for setting said threshold value.