WO2008095624A1

WO2008095624A1 - Method for initializing a device for tracking a person's face

Info

Publication number: WO2008095624A1
Application number: PCT/EP2008/000630
Authority: WO
Inventors: Bertrand Godreau
Original assignee: Continental Automotive France
Priority date: 2007-01-30
Filing date: 2008-01-28
Publication date: 2008-08-14
Also published as: FR2911978B1; FR2911978A1

Abstract

To follow the movements of a face using a method based on reconstructing a 2D model, an initialization method comprises a search for symbolic points on successive images of the face forming a sequence of images. Each symbolic point detected on a current image is stored together with its imagette extracted from the current image and, when all the symbolic points desired have been detected at least once in the sequence of images, the initialization is terminated by registration of the imagettes on a current image and a specific 2D model of the face is constructed. The method makes it possible to begin the tracking phase before the initialization phase is terminated, in particular when at least two symbolic points have been identified in the sequence of images.

Description

Method for initiating a device for tracking a person's face

The present invention belongs to the field of monitoring the vigilance of a person and its areas of interest during an activity such as driving a vehicle. More particularly, the invention relates to a method for initializing a device for tracking the position and / or movements of a face in a sequence of images.

To monitor the vigilance of a person and their areas of interest, one method is to observe the face of this person by means of a camera delivering images whose analysis can identify movements such as those of the head , eyes and eyelids and to deduce the direction towards which the filmed person looks and with what attention.

It is known to use for this purpose one or more video cameras and to submit the electronic images delivered by these cameras to digital processing to extract the information sought.

A known method for tracking the movements of the face consists, in an initialization step, of constructing a two-dimensional representation of the face of the person filmed, called the specific 2D model of the face, and then, in a tracking step, to follow the movements and deformations of this 2D representation to deduce the information sought on the position and orientation of the person's face. In order to construct a specific 2D model during an initialization step, a known method consists, as illustrated in FIG. 1, in:

Acquiring an image 10b of a face to be followed;

• locate 10c the face on the image;

• identify 11 PS symbolic points on the facial image; • in the hypothesis 12 where all the sought symbolic points have not been identified on the image, start again a search cycle of the symbolic points with a new image until all the searched points are identified on a current image ;

• memorize 13 thumbnails corresponding to each symbolic point, thumbnails extracted from the current image;

• build 14, from the positions of the symbolic points in the current image, the specific 2D model of the observed face.

Such methods of monitoring the movements of a face are known for example from the international patent application published under the number WO 02/09025. One of the difficulties of these tracking methods comes precisely from the initialization step in which symbolic points used to place the thumbnails and build the 2D model are searched in an image.

To carry out a quality pursuit, it is indeed necessary to have identified several symbolic points and, when all the symbolic points sought have not been identified on a first image the search is restarted on the following image and so on on the successive images until all the desired symbolic points have been identified on the same image.

The different morphologies of faces, as well as for the same face the different conditions of observation: lighting, position, orientation of the face, position of the eyelids and lips ... explain that the results of the initialization phase are unpredictable.

The outcome of an initialization phase turns out to depend in large part on the sensitivity of the methods used to identify the symbolic points in the images, a sensitivity that corresponds to a compromise between, on the one hand, the duration and the quantity calculations necessary to identify a symbolic point sought and secondly the risk of a bad detection in which a point is wrongly identified as a symbolic point sought.

If the means implemented to identify the desired symbolic points are not very discriminating, the rate of successful initializations is high but the initializations are bad qualities because of the probable use of points to build the specific 2D model that do not correspond. at the symbolic points sought.

If the means implemented to identify the symbolic points sought are highly discriminating, the rate of unsuccessful initializations is high because of the systematic non-detection of all the points sought on an image and the initialization time is found affected.

The present invention specifically aims at a method of tracking a face implementing an initialization method which accelerates the initialization phase without affecting the quality of the initialization and which allows to start the tracking phase while the initialization phase is not fully completed.

To obtain these results, during the initialization step: a) pre-defined PS symbolic points of the face are identified on an image of the face; b) an image of reduced size associated with each identified symbolic point, called a thumbnail, is extracted from the image of the face; c) a two-dimensional model of the pursued face, called specific 2D model, is made from the positions of the detected symbolic points; and, when during the initialization step only a part of the symbolic points PS is identified on a first image of a sequence, d) the identified symbolic points and the associated thumbnails extracted from said first image are stored; e) the undetected symbolic points are searched on successive images of the sequence; f) each symbolic point detected in an image of the sequence is, at least for the symbolic points not yet detected, stored as well as the thumbnail corresponding to said point extracted from said image of the sequence; g) the initialization step is considered complete only when each symbolic point PS has been identified at least once on an image of the sequence.

Preferably, to improve the quality of the detection of symbolic points and thumbnails, furthermore: h) all PS symbolic points are searched in each analyzed image of the sequence; i) when a symbolic point is detected in an image being analyzed, said current image, a score representative of the quality of the identification of the point as a symbolic point is attributed to said point of said image, and; j) when a symbolic point is identified in the current image and has also been identified in a previous image of the sequence: j1) its score in the current image is compared with a stored score obtained in a previous image for said point; j2) the memorized image corresponding to said symbolic point during detection of said point in the previous image:

- is retained if the score in the current image is worse than the memorized score of the previous image; is replaced by a thumbnail extracted from the current image if the score in the current image is better than the stored score of the previous image.

To take into account that the thumbnails stored during the initialization are not all extracted from the same image of the face, the thumbnails of the different symbolic points are aligned on an image of the sequence, when each symbolic point PS has been identified. at least once in different footage of the sequence, using a generic two-dimensional geometric face pattern.

Advantageously because of the morphological constants of a human face, at least six symbolic points are sought, corresponding to the outer and inner corners of the eyes and the left and right angles of the mouth of the face.

The invention also relates to any device for tracking a face using a two-dimensional face model comprising such an initialization phase. In addition, the memorization of the symbolic points PS and the images being made on the fly on each new image, the tracking phase is advantageously started before the end of the initialization phase, especially when at least two PS symbolic points have been identified.

The detailed description of an embodiment is made with reference to the figures which represent:

FIG. 1: a schematic view of the steps of a known initialization process already mentioned;

• Figure 2: an illustration of a sequence of image of a face and identification of symbolic points;

FIG. 3: a schematic view of the steps of the initialization method according to the invention; A method of initializing a face tracking device of a person according to the invention consists in identifying a series of p particular predetermined points, called PS symbolic points, on images of a face 1, to be associated with each one. of these p symbolic points a thumbnail corresponding to a fraction of the image around the corresponding symbolic point and to build from these points a specific 2D model of the face whose movements must be followed.

These symbolic points are chosen from identifiable image points, called natural points, which correspond to points of convergence of lines separating contrasting areas of the image.

The detailed description of the invention is illustrated by examples corresponding to a tracking device using six symbolic points PS, p = 6, corresponding to the corners of the eyes and the mouth. The symbolic points are in this case:

• for the right eye:

> the outer corner of the eye or OCR, corresponding to a junction of the upper and lower eyelids;

> the inner corner of the eye or RIC; • for the left eye:

> the outer corner of the eye or LOC;

> the inside corner of the eye or LIC;

• for the mouth:> the left corner of the mouth or LMC, corresponding to a junction of the upper and lower lips;

> the right corner of the mouth or RMC.

Other symbolic points, in addition to or in replacement of these, can be used as soon as they are identifiable on any face. The device comprises image acquisition means, in particular at least one video camera and image memories, able to deliver successive images constituting a sequence il, i2 ... if, as illustrated in FIG. 2.

We consider that the first image of the sequence is an image iO, and the first image taken into account in the initialization process is identified by the index il. In a conventional image acquisition system, the images of a sequence succeed each other with a frequency of the order of ten hertz or a few tens of hertz.

In a first step the device acquires 20a an image 20b having a face 1, identifies 20c the face on the image then searches for the symbolic points PS (J), for j = 1 to p, on the image iO of the sequence .

If the p symbolic points are identified, the thumbnails corresponding to each symbolic point are extracted from the image and a specific 2D model of the face is determined from the relative position of the symbolic points. The initialization phase is in this situation considered complete. However, in most cases, the PS symbolic points will not be extracted from the image iO, especially if the search for symbolic points is very discriminating and / or the means used to generate these images are of limited performance, for example. example of low resolution video cameras such as those implemented in low cost devices. If no symbolic point is identified on the image iO, the process of searching for the symbolic points is repeated with the successive successive images until at least one symbolic point PS (j) is effectively identified.

Where appropriate, in a particular mode of implementation of the invention, for each symbolic point PS (j) identified a note, called S-PSG score (O, the use of which is described later, is assigned 23 to said symbolic point PS (j) identified on said index image i according to criteria representative of a quality of the identification of the point. For example, characteristic values of the image around the point, such as contrast gradients, are compared with typical values of known symbolic points and the score assigned is even higher than the values associated with the point considered are close to typical values. In addition, a thumbnail is extracted from the index image i and is associated with said symbolic point.

The positions, scores and thumbnails of the detected points are stored.

For example, in FIG. 2, the ROC point is identified firstly on an image 11 and an S-ROC score (M) is associated with this point of the said image from which is extracted the image 11 of ROC. In Figure 2 the thumbnails are symbolized by a rectangle in which is inscribed the index of the image from which has been extracted said thumbnail.

In a second step, the symbolic points PSQ) are again searched on a next image of the sequence until an image i2 on which symbolic points are detected. As in the search described above, a score is assigned to each identified symbolic point and a thumbnail is, if necessary, extracted from the image for each symbolic point identified on the image i2.

In a first mode of implementation of this step of the method, only the PS symbol points previously undetected on images of a rank lower than the image being analyzed are searched.

In a second preferred mode, corresponding to FIGS. 2 and 3, for implementing this step of the method, all the symbolic points PS are again searched for on each image of the sequence and, when a symbolic point, for example the point ROC identified in an image i3, has already been identified in a previous image, the image it in the example, its new score, S-ROC (i3), is compared 24 to the score that said symbolic point had obtained previously, S -ROC (H) in the example. If the new score is better than the previous score, S-RIC (i3)> S-RIC (M), the thumbnail associated with the symbolic point considered is extracted from the image i3 and is stored in place of the thumbnail corresponding image extracted from the image M (case of FIG. 2).

In the opposite case, the previously extracted thumbnail is preserved, for example in FIG. 2 for the LOC point when S-LOC (i3) <S-LOC (i2).

For the symbolic points identified in an image that have not been identified in a previous image, the LOC and LMC points for the image i2 of FIG. 2, the said symbolic points are as previously described, each associated with a score and a thumbnail extracted from the image being analyzed. The process just described is repeated iteratively with the successive images of the sequence up to an image of rank if for which all the searched symbolic points have been identified at least once and associated thumbnails are stored. , even if no image of the analyzed sequence was able to allow on its own the identification of all these symbolic points.

In addition, the thumbnails associated with the symbolic points are of better quality insofar as, extracted from images that may be different, for each symbolic point the most representative image of the point considered is stored among all the possible thumbnails of the images of the sequence for this symbolic point. Some thumbnails having been extracted from images of the sequence prior to the last image if, the current image when all the desired symbolic points have been identified, the positions of the various thumbnails are adjusted 27 with respect to said current image.

Advantageously, this registration of the positions is carried out by using a generic 2D model of face 7 and a filtering which makes it possible to reconstitute the 2D model specific to the treated face.

The initialization phase is then considered complete.

Thus the method makes it possible to obtain a rapid and yet high quality initialization, pledge of a subsequent successful performance and quality, by the implementation of highly discriminating symbolic determination methods, methods of determination that would not have been possible. allowed to achieve initialization in an acceptable duration by conventional initialization methods.

In a particular embodiment of the invention, the tracking phase is started before the end of the initialization phase. Indeed it is theoretically possible to carry out a continuation on the basis of a 2D model of face as soon as at least two points are identified.

Such a continuation in a degraded mode using a limited number of points can only be envisaged if the quality of detection of the symbolic points is good that the method allows by the use of highly discriminating symbolic point detection methods.

In this particular embodiment of the invention, the tracking phase is therefore advantageously started before all the symbolic points have been detected, therefore before the end of the initialization phase.

Preferably, the tracking phase is started as soon as at least two symbolic points have been detected. In an alternative embodiment of this particular form, results of the tracking phase, in particular information of the estimated position of the face having an influence on the position of the symbolic points in the image, for example the inclination of the face, are used to improve the search performance of other symbolic points of the image that have not yet been detected by the current initialization process.

The proposed method makes it possible to substantially accelerate the initialization process of a device for tracking a face and the quality of the initialization.

The method advantageously applies to any tracking device based on the reconstruction of a two-dimensional model of the image of a face to be continued.

Of course, the method described above could also be extrapolated to be applicable to a method and a device for tracking a face that would be based on a three-dimensional model.

Claims

A method of tracking a face (1) having an initialization step in which: a) predefined PS symbolic points of the face are identified (21) on an image of the face; b) an image of reduced size associated with each identified symbolic point, called a thumbnail, is extracted (25) from the image of the face; c) a two-dimensional pattern of the pursued face, said specific 2D model, is realized (28) from the positions of the detected symbolic points; characterized in that, in the course of the initialization step, only a part of the symbolic points PS is identified on a first image of a sequence: d) the identified symbolic points and the associated thumbnails extracted from said first image are memorized ; e) the undetected symbolic points are searched on successive images of the sequence; f) each symbolic point detected in an image of the sequence is, at least for the symbolic points not yet detected, stored as well as the thumbnail corresponding to said point extracted from said image of the sequence; g) the initialization step is considered complete (26) only when each symbolic point PS has been identified at least once on an image of the sequence.

2. The method of claim 1 wherein: h) all PS symbolic points are searched in each analyzed image of the sequence; i) when a symbolic point is detected in an image being analyzed, said current image, a score representative of the quality of the identification of the point as a symbolic point is attributed to said point of said image, and; j) when a symbolic point is identified in the current image and has also been identified in a previous image of the sequence: j1) its score in the current image is compared (24) with a stored score obtained in an earlier image for said point; j2) the memorized image corresponding to said symbolic point during detection of said point in the previous image:

- is retained if the score in the current image is worse than the memorized score of the previous image;

- is replaced by a thumbnail extracted from the current image if the score in the current image is better than the memorized score of the previous image.

The method according to claim 1 or claim 2 wherein the thumbnails of the different symbolic points are aligned with an image of the sequence when each PS is identified at least once in the different images of the sequence using a template. generic two-dimensional geometric face.

4. Method according to one of the preceding claims wherein at least six symbolic points of a face are sought, said six points corresponding to the outer and inner corners of the eyes and the left and right angles of the mouth of the face.

5. Method according to one of the preceding claims characterized in that a tracking phase is started as soon as at least two symbolic points have been identified and in that the results of the pursuit are used to search for other non-symbolic points. still detected.

6. Device for tracking a person's face using a two-dimensional face model having an initialization phase according to one of claims 1 to 5.

7. Device according to claim 6 wherein a face tracking phase is started before the end of the initialization phase.

8. Device according to claim 7 wherein the tracking phase is started when at least two PS symbolic points have been identified.