WO1998058346A1

WO1998058346A1 - Pointing device with optical fingerprint recognition and tracking capability

Info

Publication number: WO1998058346A1
Application number: PCT/CA1998/000598
Authority: WO
Inventors: Roland S. Walch
Original assignee: Walch Roland S
Priority date: 1997-06-18
Filing date: 1998-06-18
Publication date: 1998-12-23
Also published as: CA2208179A1; AU7903098A

Abstract

The present invention relates to a novel pointing device for use with a computer system. The pointing device can be in the form of a mouse, trackball or any other type of apparatus that can be employed by the user of the computer system to displace a cursor on the screen. The novel pointing device features a fingerprint identification system that enables the computer system to recognize the identity of the individual whose finger is being scanned. The advantage of providing the pointing device with a fingerprint recognition capability is the ease and simplicity with which user identification can be effected. In another aspect of the invention, a pointing device is provided that is capable of optically detecting and tracking significant papillary features of the user's finger. Thus, the movement of the user's finger can be mapped to the displacement of the cursor on the screen. The main advantage of this approach is to enable a much higher resolution in detecting the finger position by comparison to traditional methods.

Description

Title: Pointing device with optical fingerprint recognition and tracking capability

Field of the invention The present invention relates to a pointing device, such as a mouse or trackball featuring a finger print recognition ability so as to enable identification of the user whose finger print is being scanned. In another aspect, the invention provides a pointing device capable of optically tracking and detecting significant papillary features of the user's finger, this allowing to map the movement of the finger to the displacement of a cursor on a computer screen.

Background of the invention Pointing devices such as a mouse or trackball are extremely popular particularly with computer systems that operate under a graphical environment. A typical pointing device includes the ball that is received in a spherical cavity and that can be displaced by the user in the direction in which the cursor on the computer screen should be moved. Detection of the ball displacement can be effected in several ways. The simplest approach is to provide in the spherical cavity a pair of rollers that detect the ball displacement along two perpendicular axes. The displacement measured in terms of angular movement of each roller is digitized and transmitted to the computer system so the cursor can be displaced on the screen by an amount corresponding to the angular rotation of the roller. These so-called "Mechanical" pointing devices are relatively inexpensive yet accurate. The mechanical nature of their construction, however, makes them prone to failures as a result of component breakage or accumulation of dirt in the mechanism. To avoid or at least alleviate this difficulty, the industry has designed and commercialized in the past recent years pointing devices that no longer depend on mechanical parts to detect the movement that the user is desirous of imparting to the cursor on the screen. One type of these improved pointing devices utilizes an optical tracking system that detects markings made on the ball that the user moves. This system is expected to be highly reliable since no mechanical parts exist that can malfunction as a result of wear or accumulation of dirt. Another type of pointing device also developed with the objective of reducing failures resulting from breakage or contamination by dirt of mechanical components is the so called "Touch sensitive pad" . This type of pointing device is particularly popular for small computers whose physical dimensions are reduced to a point where it is no longer practical to install a pointing device that uses a rolling ball. The touch sensitive pads operate by sensing finger pressure and generating an output signal providing the coordinates at which that pressure has been detected. Although this approach is conceptually simple, it has a number of drawbacks . The most important drawback resides in the inability of detecting with sufficient level of precision the area at which the finger pressure has been applied. More specifically, when the user presses with his finger on the touch sensitive surface, the area on which pressure is sensed is relatively broad and generally corresponds to the surface of the fingertip. In order to provide a precise coordinate point to the computer system, assumptions must be made. One possibility is to calculate the geometric center of the contact surface finger/touch sensitive pad. A different approach is to observe the pressure gradient at the contact surface finger/touch sensitive pad and issue as coordinates the area or point at which the pressure is maximum. Although both methods provide acceptable results in detecting the location of the user's finger, the resolution with which such detection can be made is not optimum.

Another difficulty that is often associated with the use of computer systems is the necessity to accurately identify the user that is attempting to gain access to the system so as to either validate and authorize his access or deny access. The approach that has been mostly used up to now is the provision of password systems that may provide various levels of security control. A typical example, when the computer system is connected to a network resource, different password inquiries may be used to determine the access level that will be assigned to a given user. More specifically, the computer system may require at power-up a password that determines if the user will be authorized to access to the unit alone. Assuming that access has been gained, a different password may then be required for connection to the network resource, etc. Although this security control approach is relatively simple it has a number of drawbacks. First, identity of the user cannot be positively established. The only thing that the system can actually verify is whether the user has knowledge of the appropriate password. Thus, should the confidentiality of a password become compromised then access to the system can be gained even though the individual is not authorized to do so. Secondly, the process for entering a password is sometimes difficult to accept to users that desire computer systems that are as user- friendly as possible. Often, when gaining access to a network resource, the computer system will display on the screen a prompt that requires from the user to enter the appropriate password. The necessity to enter the string of appropriate characters through the keyboard is often considered by users to constitute an annoyance.

Objects and statement of the invention

One object of the invention is to provide a novel pointing device that has the capability to scan a finger print of a finger in contact with the pointing device.

Another object of the invention is to provide a pointing device that is capable of optically tracking a movement of a finger of a user.

As embodied and broadly described herein, the invention provides a pointing device for a computer system, said pointing device comprising: a finger scanning surface for through which an image of a finger can be transmitted; means responsive to a light excitation communicated through said finger scanning surface to generate an output signal representative of an image of at least a portion of a finger.

In a most preferred embodiment of the invention, the pointing device is provided with a finger scanning surface over which the user of the computer system applies his finger so that a scanned image of the finger print against the scanning surface can be made. Typically, the scanned image is an electronic representation of the papillary lines of the finger. This representation is digitally expressed in the form of a string of bytes. Depending upon the particular application each byte may represent a single image pixel or a series of pixels. If the application is such that a high level of recognition accuracy is necessary, the scanned image of the fingerprint should be as precise as possible. As a result, each pixel may be quantized according to various gray levels. In those applications, a single byte (is assuming that a byte is assigned eight bits) can then represent the value of a single pixel having up to 256 levels of gray. In contrast, for applications where a lesser degree of image resolution is required, then a less bits will be needed to represent the image of the fingerprint. At the extreme, when each pixel is assigned only two possible values, namely black or white, a single byte can be used to represent eight consecutive pixels.

When the image scanning process has been completed, the signal that contains the optical information needs to be processed in accordance with the intended application. For identification purposes, an analysis of the signal will be performed to extract important papillary features in order to construct an identification code that uniquely identifies the fingerprint. This identification code can then be compared with one or more codes stored in a database to determine if access should be authorized and in the affirmative up to what level. In this embodiment, the pointing device thus merely generates the signal that is an electronic representation of the fingerprint. Further processing required to extract the significant papillary features along with the comparison steps necessary to determine the level of access are performed by the computer system to which the pointing device is connected. In a variant, however, the pointing device may be designed to effect some signal processing steps, such as the extraction and identification of the significant papillary features. The output signal will thus be the identification code uniquely identifying the fingerprint. The comparison or validation steps are then performed by the computer system.

The location of the finger print scanning surface on the pointing device may vary with the intended application. When the pointing device is a mouse, the scanning surface may be located on one of the buttons that the user may click to initiate desired events in the computer system. Most preferably, the scanning surface is placed on the button that is normally actuated with the index finger. This position has been found particularly advantageous because it is natural for the user, when working with a computer system, to place the index finger at that location. Thus, the access authorization procedure is performed in a manner that is as transparent as possible. It suffices for the user to place his hand on the mouse so that the authentication process can be performed.

Other possibilities exist also. The scanning surface can be placed elsewhere than the button for the index finger. In this embodiment the user will normally be required to place his finger at that specified location so the authentication process can be effected.

In a different aspect, the invention provides a novel pointing device than that is capable of detecting and tracking significant papillary features of the finger of be user. Then the information generated as a result of this tracking activity can be used to map the finger movement to the displacement of a cursor on the computer screen. In a most preferred embodiment, the pointing device includes a scanning surface over which the user can displace his finger. The size of the scanning surface is selected such that it exceeds substantially the contact surface finger/scanning surface generated when the user presses his finger against the scanning surface. This allows the user to move his finger over the scanning surface so as to cause displacement of the cursor on the screen. Typically, the scanning surface is made of any suitable light transmitting material that enables an optical system placed under the scanning surface to obtain an image of the finger print. In a specific example, the scanning surface may be made of glass or of transparent synthetic material .

Under the scanning surface is placed an optical system that converts the incoming optical information into an electrical signal. More specifically, the optical system includes suitable focusing lenses and a camera optically coupled with the lenses. The camera generates, in response to the light excitation an output signal that contains the optical information. The electric signal is processed and digitized to obtain a complete representation of the image seen through the scanning surface. This image contains the fingerprint occupying the contact surface finger/scanning surface and the optical information found at the area adjoining the contact surface finger/scanning surface.

The optical information thus generated by the camera can be expressed as a series of bytes. As discussed under the previous aspect of this invention, each byte can represent either a single pixel of the image or a plurality of pixels, depending upon the level of image detail contained in the signal. For high-resolution applications, each pixel may be represented or quantized up to a predetermined number of gray levels. In contrast, for applications that require a lesser degree of image accuracy, a single bit may represent each pixel .

The array of bytes that represents a single frame or snapshot view through de scanning surface is then processed in order to track the movement of be finger that may have occurred by comparison to a previous frame. The frame processing operation involves a measurement or assessment of the area of the contact surface finger/scanning surface. By measuring the variation of that area value from one frame to another, the pointing device can determine if the user has effected a button clicking action. In a specific example if the system observes that the area of the contact surface increases and then rapidly decreases an assumption may be made that the user touched momentary the scanning surface . Depending upon the application, this may be construed to indicate a button- clicking event. The same methods can be used to detect successive contacts between the finger and the scanning surface to detect double clicking events.

For tracking the movement of the finger that may have occurred between successive frames, the system processes the array of bytes associated with every frame in order to identify a certain papillary feature of the finger. Once a certain feature has been identified and designate as a "Target" papillary feature, the signal processing is such that at every successive frame the location of that feature is determined. The location information expressed as coordinates in the X-Y plane is then transmitted to the computer system for moving the cursor to that relative position on the screen.

The advantage of this approach is the ability to detect with a high level of precision the location of the finger by effecting the tracking of a small papillary feature. Thus, there is no need to make any assumptions based on geometric centers in the contact surface finger/scanning surface or effect pressure gradient measurements.

As embodied and broadly described herein, the invention further provides a pointing device for a computer system, said pointing device comprising: a finger scanning surface; processing means responsive to an image of at least a selected portion of a finger transmitted through said finger scanning surface to generate an output signal mapping a position of the selected portion of the finger relative the finger scanning surface.

Brief description of the drawings

Figure 1 is a perspective view of a pointing device in the form of a mouse on which the present invention can be implemented ; Figure 2 is a schematical top plan view of a pointing device in the form of a mouse including a fingerprint scanning surface on one of the buttons;

Figure 3 is a schematical view of the optical system of the fingerprint scanning device depicted in figure 2;

Figure 4 is an enlarged view of the scanning surface illustrating how the optical information is organized in pixels and lines;

Figure 5 is a flow chart of a process for operating the scanning device shown at figures 2, 3 and 4;

Figure 6 is a schematical top plan view of a pointing device featuring a scanning surface through which an optical system can track the movement of user's finger;

Figure 7 illustrates the variation in the surface area when the user momentary touches the scanning surface to simulate a button clicking event;

Figure 8 is a flow chart of a process for operating the pointing device depicted at figure 6;

Figure 9 is a schematical view of a register in which is stored information relative to successive frames during the operation of the pointing device of figure 6 ;

Figure 10 is a schematical view illustrating the boundary of the contact surface finger/scanning surface, the rectangle in the boundary depicting the area of the finger print containing the significant papillary feature; and

Figure 11 is an enlarged view of the electronic image of the area containing the significant papillary feature.

Description of preferred embodiments

With reference to the annexed drawings, more particularly figures 1,2 and 3, the present invention provides a novel pointing device, such as a mouse 10 that features a finger print scanning capability. The structure of the mouse 10 is best shown at figures 2 and 3. The mouse comprises a body 12 having a shape suitable for fitting in the palm of the user's hand. As it is customary, the mouse 10 is provided with mechanical clicking buttons 14 and 16. A characterizing element of the mouse 10 is the provision on the button 14, that is normally operated by the index finger of the user, of a scanning surface 18 through which an electronic image of the user's fingerprint can be obtained.

The structure of the clicking button 14 and the structure of the underlying optical system is best shown at figure 3. The scanning surface 18 is in the form of a substantially flat plate made of light transmissive material as glass or any suitable synthetic material. In a possible variant, the scanning surface may be designed as a concave structure to better adapt to be anatomy of the finger. Under the scanning surface is provided an optical assembly 20 that comprises a suitable focusing lens 22 optically coupled to a camera device 24 that is designed to generate an electrical signal representative of the light excitation communicated to the camera through the lens 22. Different types of cameras can be used for this application. Most preferably, the camera 24 should be small enough to fit in the button 14. In a specific example a camera manufactured by the Eurotech company in Italy under the commercial designation "Micro integrated optical sensor system" has been found satisfactory. The camera 24 generates an output signal that is supplied to a signal processor whose task is to issue a digitized representation of the image seen through the scanning surface. The output signal generated by the signal processor 26 is essentially an array of bytes, each byte representing one or more image pixels. Figure 4 illustrates the pixels distribution profile designed to provide a complete representation of the optical information. The scanning surface is divided into a plurality of M lines, each line comprising N. individual pixels. Each byte in the array of bytes output by the signal processor 26 may represent one or more pixel values of the image, in dependence on the level of resolution. When the pixels are quantized up to 256 values of gray, a single byte (assuming that a byte contains eight bits) will represent the value of a single pixel. On the other hand, when a pixel is quantized to two values only (white or black) a byte will represent eight individual pixel values .

The level of image resolution that is required is usually selected in accordance with the intended application. When it is necessary to provide a fingerprint identification with a high level of precision, high resolution images will be necessary. For other applications where the level of fingerprint identification accuracy is not as high, lower image resolution can be used which allows to reduce processing time and the amount of data to be transmitted. Figure 5 illustrates a flow chart of the process for providing an identification of the fingerprint. This process assumes that the pointing device 10 is connected to a computer system to which the user is desirous of gaining access. The first step of the operation (step 28) is to initiate the fingerprint scan. This is accomplished essentially as described earlier. The user places his index finger on the scanning surface and an image of the fingerprint is taken. Usually, the fingerprint scan may require that the computer system generates on the screen a prompt to indicate to the user that fingerprint scanning is now required. When the image collection and its electronic representation has been completed as indicated at step 30, the array of bytes generated by the pointing device is processed by the computer system at step 32 in order to generate a suitable identifier that uniquely represents the papillary features of the fingerprint. Various techniques and algorithms have been developed in the past for generating such identifiers and it is not necessary for the purpose of this description to analyze those techniques in detail. Suffice it to say that for the purpose of fingerprint analysis anyone of the following techniques, well known to those skilled in the art may be used, either alone or in combination with one another:

A) matrix pixel matching;

B) lines count distribution;

C) skeletonning; D) line orientation ratio;

E) relative line angle;

F) relative line distance;

G) multiple image processing.

For additional information on this topic, the reader is also invited to refer to the text of the International application PCT/CA96/00234 whose contents are herein incorporated by reference .

Once the identifier has been generated at step 32, the computer system will effect a search in a database, but step 34 to determine if the identifier is accepted. If the identifier generated at step 22 is found in the list of authorized identifiers, the system will take the appropriate action at step 26, which is to enable access to the computer system, network etc.

In a different aspect, of the present invention provides a novel pointing device that is best shown at figures 6, 7, 10 and 11. The pointing device 34 has the capability of tracking a skin surface feature of the user's finger so that the movement of the finger can be mapped to the displacement of the cursor on the screen. The pointing device 34 includes a scanning surface 39 that is similar to the scanning surface 18 of the previous embodiment except that it is made much larger to allow the finger tip to be displaced on it in various directions. Under the scanning surface 39 is mounted an optical system that is identical to the optical system depicted in figure 3. In essence, the optical system (and any associated signal processor) generates an upper signal in the form of an array of bytes, each array providing a snapshot view of the image as seen through the scanning surface 39. By comparing the finger displacement from one snapshot (or frame) to another snapshot the movement of the finger can be tracked. In addition, the system can also recognize certain movements that are intended by the user to simulate button- clicking events.

As described in the previous aspect of the invention, the image of a single frame comprises a predetermined number of lines, each line including a predetermined number of individual pixels. In the example shown at figure 6, the scanning surface has dimensions to define an area constituted by X pixels and Y lines. The output signal generated by the optical system comprises an array of bytes, each byte containing the values of eight consecutive pixels (this mode of construction assumes that the system is set at minimum image resolution in which each pixel can take only two possible values, namely black or white)

The process of scanning a frame and the subsequent signal analysis operations are depicted in the flowchart shown at figure 8. At step 40, the scanning process is initiated for a single frame. At step 42, the array of bytes that provides an electronic representation of the image is constructed. At step 44, the image is processed to identify the boundaries of the contact surface finger/scanning surface 39. This can be effected simply by observing and detecting pixel agglomeration patterns in the frame. The contact surface with the fingertip will generate a relatively large agglomeration of pixels whose values will be predominantly black. In the absence of such pixel agglomeration, the system assumes that no finger is applied against the scanning surface 39. On the other hand, if the pixel agglomeration is identified the area of the contact surface is calculated. This parameter is used as an indication of the amount of finger pressure applied against the scanning surface 39. A simple method of calculating the area of the contact surface is to multiply the number of pixels contained in the boundary of the contact surface by the nominal pixel surface area. This provides a relatively accurate estimation of the area of the compact surface. Once this parameter is established at step 46, the process continues at step 48 where the area value is stored for further processing.

The next step of the operation (step 50) is to further analyze the pixel agglomeration corresponding to the contact surface in order to locate a certain papillary feature that the system will track, from one frame to another in order to precisely detect the degree of finger movement across the scanning surface 39. This feature is best illustrated at figures 10 and 11.

Figure 10 illustrates schematically the contact surface between the fingertip and the surface 39. A portion of that surface that is shown as a small rectangle has been identified as containing a feature of the skin surface that will be used as a target from one frame to another to effect the tracking operation. The size of this target surface is relatively small and typically contains a small number of pixels. The selection of this target surface is a matter of choice. Typically, the surface can be selected arbitrarily, as long as it contains a pixel pattern that is easily recognizable and sufficiently unique. In one specific application, a designation of the target surface occurs as soon as the presence of a finger is detected at the scanning surface 39. Once the target surface has been identified it is tracked, as described below, until the user lifts his finger from the scanning surface 39. When the finger is again brought in contact with the scanning surface 39, a new target surface is identified and tracked again, from one frame to another.

In the case when the finger has just touched the surface of the pointing device, the process of selection of the target surface is effected, generally as described above. After the selection process is completed, the pixel pattern that forms the target surface is kept in memory so it can be used as a mask for target surface tracking during subsequent frames . In addition, the coordinates of the target surface are determined in terms of X, Y measurements and stored for further analysis.

During the analysis of the next frame, the system will process the data to identify the pixel agglomeration corresponding to the contact surface. This value is stored in memory for further processing. In addition, the pixel pattern in the boundary of the contact surface is scanned to find the pixel pattern of the papillary feature mask corresponding to the target surface selected in the frame. Once the target surface is found, its coordinates are stored in memory. This process is repeated for the subsequent frames. The data structure resulting from the above analysis is depicted at figure 9. The data is organized in a table-like manner where each row corresponds to information pertaining to a given frame. For each frame, two parameters are stored, namely the surface area of the contact surface and the coordinates of the target surface. These elements of information can then be used to determine if the finger has been moved across the scanning surface 39 so as to move by a corresponding amount the cursor on the screen. For example, by comparing the coordinates of the target surface from one frame to another, a corresponding cursor movement can easily be attained. More sophisticated strategies can also be developed in which the rate of finger displacement can be measured to obtain acceleration effects. Also, by measuring the contact surface area, button-clicking events can be detected. This is best shown at figure 7 where the movement of the finger tip momentary touching the contact surface and than slightly reducing the pressure on be contact surface 39 is shown. Thus, by observing the variation in the contact surface from one frame to another or from a block of frames to another block of frames these movements can be detected.

The above description of the invention should not be interpreted in any limiting manner since variations and refinements of the preferred embodiment are possible without departing from the spirit of the invention. The scope of the invention is defined in the appended claims and their equivalents .

Claims

I claim:

1. A pointing device for a computer system, said pointing device comprising: - a finger scanning surface for through which an image of a finger can be transmitted; means responsive to a light excitation communicated through said finger scanning surface to generate an output signal representative of an image of at least a portion of a finger.

2. A pointing device as defined in claim 1, wherein said means responsive to light excitation includes a camera optically coupled to said finger scanning surface.

3. A pointing device as defined in claim 2, wherein said camera issues an analog signal representative of the papillary lines of the finger.

4. A pointing device as defined in claim 3, further comprising a signal processing device coupled to said camera and receiving said analog signal for generating an array of bytes, each byte representing at least one image pixel .

5. A pointing device as defined in claim 2, comprising a focusing lens between said camera and said finger- scanning surface.

6. A pointing device for a computer system, said pointing device comprising: a finger scanning surface; processing means responsive to an image of at least a selected portion of a finger transmitted through said finger scanning surface to generate an output signal mapping a position of the selected portion of the finger relative the finger scanning surface.

7. A pointing device for a computer system as defined in claim 6, wherein said finger scanning surface has an area substantially exceeding an area of a tip of a finger, thereby permitting the tip of the finger to move over said finger scanning surface.

8. A pointing device for a computer system as defined in claim 7, wherein the area of said finger scanning surface is of a size selected to allow the tip of the finger to be displaced over said finger scanning surface in two orthogonal directions.

9. A pointing device for a computer system as defined in claim 6, wherein said output signal constitutes an input suitable for use in a computer system to displace a cursor on computer screen.

10. A pointing device as defined in claim 6, wherein said processing means is capable of detecting and tracking the selected portion of a finger to continuously generate said output signal mapping the position of the selected portion of the finger relative the finger scanning surface.

11. A pointing device as defined in claim 10, wherein said processing means includes means responsive to a light excitation communicated through said finger scanning surface to generate an output signal representative of an image of at least a portion of a finger.

12. A pointing device as defined in claim 11, wherein said means responsive to light excitation includes a camera optically coupled to said finger scanning surface.

13. A pointing device as defined in claim 12, wherein said camera issues an analog signal representative of the papillary lines of the finger.

14. A pointing device as defined in claim 13, further comprising a signal processing device coupled to said camera and receiving said analog signal for generating an array of bytes, each byte representing at least one image pixel .