WO2011007204A1 - Control method of a graphic interface - Google Patents
Control method of a graphic interface Download PDFInfo
- Publication number
- WO2011007204A1 WO2011007204A1 PCT/IB2009/053563 IB2009053563W WO2011007204A1 WO 2011007204 A1 WO2011007204 A1 WO 2011007204A1 IB 2009053563 W IB2009053563 W IB 2009053563W WO 2011007204 A1 WO2011007204 A1 WO 2011007204A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- hand
- controls
- electronic device
- graphic interface
- dimensional space
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
A control method of a graphic interface displayed on a monitor by means of an electronic device connected thereto, the method comprising the steps of: - installing at least two shooting cameras of a predetermined three-dimensional space wherein a movement and/or a position of at least one hand can be monitored; - connecting an image processor to shooting cameras and acquiring in the processor a plurality of hand images in the three-dimensional space; - processing acquired images and detecting predetermined hand movements and/or positions in the three-dimensional space; - associating to the predetermined movements and/or positions respective graphic interface controls; - sending the plurality of controls from the image processor to a receiving module of the electronic device, in order to control the graphic interface.
Description
Title: Control method of a graphic interface.
DESCRIPTION Field of application
The present invention relates to a control method of a graphic interface displayed on a monitor by means of an electronic device connected thereto and comprising the steps of: installing at least two shooting cameras to monitor one hand; connecting an image processor to the shooting cameras and acquiring in the processor a plurality of hand images; - processing acquired images; associating to acquired images a control of the graphic interface and sending the control from the image processor to the graphic interface electronic control device.
Prior art Control methods of a graphic interface are known, detecting a hand movement, and particularly of a hand finger being streched out, and they associate this movement to a graphic interface control.
In particular, these methods provide for the monitor to be connected to an electronic device reproducing the graphic interface and a graphic pointer for pointing interface graphic elements.
More particularly, these control methods provide for at least two cameras to be installed, oriented towards said predetermined area wherein the hand movement is monitored, as well as an image processor to be connected to the cameras and a plurality of hand images to be acquired in the processor at predetermined time intervals.
The acquired images are processed in order to find hand movements, so that a first predetermined movement of the finger is associated to a
pointing control of the graphic interface and a second predetermined movement is associated to a selection or activation control of an interface graphic element.
For example, the first predetermined movement is a translation movement of the finger, streched out perpendicularly to an imaginary plane, for example a plane vertically expanding above the cameras, and the second predetermined movement consists in crossing this imaginary plane with the finger, keeping the finger perpendicular to the plane, essentially simulating pressing a button. Even more particularly, these methods provide for a control to be sent from the processor to the electronic device graphic interface in order to move the graphic pointer on the graphic element, if the first predetermined movement is associated to the pointing control, or in order to activate the graphic element, if the second predetermined movement is associated to the activation control.
Practically speaking, these methods allow the pointer to be moved as it commonly happens with a mouse but at a distance, by means of a simple movement and essentially without physical contact with control devices, as well as to activate the pointer, for example for selecting an icon, with the same simplicity, i.e. moving the finger with a simple gesture reminding pressing a button.
A method of the above-mentioned type, implemented by Fraunhofer and commonly indicated with "iPoint 3D", consists in a module comprising two integrated Firewire cameras being capable to capture the finger movement and to associate this movement to a remote control or instruction for the electronic device interface.
The above-mentioned control methods are advantangeous since they allow the graphic interface to be remotely controlled and without gripping any object, finding different fields of application such as video games, the remote control of devices in sterile or contaminated environments, the presentation of pieces of information on wide videoprojectors, as well as different applications in video art and
_
museum applications.
Nevertheless, these methods suffer from some limitations particularly associated to the kind of movement detected by the image processor; in fact, these methods only analyse an intersection point between the finger and the imaginary plane and not the overall hand movement in the three dimensions.
Moreover, these control methods are capable to associate to processed images only some simple hand movements, i.e. finger translation along the imaginary plane and crossing thereof and, consequently, to associate only a few controls of the graphic interface to these movements.
Finally, a further disadvantage is due to the fact that the image processor is directly connected to a serial interface or other communication system of the electronic device, in order to be able to send graphic interface controls thereto and, actually, replace a serial peripheral like the mouse, limiting the use of the remote control method, and without cabling between the image processor and the electronic device.
The technical problem underlying the present invention is to provide a control method of a graphic interface allowing a plurality of controls of the graphic interface to be remotely sent from an image processor to an electronic device, and allowing the user to make natural and intuitive movements, not bound to a reference plane, overcoming the drawbacks and limitations still affecting prior art control methods. Summary of the invention
The solution idea of the present invention is to detect a plurality of hand movements and/ or positions in a three-dimensional space, and to associate to movements and positions respective controls of a graphic interface, for example associating an activation control of a graphic element to a hand closure movement in the three-dimensional space, a deactivation control of the graphic element to a hand opening movement
_ _
or a movement control of the pointer on the interface to a horizontal or vertical movement of the hand, being it closed or open, these movements and/ or positions being processed in the three-dimensional space. Advantageously, according to this solution idea, the user can freely move the bare hand in the three-dimensional space since images are processed in the three dimensions x, y, z and he is able to communicate with the device by means of natural gestures, essentially simulating a grasping, gripping, or handling gesture of an interface graphic element. Advantageously, according to the method of the present invention, the hand is bare, i.e. it does not wear any glove and it does not grip any electronic device.
For example, an interface graphic element, i.e. an icon, can be selected by positioning the pointer on the graphic element and closing the hand, simulating a gripping or grasping gesture of the graphic element, and the graphic element movement is performed by moving the clenched-fist hand, and re-opening the hand palm in a position corresponding to a point of the monitor wherein the graphic element is wanted to be placed, simulating a release or depositing gesture of the graphic element.
In particular, a "drag&drop" control of the icon can be performed by positioning the graphic pointer on the icon, with the hand open, closing the hand to select the icon, moving the pointer and the icon with the clenched fist, and re-opening the hand to deselect the pointer and release the icon in another monitor position.
According to this solution idea, said technical problem is solved by a control method of a graphic interface displayed on a monitor by means of an electronic device connected thereto, the method comprising the steps of: - installing at least two shooting cameras of a predetermined three-dimensional space wherein a movement and/ or a position of at least one hand can be monitored; - connecting an image processor to the shooting cameras and acquiring in the processor a plurality of hand
- -
images; - processing the acquired images and detecting a plurality of predetermined hand movements and/ or positions; - associating to the predetermined movements and/ or positions a respective plurality of graphic interface controls; - sending said plurality of controls from the image processor to a receiving module of the electronic device, in order to control the graphic interface.
In particular, controls are inserted in data packets, which are sent from the image processor to the receiving module. Data packets comprise pointer movement and action controls, such as the XY positioning control and the actions for mouse buttons. Advantageously, the controls inserted in data packets are universally recognized by all operating systems and they comply with a HID (Human Interface Device) standard.
The method of the present invention also provides for the insertion in data packets, besides universally recognized controls, of a transport layer for a group of proprietary controls of the method of the present invention and/ or of innovative pointing and selection systems, such as a group of controls not being universally recognized by all operating systems in the current state of the art. Preferably, the group of additional controls is in the OSC format.
In particular, the method according to the present invention comprises the transport and delivery of data packets comprising additional controls to the receiving module, which sends these controls directly to the graphic interface, without notifying the operating system. Without limiting the scope of protection of the present invention, universally recognized controls are indicated hereafter as standard HID (Human Interface Device) controls, essentially identical to standard input peripheral controls. The receiving module sends standard HID controls to an operating system of the electronic device, which controls the graphic interface.
Always without limiting the scope of protection of the invention, not universally recognized controls are indicated hereafter as non standard
_ _
W or proprietary controls, and they are additional with respect to standard input peripheral controls. The receiving module sends additional controls directly to the graphic interface.
According to an aspect of the invention, the receiving module is an application being installed in the electronic device. The electronic device comprises a UDP port receiving UDP packets comprising standard HID controls and proprietary controls.
Standard HID controls in UDP packets comprise a x, y position which is sent to a DLL of the operating system, in order to move the pointer of the graphic interface.
According to another aspect of the present invention, the receiving module is a hardware device.
The hardware device comprises a first USB interface for connection to the electronic device and a second ethernet or wireless interface for connection to the image processor, receiving standard HID controls and proprietary controls. Standard HID controls generate an interrupt in the hardware device in order to move the pointer of the graphic interface in a x, y position, while OSC proprietary controls are directly delivered to the graphics management program. Advantageously, according to the method of the present invention, the graphic interface is remotely controlled and by means of natural hand movements within a three-dimensional space x, y, z.
Advantageously, detected movements are numerous and associated not only to standard controls of the HID type, for controlling the graphic interface, as provided for by common input peripherals, but also to a plurality of proprietary and predetermined controls allowing the interaction with the graphic interface to be sped up, by means of particularly intuitive gestures reminding object practical handling.
Advantageously, the electronic device hardware receiving module receives both standard HID controls and proprietary controls and it sends these controls, avoiding the operating system or the device
_ _
WO 2011 0 graphic interface to be modified or ad hoc programs to be installed in the electronic device.
Advantageously, the receiving module replaces the electronic device input peripherals and related functions, for example the mouse movement, the right or left button click, etc., and it extends these functions by means of said proprietary controls.
Further features and advantages according to the method of the present invention will be apparent from the following description given only by way of non limiting example. Brief description of the drawings
Figure 1 is a schematic view of a processor and electronic device connected to a monitor for implementing the method according to the present invention;
Figure 2 is a schematic and enlarged view of the pair of cameras of figure 1;
Figure 3 is a block diagram view of the image processing step performed according to the method of the present invention;
Figures 4a-4e are schematic views of hand image models detected for the interface control according to the method of the present invention; Figure 5 is a more detailed view of the model of figure 4c;
Figure 6 is a schematic view of a stereoscopic correlation step according to the method of the present invention;
Figure 7 is a schematic view of a device for controlling a graphic interface according to the present invention. .Detailed description
With reference to the attached drawings a control method of graphic elements displayed on a monitor by means of an electronic device 2
- -
connected thereto is described hereafter. The method comprises the following steps: installing at least two shooting cameras 3a, 3b of a predetermined three-dimensional space x, y, z wherein a movement and/ or a position of at least one hand can be monitored; connecting an image processor 1 to the shooting cameras 3a, 3b and acquiring in the processor 1 a plurality of hand images in the three- dimensional space x, y, z; processing acquired images and detecting predetermined hand movements and/or positions in the three-dimensional space x, y, z; associating to the predetermined hand movements and/ or positions a respective plurality of graphic interface controls; sending the plurality of controls from the image processor 1 to a receiving module 2a of the electronic device 2 in order to control the graphic interface.
In particular, the image processing step comprises a plurality of operations indicated hereafter which allows a three-dimensional analysis of the hand movement and/or position to be performed.
A first group of operations on shot images provides for a hand shape to be isolated within each frame separating the hand shape from the elements not associated thereto, in order to detect and measure the shape main features.
A second group of operations, also indicated as stereoscopic correlation, provides to measure the position differences of the hand being shot by the camera 3a and by the camera 3b respectively. In particular, through a trigonometric analysis of position differences, the method of the invention draws a hand XYZ position in the three- dimensional space x y z above cameras 3a and 3b. Advantageously, the stereoscopic comparison is possible due to hand shooting in a same
moment t according to two different angle shots of the three- dimensional space x, y, z.
A third group of operations provides to detect a hand position or shape by associating the image being processed as above to a predetermined hand shape or model stored in the image processor 1. In particular, the image processor 1 stores a plurality of hand shapes or models, "open hand with united fingers", "open hand with closed fingers", "clenched fist", "clenched fist with a stretched finger", etc.
A forth group of operations provides to measure the movement performed by the hand in the three-dimensional space x, y, z, in order to detect particular complex movements, associated to graphic interface controls. In particular, the forth step only analyses the hand movement in the space, while the third step detects the hand position, regardless of the fact that the hand is still or moving. The above-described steps are performed on a plurality of pairs of images acquired at predetermined intervals tl, t2, tn, in order to detect for example a hand horizontal translation movement, i.e. towards the x axis, or vertical translation movement, i.e. towards the y axis.
The horizontal or vertical translation movement is associated to a x, y movement control of the graphic pointer on the monitor. When the control associated to the x, y movement is detected, the image processor 1 sends to the electronic device 2 the control, for example, for moving the graphic pointer.
A line or laser light beam is projected in order to define the limit of the three-dimensional shooting space of cameras 3a, 3b, allowing the user to understand when the hand is positioned within such a three- dimensional space. In particular, the hand is lit up by the light beam when it goes beyond the light beam.
Shooting cameras 3a, 3b are connected to each other by means of an electrical collector comprising an input connected to the image processor 1, receiving a synchronization pulse from the processor 1,
and -an output connected to the processor itself sending acquired images. Preferably, the electrical collector is a Firewire bus.
In particular, the synchronization pulse feeds an electronic board comprising an opto-isolating barrier feeding infrared led diodes, which are preferably twenty, which sends at the same time the synchronization pulse to the two cameras 3a, 3b.
In an aspect of the present invention, the cameras 3a, 3b, the laser projector, the image processor 1 and the electronic board are integrated in a so-called sensing head 3, schematically represented in figure 2, preferably in a body 10 made of a box- type plate and comprising its own power supply, schematically represented in figure 2. Advantageously, the body 10 is a device integrating all hardware and software components in order to implement the interface control method according to the invention. Figure 7 is a schematic and more detailed view of body 10 components.
The sensing head is placed above the three-dimensional space A, for example fixed to the ceiling, or below this area, for example fixed to the floor or in a scuttle below the floor, equipped with a transparent cover, for example out of glass, through which the cameras 3a, 3b acquire the images and the projector emits the laser beam.
In particular, the two cameras 3a, 3b are equipped with a standard Firewire interface and the sensing head 3 comprises an illuminator 6, preferably of the infrared-light pulse type, in order to light up the three- dimensional space like a flash, and a triggering system synchronizing the infrared-light pulses emitted by the illuminator 6 when the two cameras 3a, 3b are operated, in order to acquire the respective frames, i.e. the images, in optimum lighting conditions.
The image processor 1 is connected to the electronic device 2, also indicated as slave pc or client, through a connection 5 based on a standard UDP network protocol. The connection is physically performed by means of network cables, being known per se, for example of the RJ45 type, or in the wireless mode, for example by means of a Wi-Fi
_ _
WO 2011 0 protocol. Advantageously, the UDP protocol is commonly used in the electrical connection between devices and it allows the information to be sent by exploiting computer standard equipments, i.e. the electronic device can be a common personal computer. The electronic device 2 comprises the receiving module which receives the outputs of the image processor 1 and it converts them into "Human Interface Device" inputs, i.e. direct controls for the mouse and/or controls of the slave electronic device 2.
In particular, the method of the present invention defines a control protocol of the graphic interface comprising standard HID (Human Interface Device) controls for input peripherals, for example for the mouse, and a plurality of proprietary controls, associated to the predetermined hand movements.
Standard HID controls the pointer movement along the x and y interface axes, as well as pointer activation controls, corresponding to a click of the mouse left button, this control comprising the generation of a
"Mouse Down" event followed by a "Mouse Up" event. These standard
HID controls are sent to the electronic device 2 operating system executing them as if they were sent directly by a standard peripheral like the mouse.
According to the method of the invention, some hand shapes, for example the "hand with open fingers" shape or the "clenched fist" shape are associated to standard HID controls for sending basic controls to the graphic interface. Advantageously, through the control method of the present invention, a plurality of proprietary controls can be sent from the image processor 1 to the electronic device graphic interface, detecting several predetermined hand gestures.
According to the present invention, the module is a hardware or software component programmed to recognize standard HID controls and proprietary controls, since the latter are not recognized by an operating system of the type commonly used by the electronic device 2, for example by Windows system.
_ -
WO 2011/007204
In a first embodiment of the present invention, the module is a hardware device comprising a USB interface and a network connector which are respectively connected to a USB socket of the electronic device 3 and to a network cable belonging to the electronic processor 1 itself. The network connector can be replaced by a different connection interface for the image processor 1, for example a wireless interface. The module is connected to the image processor 1 by means of a UDP protocol and it receives corresponding UDP packets.
In a second embodiment of the present invention, the electronic device 3 comprises an application receiving the output emitted by the image processor 1, i.e. standard HID controls and non standard proprietary controls for the graphic interface.
Advantageously, according to the method of the present invention, it is possible to remove a standard input peripheral of the electronic device 3, for example a common mouse connected though a USB port or Wi-Fi, to insert the software or hardware module in the electronic device 3 and to send to the electronic device 3 both standard input controls (HID) and non standard input controls (proprietary), through hand movements, without further hardware or software modifications of the electronic device 2.
The software module embodiment is briefly described hereafter, with reference to a Windows operating system installed in the electronic device 1, but without limiting the scope of application of the invention to devices comprising different operating systems. A UDP port is arranged in the electronic device 2, receiving information packets from the electronic processor, pointer x and y coordinates being extracted from those packets and sent as an argument to the function "MouseMove" of a system DLL, i.e. the Windows DLL user32.dll moving the mouse pointer. If UDP packets comprise an operation control, the corresponding action is accomplished. For example, if packets comprise a standard "click" control, the function "LButtonDown" is called in the DLL user32.dll, followed for example at an interval of 100 milliseconds by the function
W 2 - -
"LButtonUp". Similarly, for the standard control "DragStart", the function "LButtonDown" is called in the DLL user32.dll, and for the control "DragEnd" the function "LbuttonUp" is recalled.
In other words, the software module of the electronic device 1 performs calls to the operating system in order to send the controls associated to hand movements in the form of machine istructions. The hardware module is programmed likewise, in order to perform calls to the operating system and send in the form of machine istructions the controls associated to hand movements, but using, instead of a system DLL, an interrupt hardware and a processor incorporated in the hardware module.
The method of the present invention provides for the pointer movement on the graphic interface to be sped up, in order to make the association between the hand movement and the respective control, as well as the control sending to the device receiving module, essentially istantaneous.
In particular, the method performs a predictive analysis of the hand movement, i.e. it anticipates, based on the hand positions being acquired in a first part of the movement, a following position. This predictive analysis allows the pointer position to be calculated in advance with respect to the position really sent from the image processor to the electronic device, considerably increasing the interface control method speed and making the delay due to the image processing and to the communication between the image processor 1 and the electronic device 2 essentially unperceivable. In particular, the software module comprises a routine or program performed every time the image processor 1 sends to the electronic device 2 a hand position value x2 y2. The operation is based on the differential calculus of a known last position xl yl of the hand before sending a current position x2 y2 and considering the following three position values: the first position is calculated as a median point between the current position x2 y2 and the known last position xl yl; the second position is the current position x2 y2 itself; the third position, i.e. the predicted or anticipated position, is calculated by
W 2 - - extending by a predetermined measure a vector joining the intermediate position and the current position x2 y2.
Advantageously, the acquisition and processing of the images associated to the hand movement by means of cameras 3a, 3b is not limited to a vertical imaginary plane, towards which a hand finger is perpendicularly pointed, but it is extended to the whole three- dimensional space x, y, z wherein the hand is shot, and it refers to the whole hand movement, comprising for example a hand advancing in order to virtually grasp the graphic element, possible partial wrist rotations, the closure of hand fingers, etc.
With reference to figure 3, image acquisition and processing steps in the image processor 1 are described hereafter in greater detail. These steps are an integral part of the invention since the sequence and synergy thereof achieve the considerable advantage of increasing the speed and precision of the processor 1 and of rapidly sending controls to the electronic device 1.
The processing comprises:
1) removal in the pairs of raster images of areas, i.e. pixels, not belonging to the hand, comparison between resulting images and hand models stored in the image processor, detection of a hand shape, and calculation of a significant coordinate. In particular, each stored hand model is associated to a significant coordinate.
For example, without limiting the scope of protection of the present invention, the significant point of the "clenched-fist" hand is the fist barycenter, the significant point of the "open-finger" hand is the median point of the hand middle finger, the significant point of the "closed- finger" hand is the center of the hand palm. etc.
2) Trigonometric correlation of the significant coordinates of the two images and achievement of absolute values in the three-dimensional space x y z;
3) Hand morphological analysis in order to detect particular hand
positions and verify the resemblance with hand models stored in the processor 1;
4) Analysis of the vector covered by the hand in the three-dimensional space x, y, z and detection of the performed movement with models stored in the processor 1.
The main steps of the control method are described in greater detail in the following description.
A first step comprises the image acquisition in the three-dimensional space x, y, z wherein the hand is subsequently shot, comprising any other object existing in the three-dimensional space x, y, z. This image, also indicated as reference ground, is acquired when no movements occur within the three-dimensional space x, y, z for at least 20 seconds.
A second step comprises the acquisition of a pair of stereo scopically- correlated images of the hand positioned in the three-dimensional space x y z from two different angle shots and corresponding to the position of cameras 3a, 3b, which are synchronized with each other by means of the illuminator 6, as above indicated. This second step also comprises a mathematical subtraction of the acquired hand image and of the reference ground and the achievement of a resulting image, comprising a plurality of pixels having all the information being necessary for image processing.
Even more particularly, before subtracting the reference ground from an image comprising the hand, this image is scanned through the first- order derivate method, being known per se as "find edges", in order to detect the edges of the object being present. In the scanned image pixels are set at value 0 if they do not correspond to the edge of an object, or at value 1 if they correspond to a detected edge.
The value 0 or 1 of scanned image pixels is mathematically subtracted from the value of the corresponding pixels of the reference ground image. If the value resulting from the subtraction is higher than 0 it means that the pixel is present only in the hand image and thus it is
kept, otherwise it means that the pixel belongs to an edge of an object being present also in the reference ground and it is thus cancelled since it does not belong to the hand.
Even more particularly, the image being shot before scanning is in grey scale and it comprises a plurality of pixels having values comprised between 0 and 255. After scanning, the image is in binary format, i.e. comprising a plurality of pixels having values 0 or 1. The binary image comprises all the information being necessary for controlling the graphic interface, but it is still contaminated by video noise, errors and irrelevant objects.
A third step comprises the cleaning of areas being unrelated to the hand, comprising an erosion operation followed by a following morphological expansion. The aim of this step is removing the numerous single pixels generated by the video signal noise. Afterwards, non-removed areas are dimensionally analysed, counting the pixels with value 1 and adjacent pixels, i.e. determining the number of pixels composing an isoarea. According to the method of the invention, if the size of an isoarea is lower than a predetermined threshold size and parameterized with respect to the binary image size, the corresponding pixels are not considered as significant and they are set at 0; vice versa, if the size of an isoarea is higher than the threshold size, the related pixels are kept unchanged and used for the following analyses.
Non-removed areas are morphologically expanded in order to join, i.e. connect, close areas, which are interpreted as belonging to the same object. The morphologically-expanded areas are measured in order to extract a so-called elongation factor, representing a direction of the image major axis.
If the elongation factor indicates that an area has not an essentially elongate shape, for eample an arm or a finger, the area is removed, otherwise it is further processed in the following step.
A forth step provides for the images obtained at the end of the third step to be analysed, detecting shapes resembling a hand, through the
_ _
W 2 analysis of a hand-shape descriptive curve. In particular, the method measures the resulting area shape, for example to detect the hand fingertip, if expanded, or the clenched fist barycenter, i.e. considerable points wherefrom the output x, y coordinates of the hand detection step are extracted. The hand position determination step provides for some predetermined hand shape parameters to be measured. In particular, a first operation comprises the tracing of an external perimeter of the detected shape, obtained from a first pixel with value 1 and positioned low on the right in the image and following the adjacencies of such a pixel, i.e. searching, preferably in order, the connections positioned at the east, north, west and south, and detecting an outline as schematized in figure 5.
Orthogonal lines or vectors to the elongation factor are traced on this outline, and hand outlines are detected along these vectors. The first detected measure is the distance between the external points, afterwards an asymmetry factor between the points with respect to the elongation factor axis is detected, and the value of the possible intermediate points on the measurement axis is finally stored. These intermediate points indicate the possible presence of fingers far from the clenched fist or from the hand closed palm. The set of these values is compared with a vector of values prestored in the system, and if a sufficient similarity value is detected, an activation flag of the recognized morphology status is set. In order to avoid false positives, this step provides for the recognized morphology status to persist in at least 3 images out of 5, preferably corresponding to an analysis period of 0,15 seconds.
Advantageously, according to the present invention, the size measurement unit is parameterized and not absolute. The determination of such a unit is obtained through the weighted mean of the last 200 wrist measurements, corresponding to a linear section in the frame lower part before a wrist enlargement. All other measures are expressed as fractions of this floating value. Advantageously, this step is performed in order to compensate hand size differences in the image, cancelling the differences caused by big hands compared to small
hands, as well as by the hand distance from shooting cameras.
The above-mentioned first four steps are performed independently on the two images acquired by shooting cameras 3a, 3b, while the step being described afterwards, i.e. the fifth step, comprises an interpolation of the previous images, calculating the differences between the two images in order to determine the hand absolute position in the three-dimensional space x, y, x.
In particular, the method of the present invention comprises the trigonometric calculation based on the stereoscopic correlation of the results being processed till now. In other words, the above-described steps are performed separately on both images shot by shooting cameras 3a, 3b while the stereoscopic correlation step analyses the pair of images in order to determine the three-dimensional-space values through a differential trigonometric calculation. In particular, known variables for the solution of the differential trigonometry equation are schematically shown in figure 6 and they comprise an interpupillary distance between the two cameras, indicated with C, a convergency angle α of the cameras, an estimated operational distance, a CCD sensor size of cameras 3a, 3b, and a focal length of the optics being used.
The position X is calculated as a median position of the two considerable points with reference to the shooting frame center.
The result of the fifth step comprises a set of data representing the hand position with respect to the sensing head 3. Even more particularly, in order to obtain an angle θ for calculating the estimated operational distance, the following formula is performed:
(ττ/2)-(α+β), where α is a known angle, and β is an angle between an optical axis of the shooting camera and the position of the hand significant point. The angle β is calculated by dividing the arc tangent of the distance between
- - the significant point position and the center of the shooting camera sensor by the estimated operational distance. In order to obtain the distance Y between the conjunction line C of the two cameras and the hand position, an Euler's formula is performed Y = C * (sin θ/sin φ) * sin σ.
A sixth step is provided in order to translate the absolute coordinates x, y, z in the three-dimensional space into coordinates related to the pointer in the monitor.
Advantageously, this step allows to control a graphic interface of an electronic device 2, whose graphic resolution is not known.
In particular, the control method provides for a virtual monitor to be defined, originating in a point with coordinates x=0, y=0 and unitary height and width (h=l, 1=1). The value of the hand position x, y is normalized on the basis of the imaginary monitor height and width, and it is sent to the electronic device 2.
A seventh step performs a morphological analysis within the hand image for a precise detection of the hand position compared to a reference model. With reference to figures 4a-4e, the main hand reference models are schematically shown, "open hand", "closed hand", "forefinger outstretching forward", "gathered fingers with outstretched thumb", "side hand".
The aim of said detection is activating controls for the electronic device 2, for example the dragging control of an icon or rather the mouse right click. For example, in order to move the pointer, the user performs the following movement:
- he opens the hand with outstretched fingers as in figure 4a, so that the image processor 1 sends to the receiving module of the device 2 a dragging preparation control of an interface graphic element;
- he closes the fist as in figure 4b, so that the processor 1 sends to the device 2 an icon-dragging starting control;
- he re-opens the hand as in figure 4c, in order to release the icon.
The detection of the hand position comprises a plurality of linked measuring steps in order to determine the image most resembling a preset hand model. In particular, the shot image is overlapped by a measuring grid 5a being orthogonal to the image elongation factor 5b and comprising a plurality of vectors 55a, as shown in figure 5.
The most external points 5c of the hand image are determined along the vectors 55a of this grid 5a and a distance between these external points 5c is measured. The position values of the most external points 5c are transferred in a diagram having on the axis of abscissas x the numer of the vector 5c and on the axis of ordinates y a width value of the hand shape and an asimmetry value between the two hand sides. The result of this step creates a characteristic curve which is compared with a plurality of predetermined curves in the image processor 1, corresponding to the redefined hand models 4a-4e.
A control of hand position data is provided in a eighth step. In particular, hand position data being acquired in a predetermined time period, for example in the last 5 seconds, are used to calculate the movement performed by the hand along the three axes x, y, z, through a known calculation providing for the first derivative to be determined on the motion vector with respect to the initial motion. These values are stored as vectors indicating directions and they are compared with reference vectors stored in the processor 1. In particular, if the sum of the squares of the differences of the measured values is lower than a predetermined threshold, the performed movement is recognized and a flag is set, enabling sending a control associated to the detected gesture.
Substantially, the eighth step is used to compare the detected hand movement with a plurality of preset hand movements. Advantageously, according to the method of the present invention, controls are not only associated to the hand position, but also to complex hand movements, corresponding to the predetermined movements. The image processor 1 associates to the detected movement an interface control, for example the closing control of a window is associated to a hand movement
tracing a "X" in the three-dimensional space or the selection control of the text tool is associated to a circular movement. In particular, the method of the present invention provides to send a control to the receiving module only if the first derivative for x, y axes of a hand motion vector differs by less than a predetermined quantity from the first derivative of a predetermined motion vector. Advantageously, the method of the present invention detects natural hand movements but it avoids executing undesirable controls if this natural movement is not performed with enough precision. Finally, a last step is performed in order to make the pointer movement on the graphic interface flowing. In order to increase through a software of the processor 1 the movement softness, much suffering for camera 3a, 3b definition, a mean of the detected values is calculated. In fact, because of the pixel size of shooting cameras 3a, 3b, hand movement detection shows some asperities which, directly transferred on the graphic interface, would produce short jerks and short changes of direction even if the resulting overall movement would essentially be curvilinear. In order to remove these irregularities, this last step provides for a mean of position values to be performed according to a RMS or "RootMeanSquare" method. The applied correction weight is a function of a pointer motion quantity, since an overintegration slows the pointer reactivity down, while an inadequate integration makes the precision movement not very precise. If the pointer speed is higher than a reference speed, preferably calculated as a ratio between the half- screen width and a second (sec), the integration is not applied. Vice versa, if the pointer speed is lower than the reference speed, the integration is applied with a weight being inversely proportional to the speed itself.
As above described, the control associated to the movement is inserted in a packet of the UDP communication protocol and sent to the electronic device 3 receiving module, preferably according to the known standard communication format for music and OSC multimedia objects. In an embodiment of the present invention, an additional communication channel is also provided in the DMX format, a standard
being universally adopted in lighting and light-control systems for entertainment, through a serial port of the image processor 1. Advantageously, according to this embodiment of the invention, it is possible to connect lighting engineering devices communicating according to the detected gestures.
Preferably, the image processor 1 comprises a broadcaster UDP module sending the UDP packet on an ethernet network whereto the electronic device 2 is connected. Advantageously, due to the standard UDP protocol use, it is possible to implement at the same time several modules in different electronic devices 3, achieving interactions between these electronic devices 3, or it is possible to distribute controls in remote locations by exploiting the existing Ethernet infrastructures.
Advantageously, according to the method of the present invention, the graphic interface is remotely controlled and through hand natural movements within a three-dimensional space x, y, z.
Advantageously, detected movements are numerous and associated not only to standard controls of the HID type, for controlling the graphic interface as provided for by common input peripherals, but also to a plurality of predetermined proprietary controls allowing the interaction with the graphic interface to be sped up, by means of particularly intuitive gestures reminding object practical handling.
Advantageously, the electronic device receiving module receives both standard HID controls and proprietary controls and it sends these controls to the device operating system, avoiding this operating system or the graphic interface to be modified or ad hoc programs to be installed in the electronic device.
Advantageously, the receiving module replaces the electronic device input peripherals and related functions, for example the mouse movement, the right or left button click, etc., extending these functions by means of said proprietary controls.
Claims
1. A control method of a graphic interface displayed on a monitor by means of an electronic device (2) connected thereto, said method comprising the steps of: - installing at least two shooting cameras (3a, 3b) of a predetermined three-dimensional space (x, y, z) wherein a movement and/or a position of at least one hand is monitored; connecting an image processor (1) to said shooting cameras (3a, 3b) and acquiring in said processor (1) a plurality of hand images in the three-dimensional space (x, y, z); processing acquired images and detecting predetermined hand movements and/ or positions in the three-dimensional space (x, y, z); associating to said predetermined movements and/or positions a respective plurality of graphic interface controls; - sending said controls from said image processor (1) to a receiving module (2a) of said electronic device (2) in order to control the graphic interface.
2. A method according to claim 1, characterised in that said controls comprise standard HID (Human Interface Device) controls replacing standard input peripheral controls, and in that said receiving module (2a) sends said standard HID controls to an operating system of the electronic device (2).
3. A method according to claim 1, characterised in that said controls comprise non standard or proprietary controls, added to standard input peripheral controls, and in that said receiving module (2a) sends additional controls directly to said graphic interface.
4. A method according to one of the previous claims, characterised in that said receiving module (2a) is an application installed in said electronic device (2).
5. A method according to claim 4, characterised in that said electronic device comprises a UDP port receiving UDP packets comprising said standard HID controls and said proprietary controls.
6. A method according to claim 4, characterised in that said standard HID controls in said UDP packets comprise a x, y position and in that said x, y position is sent to a DLL of said operating system, in order to move the pointer of said graphic interface.
7. A method according to one of claims 1 to 3, characterised in that said receiving module (2a) is a hardware device.
8. A method according to claim 7, characterised in that said hardware device comprises a first USB interface for connection to the electronic device (2) and a second ethernet or wireless interface for connection to said image processor (1), receiving said standard HID controls and said proprietary controls.
9. A method according to claim 8, characerised in that said standard HID controls generate an interrupt in said hardware device in order to move the pointer of said graphic interface in a x, y position.
10. A method according to claim 1, characterised in that said image processing step comprises a predictive analysis of a movement of said hand in the three-dimensional space (x, y, z) comprising the steps of:
- detecting a current position x2, y2 of said hand at a time t2;
- calculating an intermediate hand position as a median point between said current position x2, y2 and a previous position xl, yl at a time tl;
- anticipating a following hand position, by extending by a predetermined measure a vector joining said intermediate position and said current position x2 y2.
11. A method according to claim 1, characterised in that said image processing step comprises the steps of removing in the pair of images acquired by said shooting cameras (3a, 3b), areas not belonging to said W - - hand, comparing resulting images with hand models stored in the image processor (1), detecting a hand shape between said models, and calculating a significant coordinate of said hand, corresponding to a x, y position.
12. A method according to claim 11, characterised in that said image processing step further comprises the step of trigonometrically correlating the significant coordinates of the images shot by said shooting cameras (3a, 3b) and obtaining absolute values of said hand in the three-dimensional space (x y z).
13. A method according to claim 12, characterised in that said image processing step further comprises the step of morphologically analysing said hand, detecting particular positions thereof.
14. A method according to claim 13, characterised in that said image processing step further comprises the step of analysing a vector covered by said hand in said three-dimensional space (x, y, z) and detecting a performed movement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/IB2009/053563 WO2011007204A1 (en) | 2009-07-16 | 2009-07-16 | Control method of a graphic interface |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/IB2009/053563 WO2011007204A1 (en) | 2009-07-16 | 2009-07-16 | Control method of a graphic interface |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2011007204A1 true WO2011007204A1 (en) | 2011-01-20 |
Family
ID=41800757
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/IB2009/053563 WO2011007204A1 (en) | 2009-07-16 | 2009-07-16 | Control method of a graphic interface |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2011007204A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102981623A (en) * | 2012-11-30 | 2013-03-20 | 深圳先进技术研究院 | Method and system for triggering input instruction |
| JP2014235743A (en) * | 2013-06-03 | 2014-12-15 | 株式会社リコー | Method and equipment for determining position of hand on the basis of depth image |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040189720A1 (en) * | 2003-03-25 | 2004-09-30 | Wilson Andrew D. | Architecture for controlling a computer using hand gestures |
| US20060202953A1 (en) * | 1997-08-22 | 2006-09-14 | Pryor Timothy R | Novel man machine interfaces and applications |
| US20080107303A1 (en) * | 2006-11-03 | 2008-05-08 | Samsung Electronics Co., Ltd. | Apparatus, method, and medium for tracking gesture |
| WO2009055148A1 (en) * | 2007-10-26 | 2009-04-30 | Honda Motor Co., Ltd. | Hand sign recognition using label assignment |
-
2009
- 2009-07-16 WO PCT/IB2009/053563 patent/WO2011007204A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060202953A1 (en) * | 1997-08-22 | 2006-09-14 | Pryor Timothy R | Novel man machine interfaces and applications |
| US20040189720A1 (en) * | 2003-03-25 | 2004-09-30 | Wilson Andrew D. | Architecture for controlling a computer using hand gestures |
| US20080107303A1 (en) * | 2006-11-03 | 2008-05-08 | Samsung Electronics Co., Ltd. | Apparatus, method, and medium for tracking gesture |
| WO2009055148A1 (en) * | 2007-10-26 | 2009-04-30 | Honda Motor Co., Ltd. | Hand sign recognition using label assignment |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102981623A (en) * | 2012-11-30 | 2013-03-20 | 深圳先进技术研究院 | Method and system for triggering input instruction |
| JP2014235743A (en) * | 2013-06-03 | 2014-12-15 | 株式会社リコー | Method and equipment for determining position of hand on the basis of depth image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101757080B1 (en) | Method and system for human-to-computer gesture based simultaneous interactions using singular points of interest on a hand | |
| KR20140060297A (en) | Method for detecting motion of input body and input device using same | |
| US9734393B2 (en) | Gesture-based control system | |
| Dorfmuller-Ulhaas et al. | Finger tracking for interaction in augmented environments | |
| KR101761050B1 (en) | Human-to-computer natural three-dimensional hand gesture based navigation method | |
| US9430698B2 (en) | Information input apparatus, information input method, and computer program | |
| US8666115B2 (en) | Computer vision gesture based control of a device | |
| EP3035164B1 (en) | Wearable sensor for tracking articulated body-parts | |
| EP1879129B1 (en) | Gesture recognition simultation system and method | |
| EP3283938B1 (en) | Gesture interface | |
| WO2013008236A1 (en) | System and method for computer vision based hand gesture identification | |
| US20080013826A1 (en) | Gesture recognition interface system | |
| KR102147430B1 (en) | virtual multi-touch interaction apparatus and method | |
| CN105992988A (en) | Method and device for detecting a touch between a first object and a second object | |
| US8416189B2 (en) | Manual human machine interface operation system and method thereof | |
| US20140053115A1 (en) | Computer vision gesture based control of a device | |
| TW201737023A (en) | Information processing system, information processing device, control method, and program | |
| KR20110097504A (en) | User motion perception method and apparatus | |
| KR101338958B1 (en) | system and method for moving virtual object tridimentionally in multi touchable terminal | |
| WO2011007204A1 (en) | Control method of a graphic interface | |
| CN110489026A (en) | A kind of handheld input device and its blanking control method and device for indicating icon | |
| WO2018079384A1 (en) | Information processing system, information processing apparatus, control method, and program | |
| CN108401452A (en) | The device and method that real goal detects and controls are executed using virtual reality head mounted display systems | |
| JP6632298B2 (en) | Information processing apparatus, information processing method and program | |
| Winkler et al. | Tangible mixed-reality desktop for digital media management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 09786919 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 09786919 Country of ref document: EP Kind code of ref document: A1 |