US7417681B2 - Multifunctional integrated image sensor and application to virtual interface technology - Google Patents

Multifunctional integrated image sensor and application to virtual interface technology Download PDF

Info

Publication number
US7417681B2
US7417681B2 US10/823,337 US82333704A US7417681B2 US 7417681 B2 US7417681 B2 US 7417681B2 US 82333704 A US82333704 A US 82333704A US 7417681 B2 US7417681 B2 US 7417681B2
Authority
US
United States
Prior art keywords
engagement surface
finger engagement
data entry
user
illuminator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US10/823,337
Other versions
US20040207732A1 (en
Inventor
Klony Lieberman
Yuval Sharon
Yaniv Maor
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lumio Home Services LLC
Original Assignee
VKB Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by VKB Inc filed Critical VKB Inc
Priority to US10/823,337 priority Critical patent/US7417681B2/en
Publication of US20040207732A1 publication Critical patent/US20040207732A1/en
Priority to US12/173,218 priority patent/US20080309620A1/en
Application granted granted Critical
Publication of US7417681B2 publication Critical patent/US7417681B2/en
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G06F3/0426Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking fingers with respect to a virtual keyboard projected or printed on the surface
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16Constructional details or arrangements
    • G06F1/1613Constructional details or arrangements for portable computers
    • G06F1/1626Constructional details or arrangements for portable computers with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16Constructional details or arrangements
    • G06F1/1613Constructional details or arrangements for portable computers
    • G06F1/163Wearable computers, e.g. on a belt
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16Constructional details or arrangements
    • G06F1/1613Constructional details or arrangements for portable computers
    • G06F1/1633Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
    • G06F1/1662Details related to the integrated keyboard
    • G06F1/1673Arrangements for projecting a virtual keyboard
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16Constructional details or arrangements
    • G06F1/1613Constructional details or arrangements for portable computers
    • G06F1/1633Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
    • G06F1/1684Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
    • G06F1/1686Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being an integrated camera

Definitions

  • the present invention relates to data input methods and apparatus generally.
  • an electronic camera including an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.
  • a portable telephone including telephone functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.
  • a digital personal assistant including at least one personal digital assistant functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.
  • the electronic imaging sensor includes a CMOS sensor.
  • the first imaging functionality incorporates a first lens arrangement upstream of the electronic imaging sensor.
  • the second imaging functionality incorporates a second lens arrangement upstream of the electronic imaging sensor.
  • the second imaging functionality incorporates an infrared light generator for defining a light beam, impingement of which resulting from the user hand activity produces a light pattern representative of the data. Additionally, the second imaging functionality also incorporates at least one infrared passing light filter associated with the electronic imaging sensor.
  • the second imaging functionality incorporates a processor receiving an output from the electronic imaging sensor for providing a data entry output. Additionally, the processor is incorporated together with the electronic imaging sensor on a single chip.
  • the first functionality is a color photography functionality. Additionally or alternatively, the second functionality is an infrared beam impingement sensing data entry functionality.
  • the user-operated imaging functionality selection switch is a mechanically responsive switch which selectably mechanically associates the first and second lens arrangements upstream of the electronic imaging sensor.
  • the user-operated imaging functionality selection switch is a mechanically responsive switch which selectably mechanically associates selected filters with the electronic imaging sensor.
  • the user-operated imaging functionality selection switch is an electronically responsive switch which selectably electronically determines responsiveness to selected outputs from predetermined portions of the electronic imaging sensor.
  • the electronic camera also includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor. Additionally or alternatively, the electronic camera also includes a data entry template projector employed by the second imaging functionality. Preferably, the electronic camera also includes an illuminator employed by the second imaging functionality. Additionally or alternatively, the electronic camera also includes a synchronized illumination power variation functionality.
  • the portable telephone also includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor. Additionally or alternatively, the portable telephone also includes a data entry template projector employed by the second imaging functionality. Preferably, the portable telephone also includes an illuminator employed by the second imaging functionality. Additionally or alternatively, the portable telephone also includes a synchronized illumination power variation functionality.
  • the digital personal assistant also includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor. Additionally or alternatively, the digital personal assistant also includes a data entry template projector employed by the second imaging functionality. Preferably, the digital personal assistant also includes an illuminator employed by the second imaging functionality. Additionally or alternatively, the digital personal assistant also includes a synchronized illumination power variation functionality.
  • an electronic camera including an electronic imaging sensor providing an output representing an imaged field and an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
  • a portable telephone including telephone functionality, an electronic imaging sensor providing an output representing an imaged field and an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
  • a digital personal assistant including at least one personal digital assistant functionality, an electronic imaging sensor providing an output representing an imaged field and an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
  • an electronic camera including an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor in a visible radiation imaging mode for taking a picture of a scene in the imaged field and a second imaging functionality employing the electronic imaging sensor in an infrared radiation sensing mode for data entry responsive to user hand activity.
  • a portable telephone including telephone functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor in a visible radiation imaging mode for taking a picture of a scene in the imaged field and a second imaging functionality employing the electronic imaging sensor in an infrared radiation sensing mode for data entry responsive to user hand activity.
  • a digital personal assistant including at least one personal digital assistant functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor in a visible radiation imaging mode for taking a picture of a scene in the imaged field and a second imaging functionality employing the electronic imaging sensor in an infrared radiation sensing mode for data entry responsive to user hand activity.
  • the electronic camera includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
  • the portable telephone includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
  • the digital personal assistant includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
  • output of the pixels underlying the red, green and blue passing elements are employed.
  • output of the pixels underlying the infrared passing elements is employed to prevent distortion of color response of the imaging sensor.
  • output of the pixels underlying the infrared passing elements are employed.
  • the electronic camera also includes a processor.
  • the electronic camera forms part of a telephone.
  • the electronic camera forms part of a personal digital assistant.
  • the electronic camera forms part of a wristwatch.
  • the portable telephone also includes a processor.
  • the digital personal assistant also includes a processor.
  • the processor includes the following functionality: as each pixel value is acquired, determining, using pixel coordinates, whether that pixel lies within a predefined keystroke region, acquiring pixel values for the pixel coordinates lying within the predefined keystroke region, adding or subtracting each of the pixel values to or from a pixel total maintained for each the keystroke region based on determining a pixel function of each pixel, comparing the pixel total for each the keystroke region with a current key actuation threshold, if the pixel total exceeds the key actuation threshold for a given keystroke region in a given frame and in the previous frame the pixel total did not exceed the key actuation threshold for that keystroke region, providing a key actuation output and if the pixel total does not exceed the key actuation threshold for a given keystroke region in a given frame and in the previous frame the pixel total did exceed the key actuation threshold for that keystroke region, providing a key deactuation output.
  • the determining whether that pixel lies within a predefined keystroke region is made by employing a pixel index table which indicates for each pixel, whether that pixel lies within a predetermined keystroke region and, if so, within which keystroke region it lies.
  • both of the determining steps employ the pixel index table.
  • the pixel total is maintained for each keystroke region in a keystroke region accumulator table.
  • the comparing employs a keystroke region threshold table.
  • the processor also includes the following functionality: once all of the pixels in a frame have been processed, determining an updated background level for a frame and determining a key actuation threshold for the keystroke region threshold table by subtracting the updated background level from a predetermined threshold level which is established for each keystroke region. Additionally, the pixel function includes adding the pixel values of a plurality of pixels in the keystroke region.
  • the processor is operative to determine the “center of gravity” of pixel values of pixels in the two-dimensional image sensor.
  • the processor includes the following functionality: as each pixel value is acquired, determining, using the pixel coordinates, whether that pixel lies within a predefined active region, acquiring pixel values for various pixel coordinates and determining the “center of gravity” of the pixel values.
  • determining the “center of gravity” is achieved by multiplying the pixel values by X and Y values representing the geographic position of each pixel, summing the results along mutually perpendicular axes X and Y, summing the total of the pixel values for all relevant pixels for the active region and dividing the summed results by the total of the pixel values to determine the X and Y coordinates of the “center of gravity”, which represents a desired engagement location.
  • the pixel values are thresholded prior to summing thereof.
  • a wristwatch including wristwatch functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.
  • a vehicle including a frame, a drive train and a chassis, including at least one lockable entry portal and portal unlocking functionality and an access control assembly for governing authorized access to the at least one lockable entry portal including at least one at least partially light transmissive user finger engagement surface accessible from outside the chassis, an illuminator operative to illuminate at least one user finger engagement plane adjacent the at least one user finger engagement surface, a two-dimensional imaging sensor viewing the at least one user finger engagement surface, from a location inwardly of the at least one finger engagement surface, for sensing light from the illuminator scattered by engagement of a user's finger with the at least one finger engagement surface, and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to the portal unlocking functionality.
  • access control apparatus for use with a vehicle including a chassis having at least one lockable entry portal and portal unlocking functionality and also including at least one at least partially light transmissive user finger engagement surface accessible from outside the chassis, the access control apparatus including an illuminator operative to illuminate the at least one user finger engagement surface, a two-dimensional imaging sensor viewing the at least one user finger engagement surface from a location inwardly of the at least one finger engagement surface for sensing light from the illuminator scattered by engagement of a user's finger with the at least one finger engagement surface and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to the portal unlocking functionality.
  • access control apparatus for use with an enclosure having at least one lockable entry portal and portal unlocking functionality and also including at least one at least partially light transmissive user finger engagement surface accessible from outside the enclosure, the access control apparatus including an illuminator operative to illuminate the at least one user finger engagement surface, a two-dimensional imaging sensor viewing the at least one user finger engagement surface from a location inwardly of the at least one finger engagement surface for sensing light from the illuminator scattered by engagement of a user's finger with the at least one finger engagement surface and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to the portal unlocking functionality.
  • data entry apparatus for use with at least one at least partially light transmissive user finger engagement surface, the data entry apparatus including an illuminator operative to illuminate the at least one user finger engagement surface, a two-dimensional imaging sensor viewing the at least one user finger engagement surface from a location inwardly of the at least one finger engagement surface for sensing light from the illuminator scattered by engagement of a user's finger with the at least one finger engagement surface and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input.
  • the illuminator is located at a location inwardly of the at least one engagement surface.
  • the at least one finger engagement surface includes a one-dimensional array of a plurality of finger engagement surfaces.
  • the at least one finger engagement surface includes a two-dimensional array of a plurality of finger engagement surfaces.
  • the vehicle also includes an illumination director cooperating with the illuminator for providing an illumination beam generally parallel to the at least one finger engagement surface.
  • the access control apparatus also includes an illumination director cooperating with the illuminator for providing an illumination beam generally parallel to the at least one finger engagement surface.
  • the data entry apparatus also includes an illumination director cooperating with the illuminator for providing an illumination beam generally parallel to the at least one finger engagement surface.
  • the illumination director includes a prism.
  • data entry apparatus for use with at least one engagement surface, the data entry apparatus including a light emitting data entry engagement element, which emits light only when it is in at least predetermined propinquity to the at least one engagement surface, a two-dimensional imaging sensor viewing the at least one engagement surface for sensing light emitted by the light emitting data entry engagement element and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input.
  • the light emitting data entry engagement element includes an elongated element having an LED at an end thereof.
  • the LED includes an IR emitting LED.
  • the data entry apparatus also includes a proximity switch.
  • data entry apparatus for use with at least one engagement surface, the data entry apparatus including a projector operative to illuminate the at least one engagement surface and to define thereon at least one touchpad region and at least one keyboard region, the touchpad region being defined by a zero'th order diffracted image, an illuminator operative to illuminate the at least one engagement surface, a two-dimensional imaging sensor viewing the at least one engagement surface from a location inwardly of the at least one engagement surface for sensing light from the illuminator scattered by engagement of a user's finger with the at least one engagement surface and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input.
  • a projector operative to illuminate the at least one engagement surface and to define thereon at least one touchpad region and at least one keyboard region, the touchpad region being defined by a zero'th order diffracted image
  • an illuminator operative to illuminate the at least one engagement surface
  • a two-dimensional imaging sensor viewing the at least one engagement surface from a location in
  • the processor includes the following functionality: as each pixel value is acquired, determining, using pixel coordinates, whether that pixel lies within a predefined keystroke region, acquiring pixel values for the pixel coordinates lying within the predefined keystroke region, adding or subtracting each of the pixel values to or from a pixel total maintained for each the keystroke region based on determining a pixel function of each pixel, comparing the pixel total for each the keystroke region with a current key actuation threshold, if the pixel total exceeds the key actuation threshold for a given keystroke region in a given frame and in the previous frame the pixel total did not exceed the key actuation threshold for that keystroke region, providing a key actuation output and if the pixel total does not exceed the key actuation threshold for a given keystroke region in a given frame and in the previous frame the pixel total did exceed the key actuation threshold for that keystroke region, providing a key deactuation output.
  • a data input device including an illuminator operative to illuminate at least one engagement plane by directing light along the at least one engagement plane, a two-dimensional imaging sensor, including an optical wedge element, viewing the at least one engagement plane from a location outside the at least one engagement plane for sensing light from the illuminator scattered by engagement of a data entry object with the at least one engagement plane and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to utilization circuitry.
  • the optical wedge element enables the imaging sensor to efficiently sense light from the engagement plane.
  • a data input device including an illuminator operative to illuminate at least one engagement plane by directing light along the at least one engagement plane, a two-dimensional imaging sensor, including a pair of oppositely oriented optical wedge elements, viewing the at least one engagement plane from a location outside the at least one engagement plane for sensing light from the illuminator scattered by engagement of a data entry object with the at least one engagement plane and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to utilization circuitry.
  • a data input device including an illuminator operative to illuminate at least one engagement plane by directing light along the at least one engagement plane and to illuminate a proper positioning confirmation location on an engagement surface, a two-dimensional imaging sensor, including, viewing the at least one engagement plane from a location outside the at least one engagement plane for sensing light from the illuminator scattered by engagement of a data entry object with the at least one engagement plane and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to utilization circuitry.
  • the imaging sensor senses the presence or absence of illumination at the proper positioning confirmation location. Additionally, the data entry apparatus is operative to unpower itself when the imaging sensor senses the absence of illumination at the proper positioning confirmation location.
  • gaming apparatus including at least one game piece, a game template projector, operative to project a game template onto a surface, an illuminator operative to illuminate the surface and an imaging sensor viewing the surface for sensing light from the illuminator reflected by at least one optically encoded reflector mounted on the at least one game piece.
  • each of the at least one reflector identifies one of the at least one game piece and is operative to indicate a location of the at least one game piece on the game template. Additionally or alternatively, the at least one reflector also identifies a spatial orientation of the game piece.
  • FIGS. 1A & 1B are simplified pictorial illustrations of an electronic camera constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation;
  • FIGS. 2A & 2B are simplified pictorial illustrations of a camera-equipped cellular telephone, constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation;
  • FIGS. 3A & 3B are simplified pictorial illustrations of a camera-equipped personal digital assistant, constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation;
  • FIGS. 4A & 4B are simplified diagrammatic illustrations of the operation of imaging functionality selection in accordance with a preferred embodiment of the present invention in two different modes of operation;
  • FIGS. 5A & 5B are simplified diagrammatic illustrations of the operation of imaging functionality selection in accordance with another preferred embodiment of the present invention in two different modes of operation;
  • FIG. 6 is a simplified diagrammatic illustration of a data entry object engagement location sensing subsystem useful in the embodiments of any of FIGS. 1A-5B ;
  • FIG. 7 is a simplified flow chart illustrating operation of the data entry object engagement location sensing subsystem of FIG. 6 in accordance with a preferred embodiment of the present invention
  • FIG. 8 is a simplified illustration of a preferred data structure employed in the operation of the data entry object engagement location sensing subsystem shown in FIG. 7 ;
  • FIG. 9 is a simplified pictorial illustration of outlines of typical keystroke regions as sensed by a two-dimensional image sensor viewing a keyboard, such as the keyboard seen in FIG. 6 ;
  • FIG. 10 is a simplified pictorial illustration of outlines of typical footprints of a typical light pattern occasioned by data entry object engagement with several keystroke regions, such as those shown in FIG. 9 ;
  • FIGS. 11A , 11 B and 11 C are simplified illustrations of three alternative methodologies for determining the function of the pixel within the keystroke region in which it lies as shown in FIG. 7 ;
  • FIGS. 12A , 12 B and 12 C are simplified illustrations of traces which are useful in understanding FIGS. 11A , 11 B and 11 C;
  • FIG. 13 is a simplified flow chart illustrating operation of a data entry object engagement location sensing subsystem employed in a tracking system and methodology constructed and operative in accordance with a preferred embodiment of the present invention
  • FIG. 14 is a simplified illustration showing synchronized illumination power variation functionality useful in accordance with a preferred embodiment of the present invention.
  • FIG. 15 is a simplified partially pictorial, partially diagrammatic illustration of data entry apparatus embodied in a vehicle access control system
  • FIG. 16 is a simplified partially pictorial, partially diagrammatic illustration of a variation in the data entry apparatus of FIG. 15 ;
  • FIG. 17 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention.
  • FIGS. 18A and 18B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention.
  • FIGS. 19A and 19B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with yet another preferred embodiment of the present invention.
  • FIGS. 20A and 20B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with still another preferred embodiment of the present invention.
  • FIG. 21 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with a further preferred embodiment of the present invention.
  • FIG. 22 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention.
  • FIG. 23 is a simplified pictorial illustration of gaming apparatus constructed and operative in accordance with a preferred embodiment of the present invention.
  • FIG. 24 is a simplified pictorial illustration of gaming apparatus constructed and operative in accordance with another preferred embodiment of the present invention.
  • FIGS. 1A & 1B are simplified pictorial illustrations of an electronic camera constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation.
  • an electronic camera 100 which may be of conventional construction other than as specifically described hereinbelow, such as a Nikon COOLPIX 5700, available from Nikon Corporation, Tokyo, Japan, is equipped with an electronic imaging sensor module 102 , which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • CMOS array for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • a user-operated imaging functionality selection switch 104 is preferably provided to enable a user to select operation in one of two possible imaging functionalities.
  • the electronic imaging sensor module 102 In a first imaging functionality, illustrated in FIG. 1A , the electronic imaging sensor module 102 is employed to take a picture of a scene in the imaged field.
  • the electronic imaging sensor module 102 In a second imaging functionality, illustrated in FIG. 1B , the electronic imaging sensor module 102 is employed for data entry responsive to user hand activity.
  • the second imaging functionality employs a data entry template projector 106 , which projects a data entry template, such as that designated by reference numeral 108 , onto a surface, such as a table surface 110 .
  • An illuminator 111 preferably is provided and is operative to illuminate a region adjacent the data entry template 108 .
  • the imaging sensor module 102 views the data entry template region for sensing light from the illuminator 111 .
  • the second imaging functionality also preferably incorporates processing functionality, an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • FIG. 1A when the electronic camera 100 is operative in the first imaging functionality, also termed a picture mode, a digital picture 112 is provided.
  • FIG. 1B when the electronic camera 100 is operative in the second imaging functionality, also termed a data mode, a stream of data 114 is provided.
  • FIGS. 2A & 2B are simplified pictorial illustrations of a camera-equipped cellular telephone, constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation.
  • a camera-equipped cellular telephone 200 which may be of conventional construction other than as specifically described hereinbelow, such as a Samsung SPH-I330, available from Samsung Electronics, Seoul, Korea, is equipped with an electronic imaging sensor module 202 , which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • CMOS array for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • a user-operated imaging functionality selection switch 204 is preferably provided to enable a user to select operation in one of two possible imaging functionalities.
  • the electronic imaging sensor module 202 is employed to take a picture of a scene in the imaged field.
  • the electronic imaging sensor module 202 is employed for data entry responsive to user hand activity.
  • the second imaging functionality employs a data entry template projector 206 , which projects a data entry template, such as that designated by reference numeral 208 , onto a surface, such as a table surface 210 .
  • An illuminator 211 preferably is provided and is operative to illuminate a region adjacent the data entry template 208 .
  • the imaging sensor module 202 views the data entry template region for sensing light from the illuminator 211 .
  • the second imaging functionality also preferably incorporates processing functionality an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • FIG. 2A when the camera-equipped cellular telephone 200 is operative in the first imaging functionality, also termed a picture mode, a digital picture 212 is provided.
  • FIG. 2B when the camera-equipped cellular telephone 200 is operative in the second imaging functionality, also termed a data mode, a stream of data 214 is provided.
  • FIGS. 3A & 3B are simplified pictorial illustrations of a camera-equipped personal digital assistant, constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation.
  • a camera-equipped personal digital assistant 300 which may be of conventional construction other than as specifically described hereinbelow, such as a Samsung SPH-I700, available from Samsung Electronics, Seoul, Korea, is equipped with an electronic imaging sensor module 302 , which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • CMOS array for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • a user-operated imaging functionality selection switch 304 is preferably provided to enable a user to select operation in one of two possible imaging functionalities.
  • the electronic imaging sensor module 302 In a first imaging functionality, illustrated in FIG. 3A , the electronic imaging sensor module 302 is employed to take a picture of a scene in the imaged field.
  • the electronic imaging sensor module 302 In a second imaging functionality, illustrated in FIG. 3B , the electronic imaging sensor module 302 is employed for data entry responsive to user hand activity.
  • the second imaging functionality employs a data entry template projector 306 , which projects a data entry template, such as that designated by reference numeral 308 , onto a surface, such as a table surface 310 .
  • An illuminator 311 preferably is provided and is operative to illuminate a region adjacent the data entry template 308 .
  • the imaging sensor module 302 views the data entry template region for sensing light from the illuminator 311 .
  • the second imaging functionality also preferably incorporates processing functionality an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • FIG. 3A when the electronic camera 300 is operative in the first imaging functionality, also termed a picture mode, a digital picture 312 is provided.
  • FIG. 3B when the electronic camera 300 is operative in the second imaging functionality, also termed a data mode, a stream of data 314 is provided.
  • FIGS. 4A & 4B are simplified diagrammatic illustrations of the operation of imaging functionality selection in accordance with a preferred embodiment of the present invention in two different modes of operation.
  • one implementation of the invention involves the use of a first and second selectably positionable optical assemblies which may be selectably placed in front of an electronic imaging sensor module 402 .
  • the sensor module 402 comprises a two-dimensional CMOS detector array 404 which is overlaid by an array of color filters 406 , which pass infrared radiation.
  • a lens 408 focuses light from an imaging field onto array 404 .
  • FIG. 4A shows a first selectably positionable optical assembly 410 , preferably comprising an IR blocking filter 412 , aligned with lens 408 .
  • the imaging functionality of FIG. 4A is suitable for picture taking, since it prevents IR radiation from distorting the color response of the sensor module 402 .
  • FIG. 4B shows a second selectably positionable optical assembly 420 , preferably comprising a visible radiation blocking filter 422 , an IR band pass filter 424 and a field lens 426 , aligned with lens 408 .
  • the imaging functionality of FIG. 4B is suitable for data input, since it allows only a relatively narrow band of IR radiation to reach the sensor module 402 from a relatively wide field of view.
  • FIGS. 5A & 5B are simplified diagrammatic illustrations of the operation of imaging functionality selection in accordance with another preferred embodiment of the present invention in two different modes of operation.
  • imaging functionality selection is effected electronically, without the need for mechanically movable optical elements.
  • this implementation of the invention involves the use of an electronic imaging sensor module 502 including a CMOS detector array 504 overlaid with a repeating array of pixelwise selective light transmissive filters 506 characterized in that the array includes red (R), green (G), blue (B) and infrared (IR) transmissive filters.
  • a lens 508 focuses light from an imaging field onto detector array 504 .
  • FIG. 5A shows the imaging functionality of the embodiment of FIGS. 5A and 5 B in a picture taking mode, wherein only the outputs of the pixels underlying the R, G and B filters are employed and the respective IR content of the outputs of each of the pixels underlying the R, G and B filters is subtracted therefrom.
  • This color processing prevents IR radiation from distorting the color response of the sensor module 502 .
  • FIG. 5B shows the imaging functionality of the embodiment of FIGS. 5A and 5B in a data entry mode, wherein only the outputs of the pixels underlying the IR filters are employed.
  • FIG. 6 is a simplified and generalized illustration of a projection keyboard system and methodology which is preferably incorporated into the embodiments of FIGS. 1A-5B in accordance with a preferred embodiment of the present invention.
  • a projection subsystem 550 preferably including a solid state light source (not shown) which illuminates a spatial light modulation element (not shown), defines an image 552 of a keyboard on a projection surface 554 , preferably an inert surface, such as a desktop.
  • An illumination subsystem 556 preferably including a solid state light source (not shown), directs light in a radially directed illumination pattern 558 , which preferably extends in a plane generally parallel to the projection surface 554 . It is appreciated that the radially directed illumination pattern 558 has a very narrow spread in the direction generally perpendicular to the projection surface 554 . It is further appreciated that the radially directed illumination pattern 558 is preferably located very close to the projection surface 554 .
  • Impingement of light in the radially directed illumination pattern 558 on a data entry object 560 causes light to be scattered or reflected therefrom. It is appreciated that the light is only scattered or reflected when the data entry object 560 is in close contact with the keyboard 552 defined on projection surface 554 .
  • a detection subsystem 562 receives light reflected or scattered from data entry object 560 .
  • the received light is mapped onto the imaging sensor with respect to a representation of the keyboard for associating the location of the data entry object 560 sensed by detection subsystem 562 with a key location 563 on the keyboard 552 .
  • the sensed location of data entry object 560 indicates which key of the keyboard 552 is being engaged.
  • FIG. 7 is a simplified flow chart illustrating operation of a data entry object engagement location sensing subsystem of FIG. 6 in accordance with a preferred embodiment of the present invention
  • FIG. 8 is a simplified illustration of a preferred data structure employed in the operation of the data entry object engagement location sensing subsystem shown in FIG. 7 .
  • FIG. 8 shows a simplified illustration of a preferred data structure employed in the operation of the data entry object engagement location sensing method described hereinbelow with respect to FIG. 7 .
  • the imaging sensor of a camera such as camera 100 ( FIG. 1A ) is typically comprised of a set of M ⁇ N pixels, wherein a particular group of pixels views a defined region of the engagement plane which preferably overlies the projection surface 110 ( FIG. 1A ).
  • a particular pixel group, located in the image plane of the camera 100 may receive scattered light from a data entry object 560 ( FIG. 6 ) engaging the key location 563 ( FIG. 6 ).
  • each of the M ⁇ N pixels in the image plane of the camera 100 may receive light from a corresponding region in the engagement plane in respect of a data entry object engagement therewith.
  • pixel values such as gray level values
  • pixel values are acquired for various pixel coordinates.
  • a determination is made, using the pixel coordinates, as to whether that pixel lies within a predefined keystroke region, such as keystroke regions 600 shown in FIG. 9 .
  • This determination is preferably made by employing a pixel index table 601 which indicates, for each pixel, whether that pixel lies within a predetermined keystroke region, such as keystroke regions 605 , 606 , 607 and 608 ( FIG. 9 ), and, if so, within which keystroke region it lies.
  • the function of the pixel within the keystroke region in which it lies is then determined, preferably by employing table 601 .
  • This function is typically additive or subtractive, but may alternatively have another function.
  • the pixel value is added to or subtracted from a pixel total maintained for each keystroke region in a keystroke region accumulator table 602 .
  • an updated background level is determined for the frame and a key actuation threshold is determined typically by subtracting the updated background level from a predetermined threshold level which is established for each keystroke region. This is preferably carried out by employing a keystroke region threshold table 604 .
  • the contents of the keystroke region accumulator table 602 for each keystroke region preferably are then compared with the current key actuation threshold. If the contents of the accumulator table 602 exceed the key actuation threshold for a given key actuation region in a given frame and in the previous frame the contents of the accumulator table 602 did not exceed the key actuation threshold, a key actuation output is provided.
  • FIG. 9 is a simplified pictorial illustration of outlines of typical keystroke regions 605 , 606 , 607 and 608 as sensed by a two-dimensional image sensor ( FIG. 6 ) viewing a keyboard, such as the keyboard 552 , seen in FIG. 6 .
  • FIG. 10 is a simplified pictorial illustration of outlines of typical footprints 609 , 610 , 611 and 612 of a typical light pattern occasioned by data entry object engagement corresponding to the keystroke regions 605 , 606 , 607 and 608 ( FIG. 9 ).
  • FIGS. 11A , 11 B and 11 C are simplified illustrations of three alternative methodologies for determining the function of the pixel within the keystroke region in which it lies as shown in FIG. 8 , and to FIGS. 12A , 12 B and 12 C, which are simplified illustrations of traces which are useful in understanding FIGS. 11A , 11 B and 11 C.
  • FIG. 1A there is shown arranged along a common arbitrary axis 618 a simplified keystroke region 620 corresponding to a given key and containing a plurality of pixels 622 .
  • a typical simplified footprint of a typical light pattern occasioned by data entry object engagement with the given key is indicated by reference numeral 624 .
  • FIG. 10 shows outlines of typical footprints 609 , 610 , 611 and 612 , corresponding to keystroke regions designated 605 , 606 , 607 and 608 in FIG. 9 .
  • a typical background signal pattern is indicated by reference numeral 634 .
  • Superimposition of the footprint 624 over the background signal pattern 634 is indicated at reference numeral 635 .
  • a one dimensionally selectable overlap of footprint 624 over keystroke region 620 is indicated at reference numeral 636 .
  • a one dimensionally selectable overlap of background signal pattern 634 over keystroke region 620 is indicated at reference numeral 637 .
  • a one dimensionally selectable overlap of superimposition 635 over keystroke region 620 is indicated at reference numeral 638 .
  • FIG. 12A illustrates a bias function 640 corresponding to a cross section of the keystroke region 620 taken along axis 618 , which bias function defines keystroke region 620 along axis 618 .
  • a signal function 644 corresponding to a cross section of the footprint 624 along axis 618 ; a background signal function 646 corresponding to a cross section of the background signal pattern 634 along axis 618 and a combination signal 648 corresponding to a cross section of the superimposition 635 .
  • FIG. 12A There is also shown in FIG. 12A a one dimensional convolution 650 corresponding to one dimensionally selectable overlap 636 ; a one dimensional convolution 652 corresponding to one dimensionally selectable overlap 637 and a one dimensional convolution 654 corresponding to one dimensionally selectable overlap 638 .
  • Convolution 650 is shown with a threshold 660 ; convolution 652 is shown with a threshold 662 and convolution 654 is shown with a threshold 664 .
  • FIG. 11B there is shown arranged along a common arbitrary axis 670 a simplified keystroke region 680 corresponding to a given key and containing a plurality of pixels 682 .
  • the simplified keystroke region 680 is here shown surrounded by a simplified keystroke region border 683 .
  • a typical simplified footprint of a typical light pattern occasioned by data entry object engagement with the given key is indicated by reference numeral 684 .
  • a typical background signal pattern is indicated by reference numeral 686 .
  • Superimposition of the footprint 684 over the background signal pattern 686 is indicated at reference numeral 688 .
  • a one dimensionally selectable overlap of footprint 684 over keystroke region 680 and border 683 is indicated at reference numeral 690 .
  • a one dimensionally selectable overlap of background signal pattern 686 over keystroke region 680 and border 683 is indicated at reference numeral 692 .
  • a one dimensionally selectable overlap of superimposition 688 over keystroke region 680 and border 683 is indicated at reference numeral 694 .
  • FIG. 12B illustrates a bias function 700 corresponding to a cross section of the keystroke region 680 and of the border 683 taken along axis 670 , which bias function defines keystroke region 680 along axis 670 .
  • border 683 is assigned a negative value relative to the value of the keystroke region 680 . This value assignment is appreciated to enhance the value of data entry object engagements to the extent that they lie within the keystroke region 680 and to decrease the value of such data entry object engagements to the extent that they extend over the border 683 .
  • a signal function 704 corresponding to a cross section of the footprint 684 along axis 670 ; a background signal function 706 corresponding to a cross section of the background signal pattern 686 along axis 670 and a combination signal 708 corresponding to a cross section of the superimposition 688 .
  • FIG. 12B There is also shown in FIG. 12B a one dimensional convolution 720 , corresponding to one dimensionally selectable overlap 690 ; a one dimensional convolution 722 , corresponding to one dimensionally selectable overlap 692 and a one dimensional convolution 724 corresponding to one dimensionally selectable overlap 694 .
  • Convolution 720 is shown with a threshold 726 ; convolution 722 is shown with a threshold 727 and convolution 724 is shown with a threshold 728 .
  • FIG. 11C there is shown arranged along a common arbitrary axis 730 a simplified keystroke region 740 corresponding to a given key and containing a plurality of pixels 741 .
  • the simplified keystroke region 740 is here shown surrounded by a simplified keystroke region inner border 742 and by a simplified keystroke region outer border 743 .
  • a typical simplified footprint of a typical light pattern occasioned by data entry object engagement with the given key is indicated by reference numeral 744 .
  • a typical background signal pattern is indicated by reference numeral 746 .
  • Superimposition of the footprint 744 over the background signal pattern 746 is indicated at reference numeral 748 .
  • a one dimensionally selectable overlap of footprint 744 over keystroke region 740 and borders 742 and 743 is indicated at reference numeral 750 .
  • a one dimensionally selectable overlap of background signal pattern 746 over keystroke region 740 and borders 742 and 743 is indicated at reference numeral 752 .
  • a one dimensionally selectable overlap of superimposition 748 over keystroke region 740 and borders 742 and 743 is indicated at reference numeral 754 .
  • FIG. 12C illustrates a bias function 760 corresponding to a cross section of the keystroke region 740 and of the borders 742 and 743 taken along axis 730 , which bias function defines keystroke region 740 along axis 730 .
  • border 742 is assigned a zero value
  • border 743 is assigned a negative value relative to the value of the keystroke region 740 . It is appreciated that these value assignments enhance the value of data entry object engagements that to the extent that they lie within the keystroke region 740 and to decrease the value of such data entry object engagements to the extent that they extend across the border 742 and at least into border 743 .
  • a signal function 764 corresponding to a cross section of the footprint 744 along axis 730 ; a background signal function 766 corresponding to a cross section of the background signal pattern 746 along axis 730 and a combination signal 768 corresponding to a cross section of the superimposition 748 .
  • FIG. 12C There is also shown in FIG. 12C a one dimensional convolution 780 , corresponding to one dimensionally selectable overlap 750 ; a one dimensional convolution 782 , corresponding to one dimensionally selectable overlap 752 and a one dimensional convolution 784 corresponding to one dimensionally selectable overlap 754 .
  • Convolution 780 is shown with a threshold 786 ; convolution 782 is shown with a threshold 787 and convolution 784 is shown with a threshold 788 .
  • FIG. 13 is a simplified flow chart illustrating operation of a data entry object engagement location sensing subsystem employed in a tracking system and methodology constructed and operative in accordance with a preferred embodiment of the present invention.
  • pixel values such as gray level values
  • pixel coordinates As seen in FIG. 13 , pixel values, such as gray level values, are acquired for various pixel coordinates. As each pixel value is acquired, a determination may be made, using the pixel coordinates, as to whether that pixel lies within a predefined active region. Typically, if the pixel does not lie within a predetermined active region, its value is ignored.
  • the pixel value for each pixel is preferably thresholded and typically all pixel values falling below a given threshold are ignored.
  • the remaining pixel values may be weighted by a selected weighting parameter.
  • the thresholded and weighted pixel values are multiplied respectively by X and Y values representing the coordinate position of each pixel and the results are summed along mutually perpendicular axes X and Y and stored in X and Y accumulators.
  • the total of the thresholded and weighted pixel values for all relevant pixels are also summed and stored in a data accumulator, for the entire active region.
  • the summed thresholded and weighted pixel values along the X and Y axes respectively are divided by the total of the thresholded and weighted pixel values for the entire active region to determine the X and Y coordinates of the “center of gravity”, which represents a desired engagement location.
  • FIG. 14 is a simplified illustration showing synchronized illumination power variation functionality useful in accordance with a preferred embodiment of the present invention.
  • the functionality illustrated in FIG. 14 is directed to modulating the amount of illumination provided for data entry object engagement detection. This modulation is desirable because the intensity of light impinging on a data entry object and thus scattered thereby, decreases with the distance between an illuminator 830 and a data entry object.
  • the amount of light impinging on a data entry object 832 is substantially greater than the amount of light impinging on a data entry object 834 .
  • the amount of scattered light impinging on a detector 836 decreases with the distance between the data entry object and the detector.
  • variable intensity drive electronics 840 which is coupled to both illuminator 830 and detector 836 and which causes the intensity of light produced by the illuminator 830 to vary, typically in a ramp fashion, in synchronization to the imaging field location of detector 836 .
  • the intensity is at a minimum.
  • an intermediate portion (B) of the engagement surface 554 is being imaged, typically at the middle of detector 836
  • the intensity is at an intermediate level
  • a far portion (C) of the engagement surface 554 is being imaged, typically at the bottom portion (C) of the detector 836 , the intensity is at a maximum.
  • Variable intensity drive electronics 840 operates preferably by providing a synchronization output 842 to detector 836 and a corresponding synchronization output 844 to illuminator 830 , causing the intensity level to increase in time in synchronization with the location of a scanned image region in detector 836 .
  • FIG. 15 is a simplified partially pictorial, partially diagrammatic illustration of data entry apparatus embodied in a vehicle access control system.
  • a vehicle 1500 typically comprising a frame, a drive train and a chassis including at least one lockable entry portal or door 1502 and portal unlocking functionality, includes an access control assembly designated generally by reference numeral 1504 .
  • the access control assembly 1504 preferably includes at least one at least partially light transmissive user finger engagement surface accessible from outside the chassis.
  • plural user finger engagement surfaces 1506 are defined on a car door window 1508 .
  • An illuminator 1510 preferably located interiorly of the exterior of the vehicle, such as an IR line generator, is operative to illuminate a finger engagement plane 1512 adjacent the user finger engagement surfaces 1506 .
  • a preferred IR line generator is described in applicant's Published PCT Application WO02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • a two-dimensional imaging sensor 1514 located interiorly of the exterior of the vehicle, views the finger engagement surfaces 1506 , from a location inwardly of the car door window 1508 , for sensing light originating from the illuminator 1510 , which is scattered by engagement of a user's finger with the finger engagement surfaces 1506 .
  • a preferred two-dimensional imaging sensor 1514 is described in applicant's Published PCT Application WO02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • a data entry processor 1516 receives an output from the two-dimensional imaging sensor 1514 and provides a data entry input to a conventional door unlocking functionality (not shown).
  • FIG. 16 is a simplified diagrammatic illustration of a variation in the data entry apparatus of FIG. 15 .
  • a vehicle 1600 typically comprising a frame, a drive train and a chassis including at least one lockable entry portal or door 1602 and portal unlocking functionality, includes an access control assembly designated generally by reference numeral 1604 .
  • the access control assembly 1604 preferably includes at least one at least partially light transmissive user finger engagement surface accessible from outside the chassis.
  • plural user finger engagement surfaces 1606 are defined on a car door window 1608 .
  • the plural user finger engagement surfaces 1606 may be arranged in a two-dimensional array.
  • An illuminator 1610 preferably located interiorly of the exterior of the vehicle, such as an IR line generator, is operative to illuminate a finger engagement plane 1612 adjacent the user finger engagement surfaces 1606 .
  • a preferred IR line generator is described in applicant's Published PCT Application WO02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • the finger engagement planes 1612 are parallel to the finger engagement surfaces 1606 , by virtue of provision of a prism 1613 disposed typically on the outside of the vehicle window 1608 along the light path from illuminator 1610 .
  • a two-dimensional imaging sensor 1614 located interiorly of the exterior of the vehicle, views the finger engagement surfaces 1606 , from a location inwardly of the car door window 1608 , for sensing light originating from the illuminator 1610 , which is scattered by engagement of a user's finger with the finger engagement surfaces 1606 .
  • a preferred two-dimensional imaging sensor 1614 is described in applicant's Published PCT Application WO02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • a data entry processor 1616 receives an output from the two-dimensional imaging sensor 1614 and provides a data entry input to a conventional door unlocking functionality (not shown).
  • portions of the apparatus of FIGS. 15 and 16 may find application in other types of access control environments and in various data entry environments.
  • FIG. 17 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention.
  • a light emitting data entry engagement element preferably in the form of a stylus 1700 , emits light when and only when it is in at least predetermined propinquity to an engagement surface 1702 .
  • a two-dimensional imaging sensor 1704 views the engagement surface 1702 for sensing light emitted by the stylus 1700 .
  • a data entry processor 1706 receives an output from the two-dimensional imaging sensor 1704 and provides a data entry input.
  • the stylus 1700 preferably comprises an IR emitting LED 1708 located in an IR transmissive tip 1710 thereof.
  • a proximity switch 1712 such as a contact switch, provides an output indication of the proximity of tip 1710 to engagement surface 1702 .
  • Circuitry 1714 receives an input from proximity switch 1712 and operates the LED 1708 accordingly, so as to provide an IR emission whose location indicates the location of stylus impingement with engagement surface 1702 .
  • FIGS. 18A and 18B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention.
  • the data entry apparatus comprises a projector 1800 operative to illuminate at least one engagement surface 1802 and to define thereon at least one touchpad region 1804 and at least one keyboard region 1806 , the touchpad region 1804 being defined by a zero'th order diffracted image provided by the projector 1800 .
  • the projector 1800 comprises a light source, such as a diode laser 1810 , which outputs via a collimating lens 1812 onto a first diffractive optical element 1814 .
  • Diffractive optical element 1814 diffracts the light impinging thereon into a primary keyboard image beam 1816 which impinges on at last one keyboard region 1806 and into a secondary ghost image beam 1818 , which is preferably blocked by a barrier 1819 .
  • Residual light passing through diffractive optical element 1814 defines a zero'th order beam 1820 , which impinges onto a second diffractive optical element 1822 .
  • Diffractive optical element 1822 diffracts the light impinging thereon from the zero'th order beam 1820 into a touchpad defining beam 1824 which impinges on at least one touchpad region 1804 .
  • an illuminator 1830 operative to illuminate the finger engagement surface.
  • a two-dimensional imaging sensor 1832 views the engagement surface for sensing light from the illuminator 1830 scattered by engagement of a user's finger with the finger engagement surface.
  • a data entry processor 1834 receives an output from the two-dimensional imaging sensor 1832 and provides a data entry input.
  • FIGS. 19A and 19B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with yet another preferred embodiment of the present invention.
  • the data entry apparatus comprises a projector 1900 operative to illuminate at least one finger engagement surface 1902 and to define thereon typically at least one keyboard region 1904 .
  • An illuminator 1906 is operative to illuminate the finger engagement surface 1902 .
  • An imaging sensor module 1908 including a two-dimensional imaging sensor 1910 , views the engagement surface for sensing light from the illuminator 1906 scattered by engagement of a user's finger with the finger engagement surface 1902 .
  • a particular feature of the imaging sensor module 1908 is that it includes an optical wedge element 1912 which provides a tilt, enabling a generally vertically disposed imaging sensor 1910 to efficiently sense light from a finger engagement surface 1902 extending generally perpendicularly thereto.
  • a data entry processor 1914 receives an output from the two-dimensional imaging sensor 1910 and provides a data entry input.
  • FIGS. 20A and 20B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with still another preferred embodiment of the present invention.
  • the data entry apparatus comprises a projector 2000 operative to illuminate at least one finger engagement surface 2002 and to define thereon typically at least one keyboard region 2004 .
  • An illuminator 2006 is operative to illuminate the finger engagement surface 2002 .
  • An imaging sensor module 2008 including a two-dimensional imaging sensor 2010 , views the engagement surface for sensing light from the illuminator 2006 scattered by engagement of a user's finger with the finger engagement surface 2002 .
  • a particular feature of the imaging sensor module 2008 is that it includes a pair of oppositely oriented optical wedge elements 2012 and 2014 , each of which provides a tilt to a portion of the light incoming to sensor 2010 , enabling efficient use of area of generally rectangular imaging sensor, as seen at reference number 2016 .
  • FIG. 21 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with yet another preferred embodiment of the present invention.
  • the data entry apparatus comprises a projector 2100 operative to illuminate at least one finger engagement surface 2102 and to define thereon typically at least one keyboard region 2104 .
  • An illuminator 2106 is operative to illuminate the finger engagement surface 2102 .
  • An imaging sensor module 2108 including a two-dimensional imaging sensor 2110 , views the engagement surface for sensing light from the illuminator 2106 scattered by engagement of a user's finger with the finger engagement surface 2102 .
  • a particular feature of the illuminator 2106 is that defines an auxiliary light path 2112 which impinges on a proper positioning confirmation location 2114 on the finger engagement surface 2102 adjacent and preferably nearer to the illuminator than the at least one keyboard region 2104 .
  • the imaging sensor module 2108 senses the presence of illumination at location 2114 and in the absence of sensed illumination at that location, typically indicating improper placement of the data entry apparatus, typically unpowers the data entry apparatus.
  • a data entry processor 2116 receives an output from the two-dimensional imaging sensor 2110 and provides a data entry input.
  • FIG. 22 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention in the form of a wristwatch.
  • a wristwatch 2200 which may be of conventional construction other than as specifically described hereinbelow, is equipped with an electronic imaging sensor module 2202 , which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • CMOS array for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • a user-operated imaging functionality selection switch 2204 is preferably provided to enable a user to select operation in a data entry functionality employing the electronic imaging sensor module 2202 for data entry responsive to user hand activity.
  • the data entry functionality preferably employs a data entry template projector 2206 , which projects a data entry template, such as that designated by reference numeral 2208 , onto a surface, such as a table surface 2210 .
  • An illuminator 2212 is operative to illuminate a region adjacent the data entry template.
  • the imaging sensor module 2202 views the data entry template region for sensing light from the illuminator 2212 scattered by engagement of a user's finger with the data entry template.
  • the data entry functionality also preferably incorporates processing functionality, an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • the data entry functionality produces a stream of data 2214 .
  • FIG. 23 is a simplified pictorial illustration of gaming apparatus constructed and operative in accordance with a preferred embodiment of the present invention.
  • gaming apparatus one example of which is a board game, such as chess
  • an electronic imaging sensor module 2302 which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • CMOS array for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 La Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
  • a user-operated imaging functionality selection switch 2304 is preferably provided to enable a user to select operation of one of plural gaming functionalities, such as different board games, employing the electronic imaging sensor module 2302 for gaming piece locating.
  • the gaming functionality preferably employs a gaming template projector 2306 , which projects a game template, such as a chess board, designated by reference numeral 2308 , onto a surface, such as a table surface 2310 .
  • a game template such as a chess board, designated by reference numeral 2308
  • An illuminator 2312 is operative to illuminate a region adjacent the game template 2308 .
  • the imaging sensor module 2302 views the game template region for sensing light from the illuminator 2312 reflected by optically encoded reflectors 2314 mounted on individual gaming pieces 2316 .
  • the reflectors 2314 preferably individually identify each gaming piece and indicate its location on the game template 2308 .
  • the gaming functionality also preferably incorporates processing functionality, an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference.
  • the gaming functionality produces a stream of data 2316 , which may be used for game management and scoring.
  • FIG. 24 is a simplified pictorial illustration of gaming apparatus constructed and operative in accordance with another preferred embodiment of the present invention.
  • FIG. 24 shows gaming pieces, such as dice 2402 , which bear variously differently coded reflectors 2404 on various surfaces thereof, thereby to enable not only the identity and location but also the spatial orientation of the gaming pieces to be sensed.

Abstract

An electronic camera including an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.

Description

REFERENCE TO CO-PENDING APPLICATIONS
This application is a divisional of application Ser. No. 10/481,932 filed on Jun. 15, 2005 now U.S. Pat. No. 7,037,661 which is International Application PCT/IL03/00538 filed on Jun. 26, 2003, which designated the U.S., claims the benefit thereof and incorporates the same by reference. The nonprovisional application designated above, namely application 10/481.932, filed Dec. 23, 2003, claims the benefit of U.S. Provisional application Ser. No. 60/392,376, entitled “Virtual Interface Implementation and Application”, filed Jun. 26, 2002 and U.S. Provisional application Ser. No. 60/438,327, entitled “Multifunctional Integrated Image Sensor and Application to Virtual Interface Technology”, filed Jan. 7, 2003.
BACKGROUND OF THE INVENTION
The following patents and publications are believed to represent the current state of the art:
U.S. Pat. Nos. 6,377,238; 6,281,238; 5,767,842; 5,457,550; 5,181,181 and 4,843,568.
Foreign Patent Documents: WO 00/21024; EP 0982 676 A1; DE 298 02 435 U1; WO 01/93182 and WO 02/054169.
SUMMARY OF THE INVENTION
The present invention relates to data input methods and apparatus generally.
There is thus provided in accordance with a preferred embodiment of the present invention an electronic camera including an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.
There is also provided in accordance with another preferred embodiment of the present invention a portable telephone including telephone functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.
There is further provided in accordance with yet another preferred embodiment of the present invention a digital personal assistant including at least one personal digital assistant functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.
In accordance with another preferred embodiment of the present invention the electronic imaging sensor includes a CMOS sensor.
Preferably, the first imaging functionality incorporates a first lens arrangement upstream of the electronic imaging sensor. Additionally, the second imaging functionality incorporates a second lens arrangement upstream of the electronic imaging sensor.
In accordance with yet another preferred embodiment of the present invention the second imaging functionality incorporates an infrared light generator for defining a light beam, impingement of which resulting from the user hand activity produces a light pattern representative of the data. Additionally, the second imaging functionality also incorporates at least one infrared passing light filter associated with the electronic imaging sensor.
In accordance with still another preferred embodiment of the present invention the second imaging functionality incorporates a processor receiving an output from the electronic imaging sensor for providing a data entry output. Additionally, the processor is incorporated together with the electronic imaging sensor on a single chip.
In accordance with another preferred embodiment of the present invention the first functionality is a color photography functionality. Additionally or alternatively, the second functionality is an infrared beam impingement sensing data entry functionality.
In accordance with yet another preferred embodiment of the present the user-operated imaging functionality selection switch is a mechanically responsive switch which selectably mechanically associates the first and second lens arrangements upstream of the electronic imaging sensor. Alternatively or additionally, the user-operated imaging functionality selection switch is a mechanically responsive switch which selectably mechanically associates selected filters with the electronic imaging sensor.
In accordance with still another preferred embodiment of the present invention the user-operated imaging functionality selection switch is an electronically responsive switch which selectably electronically determines responsiveness to selected outputs from predetermined portions of the electronic imaging sensor.
In accordance with another preferred embodiment of the present invention the electronic camera also includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor. Additionally or alternatively, the electronic camera also includes a data entry template projector employed by the second imaging functionality. Preferably, the electronic camera also includes an illuminator employed by the second imaging functionality. Additionally or alternatively, the electronic camera also includes a synchronized illumination power variation functionality.
In accordance with another preferred embodiment of the present invention the portable telephone also includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor. Additionally or alternatively, the portable telephone also includes a data entry template projector employed by the second imaging functionality. Preferably, the portable telephone also includes an illuminator employed by the second imaging functionality. Additionally or alternatively, the portable telephone also includes a synchronized illumination power variation functionality.
In accordance with another preferred embodiment of the present invention the digital personal assistant also includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor. Additionally or alternatively, the digital personal assistant also includes a data entry template projector employed by the second imaging functionality. Preferably, the digital personal assistant also includes an illuminator employed by the second imaging functionality. Additionally or alternatively, the digital personal assistant also includes a synchronized illumination power variation functionality.
There is also provided in accordance with another preferred embodiment of the present invention an electronic camera including an electronic imaging sensor providing an output representing an imaged field and an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
There is further provided in accordance with still another preferred embodiment of the present invention a portable telephone including telephone functionality, an electronic imaging sensor providing an output representing an imaged field and an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
There is also provided in accordance with another preferred embodiment of the present invention a digital personal assistant including at least one personal digital assistant functionality, an electronic imaging sensor providing an output representing an imaged field and an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
There is even further provided in accordance with yet another preferred embodiment of the present invention an electronic camera including an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor in a visible radiation imaging mode for taking a picture of a scene in the imaged field and a second imaging functionality employing the electronic imaging sensor in an infrared radiation sensing mode for data entry responsive to user hand activity.
There is also provided in accordance with another preferred embodiment of the present invention a portable telephone including telephone functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor in a visible radiation imaging mode for taking a picture of a scene in the imaged field and a second imaging functionality employing the electronic imaging sensor in an infrared radiation sensing mode for data entry responsive to user hand activity.
There is further provided in accordance with yet another preferred embodiment of the present invention a digital personal assistant including at least one personal digital assistant functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor in a visible radiation imaging mode for taking a picture of a scene in the imaged field and a second imaging functionality employing the electronic imaging sensor in an infrared radiation sensing mode for data entry responsive to user hand activity.
In accordance with another preferred embodiment of the present invention the electronic camera includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
In accordance with yet another preferred embodiment of the present invention the portable telephone includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
In accordance with yet another preferred embodiment of the present invention the digital personal assistant includes an array of red, green, blue and infrared passing pixelwise elements arranged over the electronic imaging sensor.
Preferably, in the first imaging functionality output of the pixels underlying the red, green and blue passing elements are employed. Additionally, output of the pixels underlying the infrared passing elements is employed to prevent distortion of color response of the imaging sensor. Additionally, in the second imaging functionality output of the pixels underlying the infrared passing elements are employed.
In accordance with yet another preferred embodiment of the present invention the electronic camera also includes a processor. In accordance with still another preferred embodiment of the present invention the electronic camera forms part of a telephone. Alternatively, the electronic camera forms part of a personal digital assistant. Alternatively, the electronic camera forms part of a wristwatch.
In accordance with another preferred embodiment of the present invention the portable telephone also includes a processor.
In accordance with yet another preferred embodiment of the present invention the digital personal assistant also includes a processor.
In accordance with still another preferred embodiment of the present invention the processor includes the following functionality: as each pixel value is acquired, determining, using pixel coordinates, whether that pixel lies within a predefined keystroke region, acquiring pixel values for the pixel coordinates lying within the predefined keystroke region, adding or subtracting each of the pixel values to or from a pixel total maintained for each the keystroke region based on determining a pixel function of each pixel, comparing the pixel total for each the keystroke region with a current key actuation threshold, if the pixel total exceeds the key actuation threshold for a given keystroke region in a given frame and in the previous frame the pixel total did not exceed the key actuation threshold for that keystroke region, providing a key actuation output and if the pixel total does not exceed the key actuation threshold for a given keystroke region in a given frame and in the previous frame the pixel total did exceed the key actuation threshold for that keystroke region, providing a key deactuation output.
In accordance with another preferred embodiment of the present invention the determining whether that pixel lies within a predefined keystroke region is made by employing a pixel index table which indicates for each pixel, whether that pixel lies within a predetermined keystroke region and, if so, within which keystroke region it lies. Alternatively, both of the determining steps employ the pixel index table.
In accordance with yet another preferred embodiment of the present invention the pixel total is maintained for each keystroke region in a keystroke region accumulator table.
In accordance with still another preferred embodiment of the present invention the comparing employs a keystroke region threshold table.
In accordance with another preferred embodiment of the present invention the processor also includes the following functionality: once all of the pixels in a frame have been processed, determining an updated background level for a frame and determining a key actuation threshold for the keystroke region threshold table by subtracting the updated background level from a predetermined threshold level which is established for each keystroke region. Additionally, the pixel function includes adding the pixel values of a plurality of pixels in the keystroke region.
In accordance with yet another preferred embodiment of the present invention the processor is operative to determine the “center of gravity” of pixel values of pixels in the two-dimensional image sensor. Preferably, the processor includes the following functionality: as each pixel value is acquired, determining, using the pixel coordinates, whether that pixel lies within a predefined active region, acquiring pixel values for various pixel coordinates and determining the “center of gravity” of the pixel values.
In accordance with still another preferred embodiment of the present invention determining the “center of gravity” is achieved by multiplying the pixel values by X and Y values representing the geographic position of each pixel, summing the results along mutually perpendicular axes X and Y, summing the total of the pixel values for all relevant pixels for the active region and dividing the summed results by the total of the pixel values to determine the X and Y coordinates of the “center of gravity”, which represents a desired engagement location.
Preferably, the pixel values are thresholded prior to summing thereof.
There is yet further provided in accordance with another preferred embodiment of the present invention a wristwatch including wristwatch functionality, an electronic imaging sensor providing an output representing an imaged field, a first imaging functionality employing the electronic imaging sensor for taking a picture of a scene in the imaged field, a second imaging functionality employing the electronic imaging sensor for data entry responsive to user hand activity and a user-operated imaging functionality selection switch operative to enable a user to select operation in one of the first and second imaging functionalities.
There is also provided in accordance with another preferred embodiment of the present invention a vehicle including a frame, a drive train and a chassis, including at least one lockable entry portal and portal unlocking functionality and an access control assembly for governing authorized access to the at least one lockable entry portal including at least one at least partially light transmissive user finger engagement surface accessible from outside the chassis, an illuminator operative to illuminate at least one user finger engagement plane adjacent the at least one user finger engagement surface, a two-dimensional imaging sensor viewing the at least one user finger engagement surface, from a location inwardly of the at least one finger engagement surface, for sensing light from the illuminator scattered by engagement of a user's finger with the at least one finger engagement surface, and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to the portal unlocking functionality.
There is further provided in accordance with yet another preferred embodiment of the present invention access control apparatus for use with a vehicle including a chassis having at least one lockable entry portal and portal unlocking functionality and also including at least one at least partially light transmissive user finger engagement surface accessible from outside the chassis, the access control apparatus including an illuminator operative to illuminate the at least one user finger engagement surface, a two-dimensional imaging sensor viewing the at least one user finger engagement surface from a location inwardly of the at least one finger engagement surface for sensing light from the illuminator scattered by engagement of a user's finger with the at least one finger engagement surface and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to the portal unlocking functionality.
There is further provided in accordance with yet another preferred embodiment of the present invention access control apparatus for use with an enclosure having at least one lockable entry portal and portal unlocking functionality and also including at least one at least partially light transmissive user finger engagement surface accessible from outside the enclosure, the access control apparatus including an illuminator operative to illuminate the at least one user finger engagement surface, a two-dimensional imaging sensor viewing the at least one user finger engagement surface from a location inwardly of the at least one finger engagement surface for sensing light from the illuminator scattered by engagement of a user's finger with the at least one finger engagement surface and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to the portal unlocking functionality.
There is yet further provided in accordance with still another preferred embodiment of the present invention data entry apparatus for use with at least one at least partially light transmissive user finger engagement surface, the data entry apparatus including an illuminator operative to illuminate the at least one user finger engagement surface, a two-dimensional imaging sensor viewing the at least one user finger engagement surface from a location inwardly of the at least one finger engagement surface for sensing light from the illuminator scattered by engagement of a user's finger with the at least one finger engagement surface and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input.
Preferably, the illuminator is located at a location inwardly of the at least one engagement surface. Additionally or alternatively, the at least one finger engagement surface includes a one-dimensional array of a plurality of finger engagement surfaces. Alternatively, the at least one finger engagement surface includes a two-dimensional array of a plurality of finger engagement surfaces.
In accordance with another preferred embodiment of the present invention the vehicle also includes an illumination director cooperating with the illuminator for providing an illumination beam generally parallel to the at least one finger engagement surface.
In accordance with yet another preferred embodiment of the present invention the access control apparatus also includes an illumination director cooperating with the illuminator for providing an illumination beam generally parallel to the at least one finger engagement surface.
In accordance with still another preferred embodiment of the present invention the data entry apparatus also includes an illumination director cooperating with the illuminator for providing an illumination beam generally parallel to the at least one finger engagement surface.
Preferably, the illumination director includes a prism.
There is also provided in accordance with another preferred embodiment of the present invention data entry apparatus for use with at least one engagement surface, the data entry apparatus including a light emitting data entry engagement element, which emits light only when it is in at least predetermined propinquity to the at least one engagement surface, a two-dimensional imaging sensor viewing the at least one engagement surface for sensing light emitted by the light emitting data entry engagement element and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input.
Preferably, the light emitting data entry engagement element includes an elongated element having an LED at an end thereof. Additionally, the LED includes an IR emitting LED. Additionally or alternatively, the data entry apparatus also includes a proximity switch.
There is further provided in accordance with yet another preferred embodiment of the present invention data entry apparatus for use with at least one engagement surface, the data entry apparatus including a projector operative to illuminate the at least one engagement surface and to define thereon at least one touchpad region and at least one keyboard region, the touchpad region being defined by a zero'th order diffracted image, an illuminator operative to illuminate the at least one engagement surface, a two-dimensional imaging sensor viewing the at least one engagement surface from a location inwardly of the at least one engagement surface for sensing light from the illuminator scattered by engagement of a user's finger with the at least one engagement surface and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input.
Preferably, the processor includes the following functionality: as each pixel value is acquired, determining, using pixel coordinates, whether that pixel lies within a predefined keystroke region, acquiring pixel values for the pixel coordinates lying within the predefined keystroke region, adding or subtracting each of the pixel values to or from a pixel total maintained for each the keystroke region based on determining a pixel function of each pixel, comparing the pixel total for each the keystroke region with a current key actuation threshold, if the pixel total exceeds the key actuation threshold for a given keystroke region in a given frame and in the previous frame the pixel total did not exceed the key actuation threshold for that keystroke region, providing a key actuation output and if the pixel total does not exceed the key actuation threshold for a given keystroke region in a given frame and in the previous frame the pixel total did exceed the key actuation threshold for that keystroke region, providing a key deactuation output.
There is even further provided in accordance with still another preferred embodiment of the present invention a data input device including an illuminator operative to illuminate at least one engagement plane by directing light along the at least one engagement plane, a two-dimensional imaging sensor, including an optical wedge element, viewing the at least one engagement plane from a location outside the at least one engagement plane for sensing light from the illuminator scattered by engagement of a data entry object with the at least one engagement plane and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to utilization circuitry.
Preferably, the optical wedge element enables the imaging sensor to efficiently sense light from the engagement plane.
There is still further provided in accordance with yet another preferred embodiment of the present invention a data input device including an illuminator operative to illuminate at least one engagement plane by directing light along the at least one engagement plane, a two-dimensional imaging sensor, including a pair of oppositely oriented optical wedge elements, viewing the at least one engagement plane from a location outside the at least one engagement plane for sensing light from the illuminator scattered by engagement of a data entry object with the at least one engagement plane and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to utilization circuitry.
There is also provided in accordance with still another preferred embodiment of the present invention a data input device including an illuminator operative to illuminate at least one engagement plane by directing light along the at least one engagement plane and to illuminate a proper positioning confirmation location on an engagement surface, a two-dimensional imaging sensor, including, viewing the at least one engagement plane from a location outside the at least one engagement plane for sensing light from the illuminator scattered by engagement of a data entry object with the at least one engagement plane and a data entry processor receiving an output from the two-dimensional imaging sensor and providing a data entry input to utilization circuitry.
Preferably, the imaging sensor senses the presence or absence of illumination at the proper positioning confirmation location. Additionally, the data entry apparatus is operative to unpower itself when the imaging sensor senses the absence of illumination at the proper positioning confirmation location.
There is further provided in accordance with yet another preferred embodiment of the present invention gaming apparatus including at least one game piece, a game template projector, operative to project a game template onto a surface, an illuminator operative to illuminate the surface and an imaging sensor viewing the surface for sensing light from the illuminator reflected by at least one optically encoded reflector mounted on the at least one game piece.
In accordance with another preferred embodiment of the present invention each of the at least one reflector identifies one of the at least one game piece and is operative to indicate a location of the at least one game piece on the game template. Additionally or alternatively, the at least one reflector also identifies a spatial orientation of the game piece.
BRIEF DESCRIPTION OF THE DRAWINGS
The present invention will be understood and appreciated more fully from the following detailed description, taken in conjunction with the drawings in which:
FIGS. 1A & 1B are simplified pictorial illustrations of an electronic camera constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation;
FIGS. 2A & 2B are simplified pictorial illustrations of a camera-equipped cellular telephone, constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation;
FIGS. 3A & 3B are simplified pictorial illustrations of a camera-equipped personal digital assistant, constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation;
FIGS. 4A & 4B are simplified diagrammatic illustrations of the operation of imaging functionality selection in accordance with a preferred embodiment of the present invention in two different modes of operation;
FIGS. 5A & 5B are simplified diagrammatic illustrations of the operation of imaging functionality selection in accordance with another preferred embodiment of the present invention in two different modes of operation;
FIG. 6 is a simplified diagrammatic illustration of a data entry object engagement location sensing subsystem useful in the embodiments of any of FIGS. 1A-5B;
FIG. 7 is a simplified flow chart illustrating operation of the data entry object engagement location sensing subsystem of FIG. 6 in accordance with a preferred embodiment of the present invention;
FIG. 8 is a simplified illustration of a preferred data structure employed in the operation of the data entry object engagement location sensing subsystem shown in FIG. 7;
FIG. 9 is a simplified pictorial illustration of outlines of typical keystroke regions as sensed by a two-dimensional image sensor viewing a keyboard, such as the keyboard seen in FIG. 6;
FIG. 10 is a simplified pictorial illustration of outlines of typical footprints of a typical light pattern occasioned by data entry object engagement with several keystroke regions, such as those shown in FIG. 9;
FIGS. 11A, 11B and 11C are simplified illustrations of three alternative methodologies for determining the function of the pixel within the keystroke region in which it lies as shown in FIG. 7;
FIGS. 12A, 12B and 12C are simplified illustrations of traces which are useful in understanding FIGS. 11A, 11B and 11C;
FIG. 13 is a simplified flow chart illustrating operation of a data entry object engagement location sensing subsystem employed in a tracking system and methodology constructed and operative in accordance with a preferred embodiment of the present invention;
FIG. 14 is a simplified illustration showing synchronized illumination power variation functionality useful in accordance with a preferred embodiment of the present invention;
FIG. 15 is a simplified partially pictorial, partially diagrammatic illustration of data entry apparatus embodied in a vehicle access control system;
FIG. 16 is a simplified partially pictorial, partially diagrammatic illustration of a variation in the data entry apparatus of FIG. 15;
FIG. 17 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention;
FIGS. 18A and 18B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention;
FIGS. 19A and 19B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with yet another preferred embodiment of the present invention;
FIGS. 20A and 20B are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with still another preferred embodiment of the present invention;
FIG. 21 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with a further preferred embodiment of the present invention;
FIG. 22 is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention;
FIG. 23 is a simplified pictorial illustration of gaming apparatus constructed and operative in accordance with a preferred embodiment of the present invention; and
FIG. 24 is a simplified pictorial illustration of gaming apparatus constructed and operative in accordance with another preferred embodiment of the present invention.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
Reference is now made to FIGS. 1A & 1B, which are simplified pictorial illustrations of an electronic camera constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation. As seen in FIGS. 1A & 1B, an electronic camera 100, which may be of conventional construction other than as specifically described hereinbelow, such as a Nikon COOLPIX 5700, available from Nikon Corporation, Tokyo, Japan, is equipped with an electronic imaging sensor module 102, which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 Orleans Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
A user-operated imaging functionality selection switch 104 is preferably provided to enable a user to select operation in one of two possible imaging functionalities. In a first imaging functionality, illustrated in FIG. 1A, the electronic imaging sensor module 102 is employed to take a picture of a scene in the imaged field. In a second imaging functionality, illustrated in FIG. 1B, the electronic imaging sensor module 102 is employed for data entry responsive to user hand activity.
Various embodiments of the first and second imaging functionalities will be described hereinbelow, with reference to FIGS. 4A-5B. Preferably, but not necessarily, the second imaging functionality, as shown in FIG. 1B, employs a data entry template projector 106, which projects a data entry template, such as that designated by reference numeral 108, onto a surface, such as a table surface 110. An illuminator 111 preferably is provided and is operative to illuminate a region adjacent the data entry template 108. The imaging sensor module 102 views the data entry template region for sensing light from the illuminator 111. The second imaging functionality also preferably incorporates processing functionality, an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference.
It is seen in FIG. 1A that when the electronic camera 100 is operative in the first imaging functionality, also termed a picture mode, a digital picture 112 is provided. Similarly, it is seen in FIG. 1B that when the electronic camera 100 is operative in the second imaging functionality, also termed a data mode, a stream of data 114 is provided.
Reference is now made to FIGS. 2A & 2B, which are simplified pictorial illustrations of a camera-equipped cellular telephone, constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation. As seen in FIGS. 2A & 2B, a camera-equipped cellular telephone 200, which may be of conventional construction other than as specifically described hereinbelow, such as a Samsung SPH-I330, available from Samsung Electronics, Seoul, Korea, is equipped with an electronic imaging sensor module 202, which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 Orleans Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
A user-operated imaging functionality selection switch 204 is preferably provided to enable a user to select operation in one of two possible imaging functionalities. In a first imaging functionality, illustrated in FIG. 2A, the electronic imaging sensor module 202 is employed to take a picture of a scene in the imaged field. In a second imaging functionality, illustrated in FIG. 2B, the electronic imaging sensor module 202 is employed for data entry responsive to user hand activity.
Various embodiments of the first and second imaging functionalities will be described hereinbelow, with reference to FIGS. 4A-5B. Preferably, but not necessarily, the second imaging functionality, as shown in FIG. 2B, employs a data entry template projector 206, which projects a data entry template, such as that designated by reference numeral 208, onto a surface, such as a table surface 210. An illuminator 211 preferably is provided and is operative to illuminate a region adjacent the data entry template 208. The imaging sensor module 202 views the data entry template region for sensing light from the illuminator 211. The second imaging functionality also preferably incorporates processing functionality an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference.
It is seen in FIG. 2A that when the camera-equipped cellular telephone 200 is operative in the first imaging functionality, also termed a picture mode, a digital picture 212 is provided. Similarly, it is seen in FIG. 2B that when the camera-equipped cellular telephone 200 is operative in the second imaging functionality, also termed a data mode, a stream of data 214 is provided.
Reference is now made to FIGS. 3A & 3B, which are simplified pictorial illustrations of a camera-equipped personal digital assistant, constructed and operative in accordance with a preferred embodiment of the present invention in two different modes of operation. As seen in FIGS. 3A & 3B, a camera-equipped personal digital assistant 300, which may be of conventional construction other than as specifically described hereinbelow, such as a Samsung SPH-I700, available from Samsung Electronics, Seoul, Korea, is equipped with an electronic imaging sensor module 302, which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 Orleans Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
A user-operated imaging functionality selection switch 304 is preferably provided to enable a user to select operation in one of two possible imaging functionalities. In a first imaging functionality, illustrated in FIG. 3A, the electronic imaging sensor module 302 is employed to take a picture of a scene in the imaged field. In a second imaging functionality, illustrated in FIG. 3B, the electronic imaging sensor module 302 is employed for data entry responsive to user hand activity.
Various embodiments of the first and second imaging functionalities will be described hereinbelow, with reference to FIGS. 4A-5B. Preferably, but not necessarily, the second imaging functionality, as shown in FIG. 3B, employs a data entry template projector 306, which projects a data entry template, such as that designated by reference numeral 308, onto a surface, such as a table surface 310. An illuminator 311 preferably is provided and is operative to illuminate a region adjacent the data entry template 308. The imaging sensor module 302 views the data entry template region for sensing light from the illuminator 311. The second imaging functionality also preferably incorporates processing functionality an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference.
It is seen in FIG. 3A that when the electronic camera 300 is operative in the first imaging functionality, also termed a picture mode, a digital picture 312 is provided. Similarly, it is seen in FIG. 3B that when the electronic camera 300 is operative in the second imaging functionality, also termed a data mode, a stream of data 314 is provided.
Reference is now made to FIGS. 4A & 4B, which are simplified diagrammatic illustrations of the operation of imaging functionality selection in accordance with a preferred embodiment of the present invention in two different modes of operation. As seen in FIGS. 4A & 4B, one implementation of the invention involves the use of a first and second selectably positionable optical assemblies which may be selectably placed in front of an electronic imaging sensor module 402.
In the illustrated embodiment, the sensor module 402 comprises a two-dimensional CMOS detector array 404 which is overlaid by an array of color filters 406, which pass infrared radiation. A lens 408 focuses light from an imaging field onto array 404.
FIG. 4A shows a first selectably positionable optical assembly 410, preferably comprising an IR blocking filter 412, aligned with lens 408. The imaging functionality of FIG. 4A is suitable for picture taking, since it prevents IR radiation from distorting the color response of the sensor module 402.
FIG. 4B shows a second selectably positionable optical assembly 420, preferably comprising a visible radiation blocking filter 422, an IR band pass filter 424 and a field lens 426, aligned with lens 408. The imaging functionality of FIG. 4B is suitable for data input, since it allows only a relatively narrow band of IR radiation to reach the sensor module 402 from a relatively wide field of view.
Reference is now made to FIGS. 5A & 5B, which are simplified diagrammatic illustrations of the operation of imaging functionality selection in accordance with another preferred embodiment of the present invention in two different modes of operation. In this embodiment, imaging functionality selection is effected electronically, without the need for mechanically movable optical elements.
As seen in FIGS. 5A & 5B, this implementation of the invention involves the use of an electronic imaging sensor module 502 including a CMOS detector array 504 overlaid with a repeating array of pixelwise selective light transmissive filters 506 characterized in that the array includes red (R), green (G), blue (B) and infrared (IR) transmissive filters. A lens 508 focuses light from an imaging field onto detector array 504.
FIG. 5A shows the imaging functionality of the embodiment of FIGS. 5A and 5B in a picture taking mode, wherein only the outputs of the pixels underlying the R, G and B filters are employed and the respective IR content of the outputs of each of the pixels underlying the R, G and B filters is subtracted therefrom. This color processing prevents IR radiation from distorting the color response of the sensor module 502.
FIG. 5B shows the imaging functionality of the embodiment of FIGS. 5A and 5B in a data entry mode, wherein only the outputs of the pixels underlying the IR filters are employed.
Reference is now made to FIG. 6, which is a simplified and generalized illustration of a projection keyboard system and methodology which is preferably incorporated into the embodiments of FIGS. 1A-5B in accordance with a preferred embodiment of the present invention. A projection subsystem 550, preferably including a solid state light source (not shown) which illuminates a spatial light modulation element (not shown), defines an image 552 of a keyboard on a projection surface 554, preferably an inert surface, such as a desktop.
An illumination subsystem 556, preferably including a solid state light source (not shown), directs light in a radially directed illumination pattern 558, which preferably extends in a plane generally parallel to the projection surface 554. It is appreciated that the radially directed illumination pattern 558 has a very narrow spread in the direction generally perpendicular to the projection surface 554. It is further appreciated that the radially directed illumination pattern 558 is preferably located very close to the projection surface 554.
Impingement of light in the radially directed illumination pattern 558 on a data entry object 560, such as a user's finger, a stylus or other implement, causes light to be scattered or reflected therefrom. It is appreciated that the light is only scattered or reflected when the data entry object 560 is in close contact with the keyboard 552 defined on projection surface 554.
A detection subsystem 562, preferably employing a solid state imaging sensor (not shown) incorporated in a camera in the embodiments of FIGS. 1A-5B, receives light reflected or scattered from data entry object 560. The received light is mapped onto the imaging sensor with respect to a representation of the keyboard for associating the location of the data entry object 560 sensed by detection subsystem 562 with a key location 563 on the keyboard 552. Thus, the sensed location of data entry object 560 indicates which key of the keyboard 552 is being engaged.
Reference is now made to FIG. 7, which is a simplified flow chart illustrating operation of a data entry object engagement location sensing subsystem of FIG. 6 in accordance with a preferred embodiment of the present invention, and to FIG. 8, which is a simplified illustration of a preferred data structure employed in the operation of the data entry object engagement location sensing subsystem shown in FIG. 7.
FIG. 8 shows a simplified illustration of a preferred data structure employed in the operation of the data entry object engagement location sensing method described hereinbelow with respect to FIG. 7. It is appreciated that the imaging sensor of a camera, such as camera 100 (FIG. 1A), is typically comprised of a set of M×N pixels, wherein a particular group of pixels views a defined region of the engagement plane which preferably overlies the projection surface 110 (FIG. 1A). Thus, it is possible that a particular pixel group, located in the image plane of the camera 100, may receive scattered light from a data entry object 560 (FIG. 6) engaging the key location 563 (FIG. 6).
Thus, as the camera 100 views the projection surface 110, each of the M×N pixels in the image plane of the camera 100 may receive light from a corresponding region in the engagement plane in respect of a data entry object engagement therewith.
As seen in FIG. 7, pixel values, such as gray level values, are acquired for various pixel coordinates. As each pixel value is acquired, a determination is made, using the pixel coordinates, as to whether that pixel lies within a predefined keystroke region, such as keystroke regions 600 shown in FIG. 9. This determination is preferably made by employing a pixel index table 601 which indicates, for each pixel, whether that pixel lies within a predetermined keystroke region, such as keystroke regions 605, 606, 607 and 608 (FIG. 9), and, if so, within which keystroke region it lies.
The function of the pixel within the keystroke region in which it lies is then determined, preferably by employing table 601. This function is typically additive or subtractive, but may alternatively have another function. Typically, depending on the function, the pixel value is added to or subtracted from a pixel total maintained for each keystroke region in a keystroke region accumulator table 602.
Once all of the pixels in a frame have been processed as aforesaid, an updated background level is determined for the frame and a key actuation threshold is determined typically by subtracting the updated background level from a predetermined threshold level which is established for each keystroke region. This is preferably carried out by employing a keystroke region threshold table 604.
The contents of the keystroke region accumulator table 602 for each keystroke region preferably are then compared with the current key actuation threshold. If the contents of the accumulator table 602 exceed the key actuation threshold for a given key actuation region in a given frame and in the previous frame the contents of the accumulator table 602 did not exceed the key actuation threshold, a key actuation output is provided.
Similarly, if the contents of the accumulator table 602 do not exceed the key actuation threshold for a given key actuation region in a given frame and in the previous frame the contents of the accumulator table 602 did exceed the key actuation threshold, a key deactuation output is provided. In all other cases, no output need be generated.
Reference is now made to FIG. 9, which is a simplified pictorial illustration of outlines of typical keystroke regions 605, 606, 607 and 608 as sensed by a two-dimensional image sensor (FIG. 6) viewing a keyboard, such as the keyboard 552, seen in FIG. 6.
Reference is now made to FIG. 10, which is a simplified pictorial illustration of outlines of typical footprints 609, 610, 611 and 612 of a typical light pattern occasioned by data entry object engagement corresponding to the keystroke regions 605, 606, 607 and 608 (FIG. 9).
Reference is now made to FIGS. 11A, 11B and 11C, which are simplified illustrations of three alternative methodologies for determining the function of the pixel within the keystroke region in which it lies as shown in FIG. 8, and to FIGS. 12A, 12B and 12C, which are simplified illustrations of traces which are useful in understanding FIGS. 11A, 11B and 11C.
Turning now to FIG. 1A, there is shown arranged along a common arbitrary axis 618 a simplified keystroke region 620 corresponding to a given key and containing a plurality of pixels 622. A typical simplified footprint of a typical light pattern occasioned by data entry object engagement with the given key is indicated by reference numeral 624. FIG. 10 shows outlines of typical footprints 609, 610, 611 and 612, corresponding to keystroke regions designated 605, 606, 607 and 608 in FIG. 9.
A typical background signal pattern is indicated by reference numeral 634. Superimposition of the footprint 624 over the background signal pattern 634 is indicated at reference numeral 635. A one dimensionally selectable overlap of footprint 624 over keystroke region 620 is indicated at reference numeral 636. A one dimensionally selectable overlap of background signal pattern 634 over keystroke region 620 is indicated at reference numeral 637. A one dimensionally selectable overlap of superimposition 635 over keystroke region 620 is indicated at reference numeral 638.
FIG. 12A illustrates a bias function 640 corresponding to a cross section of the keystroke region 620 taken along axis 618, which bias function defines keystroke region 620 along axis 618. There is also seen a signal function 644 corresponding to a cross section of the footprint 624 along axis 618; a background signal function 646 corresponding to a cross section of the background signal pattern 634 along axis 618 and a combination signal 648 corresponding to a cross section of the superimposition 635.
There is also shown in FIG. 12A a one dimensional convolution 650 corresponding to one dimensionally selectable overlap 636; a one dimensional convolution 652 corresponding to one dimensionally selectable overlap 637 and a one dimensional convolution 654 corresponding to one dimensionally selectable overlap 638. Convolution 650 is shown with a threshold 660; convolution 652 is shown with a threshold 662 and convolution 654 is shown with a threshold 664.
Turning now to FIG. 11B, there is shown arranged along a common arbitrary axis 670 a simplified keystroke region 680 corresponding to a given key and containing a plurality of pixels 682. The simplified keystroke region 680 is here shown surrounded by a simplified keystroke region border 683. A typical simplified footprint of a typical light pattern occasioned by data entry object engagement with the given key is indicated by reference numeral 684. A typical background signal pattern is indicated by reference numeral 686. Superimposition of the footprint 684 over the background signal pattern 686 is indicated at reference numeral 688. A one dimensionally selectable overlap of footprint 684 over keystroke region 680 and border 683 is indicated at reference numeral 690. A one dimensionally selectable overlap of background signal pattern 686 over keystroke region 680 and border 683 is indicated at reference numeral 692. A one dimensionally selectable overlap of superimposition 688 over keystroke region 680 and border 683 is indicated at reference numeral 694.
FIG. 12B illustrates a bias function 700 corresponding to a cross section of the keystroke region 680 and of the border 683 taken along axis 670, which bias function defines keystroke region 680 along axis 670. It is seen that border 683 is assigned a negative value relative to the value of the keystroke region 680. This value assignment is appreciated to enhance the value of data entry object engagements to the extent that they lie within the keystroke region 680 and to decrease the value of such data entry object engagements to the extent that they extend over the border 683. There is also seen a signal function 704 corresponding to a cross section of the footprint 684 along axis 670; a background signal function 706 corresponding to a cross section of the background signal pattern 686 along axis 670 and a combination signal 708 corresponding to a cross section of the superimposition 688.
There is also shown in FIG. 12B a one dimensional convolution 720, corresponding to one dimensionally selectable overlap 690; a one dimensional convolution 722, corresponding to one dimensionally selectable overlap 692 and a one dimensional convolution 724 corresponding to one dimensionally selectable overlap 694. Convolution 720 is shown with a threshold 726; convolution 722 is shown with a threshold 727 and convolution 724 is shown with a threshold 728.
Turning now to FIG. 11C, there is shown arranged along a common arbitrary axis 730 a simplified keystroke region 740 corresponding to a given key and containing a plurality of pixels 741. The simplified keystroke region 740 is here shown surrounded by a simplified keystroke region inner border 742 and by a simplified keystroke region outer border 743. A typical simplified footprint of a typical light pattern occasioned by data entry object engagement with the given key is indicated by reference numeral 744. A typical background signal pattern is indicated by reference numeral 746. Superimposition of the footprint 744 over the background signal pattern 746 is indicated at reference numeral 748. A one dimensionally selectable overlap of footprint 744 over keystroke region 740 and borders 742 and 743 is indicated at reference numeral 750. A one dimensionally selectable overlap of background signal pattern 746 over keystroke region 740 and borders 742 and 743 is indicated at reference numeral 752. A one dimensionally selectable overlap of superimposition 748 over keystroke region 740 and borders 742 and 743 is indicated at reference numeral 754.
FIG. 12C illustrates a bias function 760 corresponding to a cross section of the keystroke region 740 and of the borders 742 and 743 taken along axis 730, which bias function defines keystroke region 740 along axis 730. It is seen that border 742 is assigned a zero value and border 743 is assigned a negative value relative to the value of the keystroke region 740. It is appreciated that these value assignments enhance the value of data entry object engagements that to the extent that they lie within the keystroke region 740 and to decrease the value of such data entry object engagements to the extent that they extend across the border 742 and at least into border 743. There is also seen a signal function 764 corresponding to a cross section of the footprint 744 along axis 730; a background signal function 766 corresponding to a cross section of the background signal pattern 746 along axis 730 and a combination signal 768 corresponding to a cross section of the superimposition 748.
There is also shown in FIG. 12C a one dimensional convolution 780, corresponding to one dimensionally selectable overlap 750; a one dimensional convolution 782, corresponding to one dimensionally selectable overlap 752 and a one dimensional convolution 784 corresponding to one dimensionally selectable overlap 754. Convolution 780 is shown with a threshold 786; convolution 782 is shown with a threshold 787 and convolution 784 is shown with a threshold 788.
It may be appreciated from a consideration of convolutions 638, 694 and 754 that the dual border arrangement of FIGS. 11C and 12C provides the best detection of data entry object keystroke engagement, while minimizing background effects.
Reference is now made to FIG. 13, which is a simplified flow chart illustrating operation of a data entry object engagement location sensing subsystem employed in a tracking system and methodology constructed and operative in accordance with a preferred embodiment of the present invention.
As seen in FIG. 13, pixel values, such as gray level values, are acquired for various pixel coordinates. As each pixel value is acquired, a determination may be made, using the pixel coordinates, as to whether that pixel lies within a predefined active region. Typically, if the pixel does not lie within a predetermined active region, its value is ignored.
The pixel value for each pixel is preferably thresholded and typically all pixel values falling below a given threshold are ignored. The remaining pixel values may be weighted by a selected weighting parameter. In order to determine the “center of gravity” of the thresholded and weighted pixel values, the thresholded and weighted pixel values are multiplied respectively by X and Y values representing the coordinate position of each pixel and the results are summed along mutually perpendicular axes X and Y and stored in X and Y accumulators. The total of the thresholded and weighted pixel values for all relevant pixels are also summed and stored in a data accumulator, for the entire active region.
Once all of the pixels in a frame have been processed as aforesaid, the summed thresholded and weighted pixel values along the X and Y axes respectively are divided by the total of the thresholded and weighted pixel values for the entire active region to determine the X and Y coordinates of the “center of gravity”, which represents a desired engagement location.
Reference is now made to FIG. 14, which is a simplified illustration showing synchronized illumination power variation functionality useful in accordance with a preferred embodiment of the present invention. The functionality illustrated in FIG. 14 is directed to modulating the amount of illumination provided for data entry object engagement detection. This modulation is desirable because the intensity of light impinging on a data entry object and thus scattered thereby, decreases with the distance between an illuminator 830 and a data entry object. Thus, it may be appreciated that the amount of light impinging on a data entry object 832 is substantially greater than the amount of light impinging on a data entry object 834. Furthermore, the amount of scattered light impinging on a detector 836 decreases with the distance between the data entry object and the detector. These two distance dependencies make detection of data entry object engagement difficult.
In order to overcome this difficulty, there is provided in accordance with a preferred embodiment of the present invention variable intensity drive electronics 840 which is coupled to both illuminator 830 and detector 836 and which causes the intensity of light produced by the illuminator 830 to vary, typically in a ramp fashion, in synchronization to the imaging field location of detector 836.
Thus, it may be seen that when a near portion (A) of the engagement surface 554 (FIG. 6) is being imaged, typically at the top portion A of detector 836, the intensity is at a minimum. When an intermediate portion (B) of the engagement surface 554 is being imaged, typically at the middle of detector 836, the intensity is at an intermediate level and when a far portion (C) of the engagement surface 554 is being imaged, typically at the bottom portion (C) of the detector 836, the intensity is at a maximum.
Variable intensity drive electronics 840 operates preferably by providing a synchronization output 842 to detector 836 and a corresponding synchronization output 844 to illuminator 830, causing the intensity level to increase in time in synchronization with the location of a scanned image region in detector 836.
Reference is now made to FIG. 15, which is a simplified partially pictorial, partially diagrammatic illustration of data entry apparatus embodied in a vehicle access control system. As seen in FIG. 15, a vehicle 1500, typically comprising a frame, a drive train and a chassis including at least one lockable entry portal or door 1502 and portal unlocking functionality, includes an access control assembly designated generally by reference numeral 1504.
The access control assembly 1504 preferably includes at least one at least partially light transmissive user finger engagement surface accessible from outside the chassis. In the illustrated embodiment, plural user finger engagement surfaces 1506 are defined on a car door window 1508. An illuminator 1510, preferably located interiorly of the exterior of the vehicle, such as an IR line generator, is operative to illuminate a finger engagement plane 1512 adjacent the user finger engagement surfaces 1506. A preferred IR line generator is described in applicant's Published PCT Application WO02/054169 A2, the disclosure of which is hereby incorporated by reference.
A two-dimensional imaging sensor 1514, located interiorly of the exterior of the vehicle, views the finger engagement surfaces 1506, from a location inwardly of the car door window 1508, for sensing light originating from the illuminator 1510, which is scattered by engagement of a user's finger with the finger engagement surfaces 1506. A preferred two-dimensional imaging sensor 1514 is described in applicant's Published PCT Application WO02/054169 A2, the disclosure of which is hereby incorporated by reference.
A data entry processor 1516 receives an output from the two-dimensional imaging sensor 1514 and provides a data entry input to a conventional door unlocking functionality (not shown).
Reference is now made to FIG. 16, which is a simplified diagrammatic illustration of a variation in the data entry apparatus of FIG. 15. As seen in FIG. 16, a vehicle 1600, typically comprising a frame, a drive train and a chassis including at least one lockable entry portal or door 1602 and portal unlocking functionality, includes an access control assembly designated generally by reference numeral 1604.
The access control assembly 1604 preferably includes at least one at least partially light transmissive user finger engagement surface accessible from outside the chassis. In the illustrated embodiment, plural user finger engagement surfaces 1606 are defined on a car door window 1608. In this embodiment, the plural user finger engagement surfaces 1606 may be arranged in a two-dimensional array. An illuminator 1610, preferably located interiorly of the exterior of the vehicle, such as an IR line generator, is operative to illuminate a finger engagement plane 1612 adjacent the user finger engagement surfaces 1606. A preferred IR line generator is described in applicant's Published PCT Application WO02/054169 A2, the disclosure of which is hereby incorporated by reference.
In the embodiment of FIG. 16, the finger engagement planes 1612 are parallel to the finger engagement surfaces 1606, by virtue of provision of a prism 1613 disposed typically on the outside of the vehicle window 1608 along the light path from illuminator 1610.
A two-dimensional imaging sensor 1614, located interiorly of the exterior of the vehicle, views the finger engagement surfaces 1606, from a location inwardly of the car door window 1608, for sensing light originating from the illuminator 1610, which is scattered by engagement of a user's finger with the finger engagement surfaces 1606. A preferred two-dimensional imaging sensor 1614 is described in applicant's Published PCT Application WO02/054169 A2, the disclosure of which is hereby incorporated by reference.
A data entry processor 1616 receives an output from the two-dimensional imaging sensor 1614 and provides a data entry input to a conventional door unlocking functionality (not shown).
It is appreciated that portions of the apparatus of FIGS. 15 and 16 may find application in other types of access control environments and in various data entry environments.
Reference is now made to FIG. 17, which is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention. As seen in FIG. 17, a light emitting data entry engagement element, preferably in the form of a stylus 1700, emits light when and only when it is in at least predetermined propinquity to an engagement surface 1702.
A two-dimensional imaging sensor 1704 views the engagement surface 1702 for sensing light emitted by the stylus 1700. A data entry processor 1706 receives an output from the two-dimensional imaging sensor 1704 and provides a data entry input.
As seen in FIG. 17, the stylus 1700 preferably comprises an IR emitting LED 1708 located in an IR transmissive tip 1710 thereof. A proximity switch 1712, such as a contact switch, provides an output indication of the proximity of tip 1710 to engagement surface 1702. Circuitry 1714 receives an input from proximity switch 1712 and operates the LED 1708 accordingly, so as to provide an IR emission whose location indicates the location of stylus impingement with engagement surface 1702.
Reference is now made to FIGS. 18A and 18B, which are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention. As seen in FIGS. 18A & 18B, the data entry apparatus comprises a projector 1800 operative to illuminate at least one engagement surface 1802 and to define thereon at least one touchpad region 1804 and at least one keyboard region 1806, the touchpad region 1804 being defined by a zero'th order diffracted image provided by the projector 1800.
Turning particularly to FIG. 18B, it is seen that the projector 1800 comprises a light source, such as a diode laser 1810, which outputs via a collimating lens 1812 onto a first diffractive optical element 1814. Diffractive optical element 1814 diffracts the light impinging thereon into a primary keyboard image beam 1816 which impinges on at last one keyboard region 1806 and into a secondary ghost image beam 1818, which is preferably blocked by a barrier 1819. Residual light passing through diffractive optical element 1814 defines a zero'th order beam 1820, which impinges onto a second diffractive optical element 1822.
Diffractive optical element 1822 diffracts the light impinging thereon from the zero'th order beam 1820 into a touchpad defining beam 1824 which impinges on at least one touchpad region 1804.
Returning to FIG. 18A, it is seen that there is also provided an illuminator 1830, operative to illuminate the finger engagement surface. A two-dimensional imaging sensor 1832 views the engagement surface for sensing light from the illuminator 1830 scattered by engagement of a user's finger with the finger engagement surface. A data entry processor 1834 receives an output from the two-dimensional imaging sensor 1832 and provides a data entry input.
Reference is now made to FIGS. 19A and 19B, which are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with yet another preferred embodiment of the present invention. As seen in FIGS. 19A & 19B, the data entry apparatus comprises a projector 1900 operative to illuminate at least one finger engagement surface 1902 and to define thereon typically at least one keyboard region 1904. An illuminator 1906 is operative to illuminate the finger engagement surface 1902.
An imaging sensor module 1908, including a two-dimensional imaging sensor 1910, views the engagement surface for sensing light from the illuminator 1906 scattered by engagement of a user's finger with the finger engagement surface 1902. A particular feature of the imaging sensor module 1908 is that it includes an optical wedge element 1912 which provides a tilt, enabling a generally vertically disposed imaging sensor 1910 to efficiently sense light from a finger engagement surface 1902 extending generally perpendicularly thereto.
A data entry processor 1914 receives an output from the two-dimensional imaging sensor 1910 and provides a data entry input.
Reference is now made to FIGS. 20A and 20B, which are respectively a simplified pictorial and a simplified sectional illustration of data entry apparatus constructed and operative in accordance with still another preferred embodiment of the present invention. As seen in FIGS. 20A & 20B, the data entry apparatus comprises a projector 2000 operative to illuminate at least one finger engagement surface 2002 and to define thereon typically at least one keyboard region 2004. An illuminator 2006 is operative to illuminate the finger engagement surface 2002.
An imaging sensor module 2008, including a two-dimensional imaging sensor 2010, views the engagement surface for sensing light from the illuminator 2006 scattered by engagement of a user's finger with the finger engagement surface 2002. A particular feature of the imaging sensor module 2008 is that it includes a pair of oppositely oriented optical wedge elements 2012 and 2014, each of which provides a tilt to a portion of the light incoming to sensor 2010, enabling efficient use of area of generally rectangular imaging sensor, as seen at reference number 2016.
Reference is now made to FIG. 21, which is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with yet another preferred embodiment of the present invention. As seen in FIG. 21, the data entry apparatus comprises a projector 2100 operative to illuminate at least one finger engagement surface 2102 and to define thereon typically at least one keyboard region 2104. An illuminator 2106 is operative to illuminate the finger engagement surface 2102.
An imaging sensor module 2108, including a two-dimensional imaging sensor 2110, views the engagement surface for sensing light from the illuminator 2106 scattered by engagement of a user's finger with the finger engagement surface 2102.
A particular feature of the illuminator 2106 is that defines an auxiliary light path 2112 which impinges on a proper positioning confirmation location 2114 on the finger engagement surface 2102 adjacent and preferably nearer to the illuminator than the at least one keyboard region 2104. The imaging sensor module 2108 senses the presence of illumination at location 2114 and in the absence of sensed illumination at that location, typically indicating improper placement of the data entry apparatus, typically unpowers the data entry apparatus.
A data entry processor 2116 receives an output from the two-dimensional imaging sensor 2110 and provides a data entry input.
Reference is now made to FIG. 22, which is a simplified pictorial illustration of data entry apparatus constructed and operative in accordance with another preferred embodiment of the present invention in the form of a wristwatch. As seen in FIG. 22, a wristwatch 2200, which may be of conventional construction other than as specifically described hereinbelow, is equipped with an electronic imaging sensor module 2202, which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 Orleans Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
A user-operated imaging functionality selection switch 2204 is preferably provided to enable a user to select operation in a data entry functionality employing the electronic imaging sensor module 2202 for data entry responsive to user hand activity.
The data entry functionality preferably employs a data entry template projector 2206, which projects a data entry template, such as that designated by reference numeral 2208, onto a surface, such as a table surface 2210. An illuminator 2212 is operative to illuminate a region adjacent the data entry template. The imaging sensor module 2202 views the data entry template region for sensing light from the illuminator 2212 scattered by engagement of a user's finger with the data entry template.
The data entry functionality also preferably incorporates processing functionality, an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference. The data entry functionality produces a stream of data 2214.
Reference is now made to FIG. 23, which is a simplified pictorial illustration of gaming apparatus constructed and operative in accordance with a preferred embodiment of the present invention. As seen in FIG. 23, gaming apparatus, one example of which is a board game, such as chess, is provided with an electronic imaging sensor module 2302, which may include a CMOS array, for example, CMOS array OVT6130, available from Omnivision Technologies, Inc., 1341 Orleans Drive, Sunnyvale, Calif., USA, providing an output representing an imaged field.
A user-operated imaging functionality selection switch 2304 is preferably provided to enable a user to select operation of one of plural gaming functionalities, such as different board games, employing the electronic imaging sensor module 2302 for gaming piece locating.
The gaming functionality preferably employs a gaming template projector 2306, which projects a game template, such as a chess board, designated by reference numeral 2308, onto a surface, such as a table surface 2310. An illuminator 2312 is operative to illuminate a region adjacent the game template 2308. The imaging sensor module 2302 views the game template region for sensing light from the illuminator 2312 reflected by optically encoded reflectors 2314 mounted on individual gaming pieces 2316. The reflectors 2314 preferably individually identify each gaming piece and indicate its location on the game template 2308.
The gaming functionality also preferably incorporates processing functionality, an example of which is described in applicant's Published PCT Application WO 02/054169 A2, the disclosure of which is hereby incorporated by reference. The gaming functionality produces a stream of data 2316, which may be used for game management and scoring.
Reference is now made to FIG. 24, which is a simplified pictorial illustration of gaming apparatus constructed and operative in accordance with another preferred embodiment of the present invention. FIG. 24 shows gaming pieces, such as dice 2402, which bear variously differently coded reflectors 2404 on various surfaces thereof, thereby to enable not only the identity and location but also the spatial orientation of the gaming pieces to be sensed.
It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove as well as modifications and variations thereof as would occur to a person of skill in the art upon reading the foregoing specification and which are not in the prior art.

Claims (23)

1. A vehicle comprising:
a frame, a drive train and a chassis, including at least one lockable entry portal and portal unlocking functionality; and
an access control assembly for governing authorized access to said at least one lockable entry portal comprising:
at least one at least partially light transmissive user finger engagement surface accessible from outside said chassis;
an illuminator operative to illuminate at least one user finger engagement plane adjacent said at least one user finger engagement surface;
a two-dimensional imaging sensor viewing said at least one user finger engagement surface, from a location inwardly of said at least one finger engagement surface, for sensing light from said illuminator scattered by engagement of a user's finger with said at least one finger engagement surface; and
a data entry processor receiving an output from said two-dimensional imaging sensor and providing a data entry input to said portal unlocking functionality.
2. A vehicle according to claim 1 wherein said illuminator is located at a location inwardly of said at least one engagement surface.
3. A vehicle according to claim 1 wherein said at least one finger engagement surface comprises a one-dimensional array of a plurality of finger engagement surfaces.
4. A vehicle according to claim 1 wherein said at least one finger engagement surface comprises a two-dimensional array of a plurality of finger engagement surfaces.
5. A vehicle according to claim 1 and also comprising an illumination director cooperating with said illuminator for providing an illumination beam generally parallel to said at least one finger engagement surface.
6. A vehicle according to claim 5 wherein said illumination director comprises a prism.
7. Access control apparatus for use with a vehicle including a chassis having at least one lockable entry portal and portal unlocking functionality and also including at least one at least partially light transmissive user finger engagement surface accessible from outside said chassis, said access control apparatus comprising:
an illuminator operative to illuminate at least one user finger engagement plane adjacent said at least one user finger engagement surface;
a two-dimensional imaging sensor viewing said at least one user finger engagement surface from a location inwardly of said at least one finger engagement surface for sensing light from said illuminator scattered by engagement of a user's finger with said at least one finger engagement surface; and
a data entry processor receiving an output from said two-dimensional imaging sensor and providing a data entry input to said portal unlocking functionality.
8. Access control apparatus according to claim 7 wherein said illuminator is located at a location inwardly of said at least one engagement surface.
9. Access control apparatus according to claim 7 wherein said at least one finger engagement surface comprises a one-dimensional array of a plurality of finger engagement surfaces.
10. Access control apparatus according to claim 7 wherein said at least one finger engagement surface comprises a two-dimensional array of a plurality of finger engagement surfaces.
11. Access control apparatus according to claim 7 and also comprising an illumination director cooperating with said illuminator for providing an illumination beam generally parallel to said at least one finger engagement surface.
12. Access control apparatus according to claim 11 and wherein said illumination director comprises a prism.
13. Access control apparatus for use with an enclosure having at least one lockable entry portal and portal unlocking functionality and also including at least one at least partially light transmissive user finger engagement surface accessible from outside said enclosure, said access control apparatus comprising:
an illuminator operative to illuminate at least one user finger engagement plane adjacent said at least one user finger engagement surface;
a two-dimensional imaging sensor viewing said at least one user finger engagement surface from a location inwardly of said at least one finger engagement surface for sensing light from said illuminator scattered by engagement of a user's finger with said at least one finger engagement surface; and
a data entry processor receiving an output from said two-dimensional imaging sensor and providing a data entry input to said portal unlocking functionality.
14. Access control apparatus according to claim 13 wherein said illuminator is located at a location inwardly of said at least one engagement surface.
15. Access control apparatus according to claim 13 wherein said at least one finger engagement surface comprises a one-dimensional array of a plurality of finger engagement surfaces.
16. Access control apparatus according to claim 13 wherein said at least one finger engagement surface comprises a two-dimensional array of a plurality of finger engagement surfaces.
17. Access control apparatus according to claim 13 and also comprising an illumination director cooperating with said illuminator for providing an illumination beam generally parallel to said at least one finger engagement surface.
18. Access control apparatus according to claim 17 wherein said illumination director comprises a prism.
19. Data entry apparatus for use with at least one at least partially light transmissive user finger engagement surface, said data entry apparatus comprising:
an illuminator operative to illuminate at least one user finger engagement plane adjacent said at least one user finger engagement surface;
an illumination director cooperating with said illuminator for providing an illumination beam generally parallel to said at least one finger engagement surface;
a two-dimensional imaging sensor viewing said at least one user finger engagement surface from a location inwardly of said at least one finger engagement surface for sensing light from said illuminator scattered by engagement of a user's finger with said at least one finger engagement surface; and
a data entry processor receiving an output from said two-dimensional imaging sensor and providing a data entry input.
20. Data entry apparatus according to claim 19 wherein said illuminator is located at a location inwardly of said at least one engagement surface.
21. Data entry apparatus according to claim 19 wherein said at least one finger engagement surface comprises a one-dimensional array of a plurality of finger engagement surfaces.
22. Data entry apparatus according to claim 19 wherein said at least one finger engagement surface comprises a two-dimensional array of a plurality of finger engagement surfaces.
23. Data entry apparatus according to claim 19 wherein said illumination director comprises a prism.
US10/823,337 2002-06-26 2004-04-13 Multifunctional integrated image sensor and application to virtual interface technology Expired - Fee Related US7417681B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US10/823,337 US7417681B2 (en) 2002-06-26 2004-04-13 Multifunctional integrated image sensor and application to virtual interface technology
US12/173,218 US20080309620A1 (en) 2002-06-26 2008-07-15 Multifunctional integrated image sensor and application to virtual interface technology

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US39237602P 2002-06-26 2002-06-26
US43832703P 2003-01-07 2003-01-07
PCT/IL2003/000538 WO2004003656A2 (en) 2002-06-26 2003-06-26 Multifunctional integrated image sensor and application to virtual interface technology
US10/481,932 US7307661B2 (en) 2002-06-26 2003-06-26 Multifunctional integrated image sensor and application to virtual interface technology
US10/823,337 US7417681B2 (en) 2002-06-26 2004-04-13 Multifunctional integrated image sensor and application to virtual interface technology

Related Parent Applications (3)

Application Number Title Priority Date Filing Date
US10481932 Division 2003-06-26
PCT/IL2003/000538 Division WO2004003656A2 (en) 2002-06-26 2003-06-26 Multifunctional integrated image sensor and application to virtual interface technology
US10/481,932 Division US7307661B2 (en) 2002-06-26 2003-06-26 Multifunctional integrated image sensor and application to virtual interface technology

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US12/173,218 Division US20080309620A1 (en) 2002-06-26 2008-07-15 Multifunctional integrated image sensor and application to virtual interface technology

Publications (2)

Publication Number Publication Date
US20040207732A1 US20040207732A1 (en) 2004-10-21
US7417681B2 true US7417681B2 (en) 2008-08-26

Family

ID=30003248

Family Applications (3)

Application Number Title Priority Date Filing Date
US10/481,932 Expired - Fee Related US7307661B2 (en) 2002-06-26 2003-06-26 Multifunctional integrated image sensor and application to virtual interface technology
US10/823,337 Expired - Fee Related US7417681B2 (en) 2002-06-26 2004-04-13 Multifunctional integrated image sensor and application to virtual interface technology
US12/173,218 Abandoned US20080309620A1 (en) 2002-06-26 2008-07-15 Multifunctional integrated image sensor and application to virtual interface technology

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/481,932 Expired - Fee Related US7307661B2 (en) 2002-06-26 2003-06-26 Multifunctional integrated image sensor and application to virtual interface technology

Family Applications After (1)

Application Number Title Priority Date Filing Date
US12/173,218 Abandoned US20080309620A1 (en) 2002-06-26 2008-07-15 Multifunctional integrated image sensor and application to virtual interface technology

Country Status (7)

Country Link
US (3) US7307661B2 (en)
EP (1) EP1540641A2 (en)
JP (1) JP2005533463A (en)
CN (1) CN1666248A (en)
AU (1) AU2003238660A1 (en)
CA (1) CA2490292A1 (en)
WO (1) WO2004003656A2 (en)

Cited By (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080246738A1 (en) * 2005-05-04 2008-10-09 Koninklijke Philips Electronics, N.V. System and Method for Projecting Control Graphics
US20090241437A1 (en) * 2008-03-19 2009-10-01 Wolfgang Steinle Embedding unit for display devices
DE202009005254U1 (en) 2009-03-27 2009-11-12 Tpk Touch Solutions Inc. Touch device, laser source module and laser source structure
US20090289188A1 (en) * 2008-05-20 2009-11-26 Everspring Industry Co., Ltd. Method for controlling an electronic device through infrared detection
DE202009005253U1 (en) 2009-09-09 2010-01-07 Tpk Touch Solutions Inc. Touch device, laser source module and laser source structure
US20100008582A1 (en) * 2008-07-10 2010-01-14 Samsung Electronics Co., Ltd. Method for recognizing and translating characters in camera-based image
US20100073404A1 (en) * 2008-09-24 2010-03-25 Douglas Stuart Brown Hand image feedback method and system
US20100142016A1 (en) * 2008-12-08 2010-06-10 Light Blue Optics Ltd. Holographic image projection systems
US20100315376A1 (en) * 2009-06-10 2010-12-16 Dave Jaeyeong Choi Control interface assemblies and vehicles including same
US20100328243A1 (en) * 2009-06-30 2010-12-30 E-Pin Optical Industry Co., Ltd Mems scanning touch panel and coordinate dection method thereof
US8077147B2 (en) 2005-12-30 2011-12-13 Apple Inc. Mouse with optical sensing surface
US8228305B2 (en) 1995-06-29 2012-07-24 Apple Inc. Method for providing human input to a computer
US8239784B2 (en) 2004-07-30 2012-08-07 Apple Inc. Mode-based graphical user interfaces for touch sensitive input devices
US20120268376A1 (en) * 2011-04-20 2012-10-25 Qualcomm Incorporated Virtual keyboards and methods of providing the same
US8314773B2 (en) 2002-09-09 2012-11-20 Apple Inc. Mouse having an optically-based scrolling feature
US20120306817A1 (en) * 2011-05-30 2012-12-06 Era Optoelectronics Inc. Floating virtual image touch sensing apparatus
WO2012172363A2 (en) 2011-06-16 2012-12-20 Light Blue Optics Ltd Touch sensitive display devices
WO2012172364A2 (en) 2011-06-16 2012-12-20 Light Blue Optics Ltd Touch-sensitive display devices
WO2012172360A2 (en) 2011-06-16 2012-12-20 Light Blue Optics Ltd Touch-sensitive display devices
US8381135B2 (en) 2004-07-30 2013-02-19 Apple Inc. Proximity detector in handheld device
WO2013054096A1 (en) 2011-10-11 2013-04-18 Light Blue Optics Limited Touch-sensitive display devices
US8482535B2 (en) 1999-11-08 2013-07-09 Apple Inc. Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US20130179811A1 (en) * 2012-01-05 2013-07-11 Visteon Global Technologies, Inc. Projection dynamic icon knobs
WO2013108031A2 (en) 2012-01-20 2013-07-25 Light Blue Optics Limited Touch sensitive image display devices
WO2013108032A1 (en) 2012-01-20 2013-07-25 Light Blue Optics Limited Touch sensitive image display devices
WO2013144599A2 (en) 2012-03-26 2013-10-03 Light Blue Optics Ltd Touch sensing systems
US8576199B1 (en) 2000-02-22 2013-11-05 Apple Inc. Computer control systems
US8610674B2 (en) 1995-06-29 2013-12-17 Apple Inc. Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US8791923B2 (en) 2009-03-27 2014-07-29 Tpk Touch Solutions Inc. Touching device, laser source module, and laser source structure thereof
US8994495B2 (en) 2012-07-11 2015-03-31 Ford Global Technologies Virtual vehicle entry keypad and method of use thereof
US20150168721A1 (en) * 2013-12-18 2015-06-18 Fuji Jukogyo Kabushiki Kaisha Onboard image display device for vehicle
US9239673B2 (en) 1998-01-26 2016-01-19 Apple Inc. Gesturing with a multipoint sensing device
US9239677B2 (en) 2004-05-06 2016-01-19 Apple Inc. Operation of a computer with touch screen interface
US9292111B2 (en) 1998-01-26 2016-03-22 Apple Inc. Gesturing with a multipoint sensing device
US9513744B2 (en) 1994-08-15 2016-12-06 Apple Inc. Control systems employing novel physical controls and touch screens
US9880629B2 (en) 2012-02-24 2018-01-30 Thomas J. Moscarillo Gesture recognition devices and methods with user authentication
US10137857B1 (en) * 2017-08-22 2018-11-27 Ford Global Technologies, Llc Vehicle unlocking systems, devices, and methods
US10300889B2 (en) 2017-09-08 2019-05-28 Ford Global Technologies, Llc Iris-detection alignment for vehicle feature activation
US20210365492A1 (en) * 2012-05-25 2021-11-25 Atheer, Inc. Method and apparatus for identifying input features for later recognition

Families Citing this family (56)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004523031A (en) * 2001-01-08 2004-07-29 ヴィーケービー インコーポレイテッド Data input device
AU2003238660A1 (en) * 2002-06-26 2004-01-19 Vkb Inc. Multifunctional integrated image sensor and application to virtual interface technology
EP1710665A4 (en) * 2004-01-15 2012-12-26 Vodafone Plc Mobile communication terminal
US20050273201A1 (en) * 2004-06-06 2005-12-08 Zukowski Deborra J Method and system for deployment of sensors
KR100663515B1 (en) * 2004-11-08 2007-01-02 삼성전자주식회사 A portable terminal apparatus and method for inputting data for the portable terminal apparatus
WO2006090386A2 (en) 2005-02-24 2006-08-31 Vkb Inc. A virtual keyboard device
JP4612853B2 (en) * 2005-03-29 2011-01-12 キヤノン株式会社 Pointed position recognition device and information input device having the same
TWI268095B (en) * 2005-06-02 2006-12-01 Benq Corp Scanner
US8018579B1 (en) 2005-10-21 2011-09-13 Apple Inc. Three-dimensional imaging and display system
FR2898315B1 (en) * 2006-03-08 2009-02-20 Peugeot Citroen Automobiles Sa CONTROL INTERFACE OF A FIXED EQUIPMENT OR NOMAD OF VEHICLE, WITH USE OF A VIRTUAL KEYBOARD
DE102006049172A1 (en) * 2006-10-18 2008-04-30 Dräger Medical AG & Co. KG Virtual display and control system for medical devices
KR20080044017A (en) * 2006-11-15 2008-05-20 삼성전자주식회사 Touch screen
KR101534363B1 (en) * 2007-09-24 2015-07-06 레이저 테크놀로지, 인코포레이티드 Integrated still image, motion video and speed measurement system
US8487881B2 (en) * 2007-10-17 2013-07-16 Smart Technologies Ulc Interactive input system, controller therefor and method of controlling an appliance
US9171454B2 (en) * 2007-11-14 2015-10-27 Microsoft Technology Licensing, Llc Magic wand
US20090159799A1 (en) * 2007-12-19 2009-06-25 Spectral Instruments, Inc. Color infrared light sensor, camera, and method for capturing images
JP5277703B2 (en) * 2008-04-21 2013-08-28 株式会社リコー Electronics
US8952894B2 (en) * 2008-05-12 2015-02-10 Microsoft Technology Licensing, Llc Computer vision-based multi-touch sensing using infrared lasers
WO2009148210A1 (en) * 2008-06-02 2009-12-10 Lg Electronics Inc. Virtual optical input unit and control method thereof
US8068641B1 (en) 2008-06-19 2011-11-29 Qualcomm Incorporated Interaction interface for controlling an application
US7777899B1 (en) * 2008-06-19 2010-08-17 Gesturetek, Inc. Interaction interface for controlling an application
TW201007254A (en) * 2008-08-04 2010-02-16 Pixart Imaging Inc Image-sensing module and image-sensing system
US8847739B2 (en) 2008-08-04 2014-09-30 Microsoft Corporation Fusing RFID and vision for surface object tracking
CN101650520A (en) * 2008-08-15 2010-02-17 索尼爱立信移动通讯有限公司 Visual laser touchpad of mobile telephone and method thereof
FR2937441B1 (en) * 2008-10-16 2010-12-31 Neopost Technologies VIRTUAL USER INTERFACE FOR MAIL PROCESSING SYSTEM.
JP5276417B2 (en) * 2008-11-25 2013-08-28 シャープ株式会社 LASER PROJECTOR DEVICE AND MOBILE PHONE TERMINAL HAVING THE SAME
US20100153457A1 (en) * 2008-12-15 2010-06-17 Grant Isaac W Gestural Interface Device and Method
GB2466497B (en) 2008-12-24 2011-09-14 Light Blue Optics Ltd Touch sensitive holographic displays
US7763841B1 (en) 2009-05-27 2010-07-27 Microsoft Corporation Optical component for a depth sensor
KR101773464B1 (en) 2009-06-30 2017-08-31 엘지전자 주식회사 Mobile terminal for providing multiplayer game and operation method thereof
GB0920754D0 (en) * 2009-11-27 2010-01-13 Compurants Ltd Inamo big book 1
KR200480404Y1 (en) * 2010-05-24 2016-05-20 카닛 보디팟 An apparatus for a virtual input device for a mobile computing device and the method therein
US8542218B2 (en) * 2010-08-19 2013-09-24 Hyundai Motor Company Electronic switch apparatus for vehicle
EP2441635B1 (en) * 2010-10-06 2015-01-21 Harman Becker Automotive Systems GmbH Vehicle User Interface System
US9195345B2 (en) * 2010-10-28 2015-11-24 Microsoft Technology Licensing, Llc Position aware gestures with visual feedback as input method
CN102169366B (en) * 2011-03-18 2012-11-07 汤牧天 Multi-target tracking method in three-dimensional space
WO2012176315A1 (en) * 2011-06-23 2012-12-27 富士通株式会社 Information processing device, input control method, and input control program
CN103019457A (en) * 2011-09-23 2013-04-03 原相科技股份有限公司 Optical touch system
WO2013104061A1 (en) 2012-01-11 2013-07-18 Smart Technologies Ulc Calibration of an interactive light curtain
TW201409298A (en) * 2012-08-21 2014-03-01 Wintek Corp Display module
CN103677358A (en) * 2012-09-06 2014-03-26 东莞万士达液晶显示器有限公司 Display module
JP5971053B2 (en) * 2012-09-19 2016-08-17 船井電機株式会社 Position detection device and image display device
US9377355B2 (en) * 2013-02-18 2016-06-28 Eminent Electronic Technology Corp. Ltd. Optical sensor apparatus and image sensing apparatus integrating multiple functions
US9047514B2 (en) * 2013-07-10 2015-06-02 Christie Digital Systems Usa, Inc. Apparatus, system and method for projecting images onto predefined portions of objects
US9049369B2 (en) * 2013-07-10 2015-06-02 Christie Digital Systems Usa, Inc. Apparatus, system and method for projecting images onto predefined portions of objects
JP2015079169A (en) * 2013-10-18 2015-04-23 増田 麻言 Projection device
CN104202547B (en) * 2014-08-27 2017-10-10 广东威创视讯科技股份有限公司 Method, projection interactive approach and its system of target object are extracted in projected picture
JP2016114963A (en) * 2014-12-11 2016-06-23 株式会社リコー Input operation detection device, projector device, electronic blackboard device, digital signage device, and projector system
CN107111217B (en) * 2014-12-25 2020-10-27 索尼公司 Projection display unit
US20160255323A1 (en) 2015-02-26 2016-09-01 Dual Aperture International Co. Ltd. Multi-Aperture Depth Map Using Blur Kernels and Down-Sampling
FR3034889B1 (en) * 2015-04-10 2017-04-28 Cn2P Sas ELECTRONIC BRACELET FOR DISPLAYING AN INTERACTIVE DIGITAL CONTENT TO BE PROJECTED ON A ZONE OF AN ARM
US9898142B2 (en) * 2016-04-01 2018-02-20 Ford Global Technologies, Llc Touch detection on a curved surface
WO2017212601A1 (en) * 2016-06-09 2017-12-14 日立マクセル株式会社 Optical distance-measurement device and image projection device provided with same
WO2018203595A1 (en) * 2017-05-04 2018-11-08 임성현 Projector capable of touch interaction
US11287885B1 (en) * 2019-01-22 2022-03-29 Facebook Technologies, Llc Apparatus, system, and method for determining the position of wearables donned by users of artificial reality systems
CN112791376B (en) * 2021-01-28 2023-05-12 济南其石电子科技有限公司 Electronic chessboard controller and implementation method of electronic chessboard

Citations (43)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4561017A (en) 1983-08-19 1985-12-24 Richard Greene Graphic input apparatus
US4843568A (en) 1986-04-11 1989-06-27 Krueger Myron W Real time perception of and response to the actions of an unencumbered participant/user
US5181108A (en) 1991-10-07 1993-01-19 Greene Richard M Graphic input device with uniform sensitivity and no keystone distortion
US5181181A (en) 1990-09-27 1993-01-19 Triton Technologies, Inc. Computer apparatus input device for three-dimensional information
US5182659A (en) 1991-02-20 1993-01-26 Holographix, Inc. Holographic recording and scanning system and method
US5457550A (en) 1991-02-27 1995-10-10 Ricoh Company, Ltd. Optical scanner unit having recursive optical system
US5680205A (en) 1996-08-16 1997-10-21 Dew Engineering And Development Ltd. Fingerprint imaging apparatus with auxiliary lens
DE29802435U1 (en) 1998-02-12 1998-05-07 Siemens Nixdorf Inf Syst Arrangement of the projection surface of a virtual input unit
FR2756077A1 (en) 1996-11-19 1998-05-22 Opto System Tactile screen and viewing system
US5767842A (en) 1992-02-07 1998-06-16 International Business Machines Corporation Method and device for optical input of commands or data
US5781252A (en) 1996-04-02 1998-07-14 Kopin Corporation Dual light valve color projector system
US5952731A (en) * 1998-02-02 1999-09-14 Lear Automotive Dearborn, Inc. Membrane keyless entry switch for vehicles
EP0982676A1 (en) 1998-08-27 2000-03-01 Hewlett-Packard Company A method and apparatus for a virtual display/keyboard for a PDA
US6043839A (en) 1997-10-06 2000-03-28 Adair; Edwin L. Reduced area imaging devices
WO2000021024A1 (en) 1998-10-07 2000-04-13 Intel Corporation Input device using scanning sensors
EP1039365A2 (en) 1999-03-26 2000-09-27 Nokia Mobile Phones Ltd. An input arrangement for manual entry of data and a mobile phone
US6218967B1 (en) * 1996-04-01 2001-04-17 Kyosti Veijo Olavi Maula Arrangement for the optical remote control of apparatus
WO2001059975A2 (en) 2000-02-11 2001-08-16 Canesta, Inc. Method and apparatus for entering data using a virtual input device
US6281238B1 (en) 1998-05-12 2001-08-28 American Home Products Corporation 4-aryl-1-oxa-9-thia-cylopenta[B]fluorenes
US6281878B1 (en) * 1994-11-01 2001-08-28 Stephen V. R. Montellese Apparatus and method for inputing data
US6297894B1 (en) 1998-08-31 2001-10-02 R. J. Dwayne Miller Optical scheme for holographic imaging of complex diffractive elements in materials
WO2001093182A1 (en) 2000-05-29 2001-12-06 Vkb Inc. Virtual data entry device and method for input of alphanumeric and other data
US20020021287A1 (en) 2000-02-11 2002-02-21 Canesta, Inc. Quasi-three-dimensional method and apparatus to detect and localize interaction of user-object and virtual transfer device
US6377238B1 (en) 1993-04-28 2002-04-23 Mcpheters Robert Douglas Holographic control arrangement
WO2002054169A2 (en) 2001-01-08 2002-07-11 Vkb Inc. A data input device
US6424338B1 (en) 1999-09-30 2002-07-23 Gateway, Inc. Speed zone touchpad
US20030132921A1 (en) * 1999-11-04 2003-07-17 Torunoglu Ilhami Hasan Portable sensory input device
US6607277B2 (en) 1996-09-24 2003-08-19 Seiko Epson Corporation Projector display comprising light source units
US6611252B1 (en) 2000-05-17 2003-08-26 Dufaux Douglas P. Virtual data input device
US6650318B1 (en) 2000-10-13 2003-11-18 Vkb Inc. Data input device
US6690357B1 (en) * 1998-10-07 2004-02-10 Intel Corporation Input device using scanning sensors
US6690354B2 (en) * 2000-11-19 2004-02-10 Canesta, Inc. Method for enhancing performance in a system utilizing an array of sensors that sense at least two-dimensions
US6750849B2 (en) 2000-12-15 2004-06-15 Nokia Mobile Phones, Ltd. Method and arrangement for accomplishing a function in an electronic apparatus and an electronic apparatus
US6854870B2 (en) * 2001-06-30 2005-02-15 Donnelly Corporation Vehicle handle assembly
US6911972B2 (en) 2001-04-04 2005-06-28 Matsushita Electric Industrial Co., Ltd. User interface device
US20050169527A1 (en) 2000-05-26 2005-08-04 Longe Michael R. Virtual keyboard system with automatic correction
US20060190836A1 (en) * 2005-02-23 2006-08-24 Wei Ling Su Method and apparatus for data entry input
US7151530B2 (en) * 2002-08-20 2006-12-19 Canesta, Inc. System and method for determining an input selected by a user through a virtual interface
US7215327B2 (en) * 2002-12-31 2007-05-08 Industrial Technology Research Institute Device and method for generating a virtual keyboard/display
US7230611B2 (en) * 2002-12-20 2007-06-12 Siemens Aktiengesellschaft HMI device with optical touch screen
US7248151B2 (en) * 2005-01-05 2007-07-24 General Motors Corporation Virtual keypad for vehicle entry control
US7307661B2 (en) * 2002-06-26 2007-12-11 Vbk Inc. Multifunctional integrated image sensor and application to virtual interface technology
US20080074386A1 (en) * 2006-09-27 2008-03-27 Chia-Hoang Lee Virtual input device and the input method thereof

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6211521B1 (en) * 1998-03-13 2001-04-03 Intel Corporation Infrared pixel sensor and infrared signal correction
US6657663B2 (en) * 1998-05-06 2003-12-02 Intel Corporation Pre-subtracting architecture for enabling multiple spectrum image sensing
US6700613B1 (en) * 1998-06-16 2004-03-02 Eastman Kodak Company Data-reading image capture apparatus, camera, and method of use
US6512838B1 (en) * 1999-09-22 2003-01-28 Canesta, Inc. Methods for enhancing performance and data acquired from three-dimensional image systems
US20030174125A1 (en) * 1999-11-04 2003-09-18 Ilhami Torunoglu Multiple input modes in overlapping physical space
US7042442B1 (en) * 2000-06-27 2006-05-09 International Business Machines Corporation Virtual invisible keyboard
US6761634B1 (en) * 2001-06-07 2004-07-13 Hasbro, Inc. Arcade table
US7071924B2 (en) * 2002-01-10 2006-07-04 International Business Machines Corporation User input method and apparatus for handheld computers
JP4311988B2 (en) * 2003-06-12 2009-08-12 アキュートロジック株式会社 Color filter for solid-state image sensor and color image pickup apparatus using the same
US20070115261A1 (en) * 2005-11-23 2007-05-24 Stereo Display, Inc. Virtual Keyboard input system using three-dimensional motion detection by variable focal length lens
JP2006033483A (en) * 2004-07-16 2006-02-02 Aisin Seiki Co Ltd Color imaging apparatus

Patent Citations (50)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4561017A (en) 1983-08-19 1985-12-24 Richard Greene Graphic input apparatus
US4843568A (en) 1986-04-11 1989-06-27 Krueger Myron W Real time perception of and response to the actions of an unencumbered participant/user
US5181181A (en) 1990-09-27 1993-01-19 Triton Technologies, Inc. Computer apparatus input device for three-dimensional information
US5182659A (en) 1991-02-20 1993-01-26 Holographix, Inc. Holographic recording and scanning system and method
US5457550A (en) 1991-02-27 1995-10-10 Ricoh Company, Ltd. Optical scanner unit having recursive optical system
US5181108A (en) 1991-10-07 1993-01-19 Greene Richard M Graphic input device with uniform sensitivity and no keystone distortion
US5767842A (en) 1992-02-07 1998-06-16 International Business Machines Corporation Method and device for optical input of commands or data
US6377238B1 (en) 1993-04-28 2002-04-23 Mcpheters Robert Douglas Holographic control arrangement
US6281878B1 (en) * 1994-11-01 2001-08-28 Stephen V. R. Montellese Apparatus and method for inputing data
US6218967B1 (en) * 1996-04-01 2001-04-17 Kyosti Veijo Olavi Maula Arrangement for the optical remote control of apparatus
US5781252A (en) 1996-04-02 1998-07-14 Kopin Corporation Dual light valve color projector system
US5680205A (en) 1996-08-16 1997-10-21 Dew Engineering And Development Ltd. Fingerprint imaging apparatus with auxiliary lens
US6607277B2 (en) 1996-09-24 2003-08-19 Seiko Epson Corporation Projector display comprising light source units
FR2756077A1 (en) 1996-11-19 1998-05-22 Opto System Tactile screen and viewing system
US6043839A (en) 1997-10-06 2000-03-28 Adair; Edwin L. Reduced area imaging devices
US5952731A (en) * 1998-02-02 1999-09-14 Lear Automotive Dearborn, Inc. Membrane keyless entry switch for vehicles
DE29802435U1 (en) 1998-02-12 1998-05-07 Siemens Nixdorf Inf Syst Arrangement of the projection surface of a virtual input unit
US6281238B1 (en) 1998-05-12 2001-08-28 American Home Products Corporation 4-aryl-1-oxa-9-thia-cylopenta[B]fluorenes
EP0982676A1 (en) 1998-08-27 2000-03-01 Hewlett-Packard Company A method and apparatus for a virtual display/keyboard for a PDA
US6297894B1 (en) 1998-08-31 2001-10-02 R. J. Dwayne Miller Optical scheme for holographic imaging of complex diffractive elements in materials
WO2000021024A1 (en) 1998-10-07 2000-04-13 Intel Corporation Input device using scanning sensors
US6690357B1 (en) * 1998-10-07 2004-02-10 Intel Corporation Input device using scanning sensors
EP1039365A2 (en) 1999-03-26 2000-09-27 Nokia Mobile Phones Ltd. An input arrangement for manual entry of data and a mobile phone
US6424338B1 (en) 1999-09-30 2002-07-23 Gateway, Inc. Speed zone touchpad
US20030132921A1 (en) * 1999-11-04 2003-07-17 Torunoglu Ilhami Hasan Portable sensory input device
US6614422B1 (en) * 1999-11-04 2003-09-02 Canesta, Inc. Method and apparatus for entering data using a virtual input device
US20040046744A1 (en) * 1999-11-04 2004-03-11 Canesta, Inc. Method and apparatus for entering data using a virtual input device
US20020021287A1 (en) 2000-02-11 2002-02-21 Canesta, Inc. Quasi-three-dimensional method and apparatus to detect and localize interaction of user-object and virtual transfer device
WO2001059975A2 (en) 2000-02-11 2001-08-16 Canesta, Inc. Method and apparatus for entering data using a virtual input device
US6710770B2 (en) 2000-02-11 2004-03-23 Canesta, Inc. Quasi-three-dimensional method and apparatus to detect and localize interaction of user-object and virtual transfer device
US6798401B2 (en) 2000-05-17 2004-09-28 Tree Frog Technologies, Llc Optical system for inputting pointer and character data into electronic equipment
US6611252B1 (en) 2000-05-17 2003-08-26 Dufaux Douglas P. Virtual data input device
US20050169527A1 (en) 2000-05-26 2005-08-04 Longe Michael R. Virtual keyboard system with automatic correction
WO2001093182A1 (en) 2000-05-29 2001-12-06 Vkb Inc. Virtual data entry device and method for input of alphanumeric and other data
US20060101349A1 (en) * 2000-05-29 2006-05-11 Klony Lieberman Virtual data entry device and method for input of alphanumeric and other data
US6650318B1 (en) 2000-10-13 2003-11-18 Vkb Inc. Data input device
US6690354B2 (en) * 2000-11-19 2004-02-10 Canesta, Inc. Method for enhancing performance in a system utilizing an array of sensors that sense at least two-dimensions
US6750849B2 (en) 2000-12-15 2004-06-15 Nokia Mobile Phones, Ltd. Method and arrangement for accomplishing a function in an electronic apparatus and an electronic apparatus
US7242388B2 (en) * 2001-01-08 2007-07-10 Vkb Inc. Data input device
WO2002054169A2 (en) 2001-01-08 2002-07-11 Vkb Inc. A data input device
US20070222760A1 (en) * 2001-01-08 2007-09-27 Vkb Inc. Data input device
US6911972B2 (en) 2001-04-04 2005-06-28 Matsushita Electric Industrial Co., Ltd. User interface device
US6854870B2 (en) * 2001-06-30 2005-02-15 Donnelly Corporation Vehicle handle assembly
US7307661B2 (en) * 2002-06-26 2007-12-11 Vbk Inc. Multifunctional integrated image sensor and application to virtual interface technology
US7151530B2 (en) * 2002-08-20 2006-12-19 Canesta, Inc. System and method for determining an input selected by a user through a virtual interface
US7230611B2 (en) * 2002-12-20 2007-06-12 Siemens Aktiengesellschaft HMI device with optical touch screen
US7215327B2 (en) * 2002-12-31 2007-05-08 Industrial Technology Research Institute Device and method for generating a virtual keyboard/display
US7248151B2 (en) * 2005-01-05 2007-07-24 General Motors Corporation Virtual keypad for vehicle entry control
US20060190836A1 (en) * 2005-02-23 2006-08-24 Wei Ling Su Method and apparatus for data entry input
US20080074386A1 (en) * 2006-09-27 2008-03-27 Chia-Hoang Lee Virtual input device and the input method thereof

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Leibe, et al., "Toward spontaneous interaction with the perceptive workbench", IEEE computer graphics and applications, IEEE service center, New York, NY vol. 20, No. 6, Nov. 2000, pp. 54-64.
Supplementary European Search Report dated Aug. 14, 2007 for European Application No. EP 01 27 2779.

Cited By (56)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9513744B2 (en) 1994-08-15 2016-12-06 Apple Inc. Control systems employing novel physical controls and touch screens
US8228305B2 (en) 1995-06-29 2012-07-24 Apple Inc. Method for providing human input to a computer
US8427449B2 (en) 1995-06-29 2013-04-23 Apple Inc. Method for providing human input to a computer
US9758042B2 (en) 1995-06-29 2017-09-12 Apple Inc. Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US8610674B2 (en) 1995-06-29 2013-12-17 Apple Inc. Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US9239673B2 (en) 1998-01-26 2016-01-19 Apple Inc. Gesturing with a multipoint sensing device
US9292111B2 (en) 1998-01-26 2016-03-22 Apple Inc. Gesturing with a multipoint sensing device
US8482535B2 (en) 1999-11-08 2013-07-09 Apple Inc. Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US8576199B1 (en) 2000-02-22 2013-11-05 Apple Inc. Computer control systems
US9606668B2 (en) 2002-02-07 2017-03-28 Apple Inc. Mode-based graphical user interfaces for touch sensitive input devices
US8314773B2 (en) 2002-09-09 2012-11-20 Apple Inc. Mouse having an optically-based scrolling feature
US9239677B2 (en) 2004-05-06 2016-01-19 Apple Inc. Operation of a computer with touch screen interface
US11036282B2 (en) 2004-07-30 2021-06-15 Apple Inc. Proximity detector in handheld device
US8381135B2 (en) 2004-07-30 2013-02-19 Apple Inc. Proximity detector in handheld device
US8239784B2 (en) 2004-07-30 2012-08-07 Apple Inc. Mode-based graphical user interfaces for touch sensitive input devices
US8479122B2 (en) 2004-07-30 2013-07-02 Apple Inc. Gestures for touch sensitive input devices
US8612856B2 (en) 2004-07-30 2013-12-17 Apple Inc. Proximity detector in handheld device
US9348458B2 (en) 2004-07-30 2016-05-24 Apple Inc. Gestures for touch sensitive input devices
US10042418B2 (en) 2004-07-30 2018-08-07 Apple Inc. Proximity detector in handheld device
US20080246738A1 (en) * 2005-05-04 2008-10-09 Koninklijke Philips Electronics, N.V. System and Method for Projecting Control Graphics
US8077147B2 (en) 2005-12-30 2011-12-13 Apple Inc. Mouse with optical sensing surface
US8307591B2 (en) * 2008-03-19 2012-11-13 Brainlab Ag Embedding unit for display devices
US20090241437A1 (en) * 2008-03-19 2009-10-01 Wolfgang Steinle Embedding unit for display devices
US20090289188A1 (en) * 2008-05-20 2009-11-26 Everspring Industry Co., Ltd. Method for controlling an electronic device through infrared detection
US20100008582A1 (en) * 2008-07-10 2010-01-14 Samsung Electronics Co., Ltd. Method for recognizing and translating characters in camera-based image
US8228345B2 (en) * 2008-09-24 2012-07-24 International Business Machines Corporation Hand image feedback method and system
US8421824B2 (en) 2008-09-24 2013-04-16 International Business Machines Corporation Hand image feedback
US20100073404A1 (en) * 2008-09-24 2010-03-25 Douglas Stuart Brown Hand image feedback method and system
US20100142016A1 (en) * 2008-12-08 2010-06-10 Light Blue Optics Ltd. Holographic image projection systems
US8154780B2 (en) 2008-12-08 2012-04-10 Light Blue Optics, Ltd. Holographic image projection systems
DE202009005254U1 (en) 2009-03-27 2009-11-12 Tpk Touch Solutions Inc. Touch device, laser source module and laser source structure
US8791923B2 (en) 2009-03-27 2014-07-29 Tpk Touch Solutions Inc. Touching device, laser source module, and laser source structure thereof
US20100315376A1 (en) * 2009-06-10 2010-12-16 Dave Jaeyeong Choi Control interface assemblies and vehicles including same
US20100328243A1 (en) * 2009-06-30 2010-12-30 E-Pin Optical Industry Co., Ltd Mems scanning touch panel and coordinate dection method thereof
DE202009005253U1 (en) 2009-09-09 2010-01-07 Tpk Touch Solutions Inc. Touch device, laser source module and laser source structure
US8928589B2 (en) * 2011-04-20 2015-01-06 Qualcomm Incorporated Virtual keyboards and methods of providing the same
US20120268376A1 (en) * 2011-04-20 2012-10-25 Qualcomm Incorporated Virtual keyboards and methods of providing the same
US20120306817A1 (en) * 2011-05-30 2012-12-06 Era Optoelectronics Inc. Floating virtual image touch sensing apparatus
WO2012172364A2 (en) 2011-06-16 2012-12-20 Light Blue Optics Ltd Touch-sensitive display devices
WO2012172363A2 (en) 2011-06-16 2012-12-20 Light Blue Optics Ltd Touch sensitive display devices
WO2012172360A2 (en) 2011-06-16 2012-12-20 Light Blue Optics Ltd Touch-sensitive display devices
US9524061B2 (en) 2011-06-16 2016-12-20 Promethean Limited Touch-sensitive display devices
WO2013054096A1 (en) 2011-10-11 2013-04-18 Light Blue Optics Limited Touch-sensitive display devices
US20130179811A1 (en) * 2012-01-05 2013-07-11 Visteon Global Technologies, Inc. Projection dynamic icon knobs
WO2013108032A1 (en) 2012-01-20 2013-07-25 Light Blue Optics Limited Touch sensitive image display devices
WO2013108031A2 (en) 2012-01-20 2013-07-25 Light Blue Optics Limited Touch sensitive image display devices
US9880629B2 (en) 2012-02-24 2018-01-30 Thomas J. Moscarillo Gesture recognition devices and methods with user authentication
US11009961B2 (en) 2012-02-24 2021-05-18 Thomas J. Moscarillo Gesture recognition devices and methods
WO2013144599A2 (en) 2012-03-26 2013-10-03 Light Blue Optics Ltd Touch sensing systems
US20210365492A1 (en) * 2012-05-25 2021-11-25 Atheer, Inc. Method and apparatus for identifying input features for later recognition
US8994495B2 (en) 2012-07-11 2015-03-31 Ford Global Technologies Virtual vehicle entry keypad and method of use thereof
US20150168721A1 (en) * 2013-12-18 2015-06-18 Fuji Jukogyo Kabushiki Kaisha Onboard image display device for vehicle
US10042163B2 (en) * 2013-12-18 2018-08-07 Subaru Corporation Onboard image display device for vehicle
US10137857B1 (en) * 2017-08-22 2018-11-27 Ford Global Technologies, Llc Vehicle unlocking systems, devices, and methods
US10300889B2 (en) 2017-09-08 2019-05-28 Ford Global Technologies, Llc Iris-detection alignment for vehicle feature activation
US11167725B2 (en) 2017-09-08 2021-11-09 Ford Global Technologies, Llc Iris-detection alignment for vehicle feature activation

Also Published As

Publication number Publication date
WO2004003656A3 (en) 2004-06-24
EP1540641A2 (en) 2005-06-15
US20080309620A1 (en) 2008-12-18
CA2490292A1 (en) 2004-01-08
WO2004003656A2 (en) 2004-01-08
US20040207732A1 (en) 2004-10-21
US20040246338A1 (en) 2004-12-09
JP2005533463A (en) 2005-11-04
US7307661B2 (en) 2007-12-11
AU2003238660A1 (en) 2004-01-19
CN1666248A (en) 2005-09-07

Similar Documents

Publication Publication Date Title
US7417681B2 (en) Multifunctional integrated image sensor and application to virtual interface technology
JP2005533463A5 (en)
CN111316287B (en) Lens-pinhole array design in ultra-thin underscreen optical sensors for on-screen fingerprint sensing
US10360432B1 (en) Optical imaging via imaging lens and imaging pinhole in under-screen optical sensor module for on-screen fingerprint sensing in devices having organic light emitting diode (OLED) screens or other screens
TW511038B (en) High contrast, low distortion optical acquisition systems for image capturing
US7627364B2 (en) Image apparatus and method, and communication terminal device
KR102579508B1 (en) Camera assembly and electronics
US10885299B2 (en) Electronic device including pin hole array mask above optical image sensor and laterally adjacent light source and related methods
US7302174B2 (en) Method and apparatus for capturing images using a color laser projection display
US7557935B2 (en) Optical coordinate input device comprising few elements
US6911972B2 (en) User interface device
US8103064B2 (en) Finger identification method and apparatus
CN109074474A (en) Electronic equipment and correlation technique including the processing circuit for sensing the image from subarray spaced apart
US20070222760A1 (en) Data input device
CN107430681A (en) Include the electronic equipment and correlation technique of the pinhole array mask above optical image sensor
US20050239525A1 (en) Gaming machine
AU2005200822A1 (en) Information input device
CN114270803A (en) Phase Detection Autofocus (PDAF) sensor
JPH11195082A (en) Optical information reading device
KR20050019794A (en) Multifunctional integrated image sensor and application to virtual interface technology
JP2021028733A (en) Object identification device and object identification system
EP3987248B1 (en) Projecting a structured light pattern from an apparatus having an oled display screen
JPH07200140A (en) Indicated position detection device for display device
US11920919B2 (en) Projecting a structured light pattern from an apparatus having an OLED display screen
KR20220037473A (en) Shooting device and authentication device

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 4

REMI Maintenance fee reminder mailed
LAPS Lapse for failure to pay maintenance fees
STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20160826