US20090146972A1 - Apparatus and method for detecting a pointer relative to a touch surface - Google Patents

Apparatus and method for detecting a pointer relative to a touch surface Download PDF

Info

Publication number
US20090146972A1
US20090146972A1 US12/369,904 US36990409A US2009146972A1 US 20090146972 A1 US20090146972 A1 US 20090146972A1 US 36990409 A US36990409 A US 36990409A US 2009146972 A1 US2009146972 A1 US 2009146972A1
Authority
US
United States
Prior art keywords
input system
distortion
interactive input
pointer
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/369,904
Inventor
Gerald D. Morrison
David E. Holmgren
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Smart Technologies ULC
Original Assignee
Smart Technologies ULC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Smart Technologies ULC filed Critical Smart Technologies ULC
Priority to US12/369,904 priority Critical patent/US20090146972A1/en
Publication of US20090146972A1 publication Critical patent/US20090146972A1/en
Assigned to MORGAN STANLEY SENIOR FUNDING INC. reassignment MORGAN STANLEY SENIOR FUNDING INC. SECURITY AGREEMENT Assignors: SMART TECHNOLOGIES INC., SMART TECHNOLOGIES ULC
Assigned to MORGAN STANLEY SENIOR FUNDING, INC. reassignment MORGAN STANLEY SENIOR FUNDING, INC. SECURITY AGREEMENT Assignors: SMART TECHNOLOGIES INC., SMART TECHNOLOGIES ULC
Assigned to SMART TECHNOLOGIES INC., SMART TECHNOLOGIES ULC reassignment SMART TECHNOLOGIES INC. RELEASE OF ABL SECURITY INTEREST Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to SMART TECHNOLOGIES ULC, SMART TECHNOLOGIES INC. reassignment SMART TECHNOLOGIES ULC RELEASE OF TERM LOAN SECURITY INTEREST Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to SMART TECHNOLOGIES ULC, SMART TECHNOLOGIES INC. reassignment SMART TECHNOLOGIES ULC RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to SMART TECHNOLOGIES ULC, SMART TECHNOLOGIES INC. reassignment SMART TECHNOLOGIES ULC RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0428Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/0416Control or interface arrangements specially adapted for digitisers
    • G06F3/0418Control or interface arrangements specially adapted for digitisers for error correction or compensation, e.g. based on parallax, calibration or alignment
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/70Determining position or orientation of objects or cameras
    • G06T7/73Determining position or orientation of objects or cameras using feature-based methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/80Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85Stereo camera calibration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10004Still image; Photographic image
    • G06T2207/10012Stereo images
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20092Interactive image processing based on input by user
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30244Camera pose

Definitions

  • the present invention relates generally to interactive input systems and in particular to an apparatus and method for detecting a pointer relative to a touch surface.
  • Touch systems are well known in the art and typically include a touch screen having a touch surface on which contacts are made using a pointer in order to generate user input. Pointer contacts with the touch surface are detected and are used to generate corresponding output depending on areas of the touch surface where the pointer contacts are made. Common touch systems utilize analog resistive, electromagnetic, capacitive, acoustic or machine vision techniques to identify pointer contacts on the touch surface.
  • a camera-based touch system comprising a touch screen that includes a passive touch surface on which a computer-generated image is presented.
  • a rectangular bezel or frame surrounds the touch surface and supports digital cameras at its corners.
  • the digital cameras have overlapping fields of view that encompass and look generally across the plane of the touch surface.
  • the digital cameras acquire images looking across the touch surface from different locations and generate image data.
  • Image data acquired by the digital cameras is processed by digital signal processors to determine if a pointer exists in the captured image data.
  • the digital signal processors convey pointer characteristic data to a master controller, which in turn processes the pointer characteristic data to determine the location of the pointer in (x, y)-co-ordinates relative to the touch surface using triangulation.
  • the pointer co-ordinate data is conveyed to a computer executing one or more applications programs.
  • the computer uses the pointer co-ordinate data to update the computer-generated image that is presented on the touch surface. Pointer contacts on the touch surface can therefore be recorded as writing or drawing or used to control execution of applications programs executed by the computer.
  • an apparatus for detecting a pointer relative to a touch surface comprising at least two spaced imaging devices having overlapping fields of view encompassing the touch surface.
  • the imaging devices see the touch surface in three-dimensions as a perspective view.
  • the imaging devices acquire images from different locations.
  • a processor receives and processes image data generated by at least one of the imaging devices to determine the location of the pointer relative to the touch surface.
  • Each imaging device is calibrated to establish the relationship between points (X, Y, Z) in its perspective view and points (x, y) in acquired images.
  • Each imaging device generates pointer co-ordinate data when a pointer is captured in an acquired image.
  • the processor triangulates the pointer co-ordinate data to determine the location of the pointer relative to the touch surface.
  • the apparatus includes a pair of imaging devices with each imaging device being positioned adjacent a different corner of the touch surface. Each imaging device is spaced from and spaced in front of the touch surface. Each imaging device is positioned relative to the touch surface so that at a minimum the entire periphery of the touch surface is within its perspective view.
  • calibration points (X, Y, Z) on the touch surface and image points (x, y) corresponding to the calibration points are measured.
  • Collinearity equations are solved using the measured calibration and image points to determine external and internal orientation parameters of the imaging devices.
  • the collinearity equations are solved using a least-squares method.
  • the calibrations points are at spaced location along the periphery of the touch surface such as at the corners and edge mid-points of the touch surface.
  • the external orientation parameters of the imaging devices are determined using a vanishing point method.
  • the external and internal orientation parameters of the imaging devices are determined using planar homography.
  • the external orientation parameters of the imaging devices are determined using a three-point method.
  • each imaging device generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer of the acquired image.
  • the certainty value is used by the processor to determine pointer co-ordinate data to be used to determine the position of the pointer relative to the touch surface.
  • the processor ignores pointer co-ordinate data generated by the imaging device when the certainty value associated therewith is below a threshold level.
  • a camera-based touch system comprising a generally rectangular passive touch surface on which contacts are made using a pointer.
  • Camera devices are removably mounted adjacent at least two corners of the touch surface.
  • Each of the camera devices has a field of view looking across and back towards the touch surface and is disposed in front of the plane of the touch surface.
  • the fields of view of the camera devices overlap over the touch surface.
  • the camera devices acquire images of the touch surface.
  • a processor receives and processes the image data to determine the location of the pointer relative to the touch surface via triangulation.
  • an apparatus for detecting a pointer relative to a generally rectangular touch surface comprising at least two spaced imaging devices having overlapping fields of view encompassing the touch surface.
  • the imaging devices see the touch surface in three-dimensions as a perspective view with the perspective view at a minimum including the four corners of the touch surface.
  • the imaging devices acquire overlapping images from different locations.
  • a processor receives and processes image data generated by at least one of the imaging devices to determine the location of the pointer relative to the touch surface using triangulation.
  • the present invention provides advantages in that since the imaging devices see the touch surface in three-dimensions as a perspective view, the imaging devices see the entire touch surface as well as its surrounding area. As a result, during image processing it is not necessary to process the entire images captured by the imaging devices but rather only pixels corresponding to information within the boundaries of the touch surface. Noise and other aberrations occurring in areas outside of the touch surface can be disregarded.
  • the three-dimensional perspective of the imaging devices allows the apparatus to be automatically calibrated and calibrated on a continuing basis without the need for user intervention.
  • the three-dimensional perspective of the imaging devices allows (x, y, z) co-ordinates to be assigned to each pointer appearing within the fields of view of the imaging devices. Thus, the apparatus is able to disambiguate between multiple pointers contacting the touch surface.
  • the present invention also provides advantages in that since the imaging devices are portable, they can be used to turn basically any surface into a touch surface.
  • the imaging devices simply need to extend forwardly of the surface a sufficient distance so that their fields of view looking back and across the touch surface see the corners of the surface and are not obstructed by any bezel or framing surrounding the surface.
  • the use of portable imaging devices that see the touch surface in three-dimensions as a perspective view also supports arbitrarily large or curved touch surfaces.
  • FIG. 1 is a schematic illustration, partially in perspective, of an apparatus for detecting a pointer relative to a touch surface
  • FIG. 2 is a schematic front plan view of the apparatus of FIG. 1 ;
  • FIG. 3 is a schematic side view of the apparatus of FIG. 1 ;
  • FIG. 4 is a perspective view of an imaging assembly forming part of the apparatus of FIG. 1 ;
  • FIG. 5 is a schematic block diagram of the imaging assembly of FIG. 4 ;
  • FIG. 6 is a flow chart showing the steps performed during calibration of the apparatus of FIG. 1 ;
  • FIG. 7 is a flow chart showing the steps performed during triangulation of pointer data extracted from acquired images to determine the location of a pointer contact on the touch surface;
  • FIG. 8 a to 8 d show the number of pixel rows in a captured image that must be processed for different spacings between an imaging assembly and the plane of the touch surface;
  • FIG. 9 is a perspective view of another embodiment of apparatus for detecting a pointer relative to a touch surface
  • FIG. 10 is a front view of yet another embodiment of an apparatus for detecting a pointer relative to a touch surface
  • FIG. 11 is still yet another embodiment of an apparatus for detecting a pointer relative to a touch surface.
  • FIGS. 12 a to 12 c show different pointer contacts on the touch surface of the apparatus of FIG. 11 .
  • apparatus 50 is in the form of a touch system and includes a touch screen 52 having a touch surface 54 on which pointer contacts are to be made.
  • Touch screen 52 is a flat panel display such as for example, a plasma display, a high-definition television (HDTV), a liquid crystal display (LCD) or the like that presents an image on the touch surface 54 .
  • a bezel 56 surrounds the touch surface 54 .
  • a pair of imaging assemblies 58 is positioned adjacent the top corners of the touch screen 52 . The imaging assemblies 58 communicate with each other as well as with a computer 60 executing one or more application programs.
  • the computer 60 processes pointer co-ordinate data generated by the imaging assemblies 58 to determine the locations of pointer contacts made on the touch surface 54 and updates the image that is presented on the touch surface 54 in response to the pointer contacts. Pointer contacts on the touch surface 54 can therefore be recorded as writing or drawing or used to control execution of application programs executed by the computer 60 .
  • the imaging assembly 58 includes a housing 72 shaped complimentary to the corner of the touch screen 52 . Formations (not shown) on the housing 72 allow the imaging assembly 58 to be secured in position relative to the corner of the touch screen. The imaging assembly 58 can however be removed from the touch screen 52 making the imaging assembly 58 portable.
  • a digital camera 74 is accommodated by the housing 72 and is positioned in front of the plane of the touch surface 54 by a distance equal to approximately 2.5 cm.
  • This distance between the digital camera 74 and the plane of the touch surface 54 is selected so that the bezel 56 does not obstruct the field of view of the digital camera 74 allowing the digital camera to see the entire touch surface 54 while still permitting useful three-dimensional processing of image data.
  • the digital camera 74 is also positioned above the touch surface 54 by a distance equal to approximately 10 cm.
  • the digital camera 74 is aimed so that its field of view (FOV) looks back at and across the entire touch surface 54 .
  • FOV field of view
  • the digital camera 74 sees the touch surface 54 including its four corners in three-dimensions as a perspective view.
  • the optical axis of the digital camera 74 is in line with the top corner of the touch screen 52 and forms an angle equal to approximately 45 degrees with respect to the top edge of the touch screen 52 .
  • Housing 72 also accommodates a circuit board on which the digital camera 74 and image processing circuitry 80 are mounted as shown in FIG. 5 .
  • the image processing circuitry 80 includes a digital signal processor (DSP) 82 coupled to the digital camera 74 and to static random access memory (SRAM) 84 .
  • the DSP 82 also communicates with the computer 60 over a data bus 86 .
  • the digital camera 74 is a high-resolution CMOS digital camera having a 640 ⁇ 480 pixel array such as that manufactured by National Semiconductor under model No. LM9618 and an associated lens 88 .
  • the DSP 82 is of the type manufactured by Analog Devices Inc. under model No. Blackfin BF 533 and includes a feature that allows image data to be moved from the digital camera 74 to the SRAM 84 quickly without interrupting DSP processing.
  • each of the digital cameras 74 acquires an image including the pointer.
  • the DSP 82 of each imaging assembly 58 moves the image data from the digital camera 74 to the SRAM 84 and then compares the image data with a mathematical model describing the boundary of the touch surface 54 as seen by the digital camera 74 . This enables a pixel subset of the captured image including only relevant pixels to be processed.
  • the DSP 82 of each imaging assembly 58 in turn processes the pixel subset to determine the (x, y) position of the pointer within the captured image.
  • the imaging assemblies 58 in turn convey this pointer co-ordinate data to the computer 60 , which uses the pointer co-ordinate data to calculate the (X, Y, Z) location of the pointer contact on the touch surface 54 using triangulation.
  • the pointer location data is then either recorded by the computer 60 as writing or drawing or interpreted as a mouse or other event.
  • the touch system 50 In order to enable pointer contacts relative to the touch surface 54 to be calculated accurately using triangulation, the touch system 50 needs to be calibrated so that either or both imaging assemblies 58 know how a particular point in a captured image relates to a physical three-dimensional location on the touch surface 54 (the “scene”). During calibration, a transformation that establishes the relationship between any point in the three-dimensional scene that each imaging assembly 58 observes and any point in a captured two-dimensional image is established. This step is necessary, since when the touch system 50 is set up, a priori information concerning the locations and orientations of the digital cameras 74 with respect to a touch surface 54 are not known.
  • (x, y) are the co-ordinates of a point in a captured image corresponding to a point (X, Y, Z) in the three-dimensional scene;
  • (x 0 , y 0 ) are the co-ordinates of the principal point of the digital camera 74 , the location at which the optical axis of the digital camera meets the focal plane of the digital camera with the optical axis being approximately normal to the focal plane;
  • f is the digital camera focal length
  • R 1 , R 2 , R 3 are terms depending on point (X, Y, Z), the spatial location of the optical center of the digital camera (X 0 , Y 0 , Z 0 ) and the orientation angles ( ⁇ , ⁇ , ⁇ ) of the digital camera optical axis with respect to the three-dimensional co-ordinate system of the touch surface 54 .
  • each digital camera 74 is idealized as a very small aperture at the location of the digital camera's optical center (focal point), which is taken to be the position of the digital camera in three-dimensional space.
  • the three-dimensional nature of the digital cameras' view is important in that it allows the digital cameras to see over the bezels 56 , if the touch surface 54 is assumed to be planar allows the plane of the touch surface to be determined, allows a determination to be made at any point as to whether a pointer is in contact with the touch surface 54 or hovering above the touch surface 54 , and allows the position of the pointer relative to the touch surface 54 to be determined.
  • the above collinearity equations express that a point in a three-dimensional scene with co-ordinates (X, Y, Z) projects into a two-dimensional image at point (x, y).
  • the external orientation parameters (X 0 , Y 0 , Z 0 ) and ⁇ , ⁇ , ⁇ and the internal orientation parameters f, x 0 , y 0 and ⁇ x , ⁇ y of the digital cameras 74 need to be determined.
  • each lens 88 can be represented by terms relating specifically to both radial distortion and decentering distortion. Due to the relativity low quality of each lens 88 and captured image data, in the present embodiment, only the first-order radial distortion term is recorded. As a result the lens distortion terms can be expressed as:
  • ⁇ x ( x ⁇ x 0 ) K 1 (( x ⁇ x 0 ) 2 +( y ⁇ y 0 ) 2 )
  • ⁇ y ( y ⁇ y 0 ) K 1 (( x 1 ⁇ x 0 ) 2 +( y ⁇ y 0 ) 2 )
  • lens distortion can be summarized through parameter K 1 .
  • a self-calibration technique is used to calibrate the touch system 50 .
  • Self-calibration is based on the fact that if the three-dimensional positions of reference points on an object are known and the two-dimensional positions of the same points can be measured in one or more images of the object, these data completely specify the location of the imaging assembly capturing the image, the angular orientation of the imaging assembly and parameters relating to the lens of the imaging assembly.
  • the positions (X 0 , Y 0 , Z 0 ) of the digital cameras 74 in three-dimensional space may be measured in absolute units (e.g., centimeters) or in relative terms by assuming a unit of length corresponding to a reference length such as for example the shorter dimension of the touch surface 54 .
  • Each digital camera's angular orientation is represented by the three angles ⁇ , ⁇ , ⁇ allowing a rotation matrix R for each digital camera 74 to be defined.
  • the rotation matrix R describes the transformation between the co-ordinate system of the three-dimensional scene and that of the digital camera. Calculating the focal length f, principal point (x 0 , y 0 ), and lens distortion coefficient(s) for each digital camera 74 is not necessary if precise values for these digital camera parameters are known.
  • the touch surface 54 corresponds to the X-Y plane, and that the Z axis is pointing outward from the touch surface 54 at an angle generally normal to the plane of the touch surface.
  • image positions (x, y) corresponding to a number of scene points (X, Y, Z) are measured from an image, and the positions of the scene points (X, Y, Z) are known (e.g., in centimeters)
  • the collinearity equations may be set up for each point and solved using a least-squares technique to enable the external and internal orientation parameters to be determined.
  • the least-squares method is used due to the non-linear nature of the collinearity equation model.
  • eight (8) calibration points around the periphery of the touch surface 54 are chosen since by doing so yields sixteen (16) equations and ten (10) unknowns, which is sufficient for a good least-squares solution.
  • the corresponding image points are defined by either measuring the positions of a pointer at the calibration points captured in an image, or by measuring the positions of markers at the calibration points in a captured image.
  • the (X, Y, Z) positions of the calibration points on the touch surface 54 are determined (step 100 ). Specifically, the positions of the calibration points on the touch surface 54 are determined by measuring the positions of the touch surface corners and the mid-points of its side edges relative to one of the touch surface corners that is designated to be at the co-ordinate origin. Alternatively, if the aspect ratio of the touch surface 54 is known, then the short dimension of the touch surface can be taken as a unit of length, allowing the relative positions of the corners and side edge mid-points to be determined.
  • the (x, y) positions of the calibrations points in the images are then determined through image processing (step 102 ), with possible initial guess input from external image processing as will be described (step 104 ).
  • a calibration solution is then computed, including the internal orientation parameters if desired (step 106 ). If the computed solution is based on initial guess input, a least-squares refinement is computed (step 108 ).
  • FIG. 7 is a flow chart illustrating the general steps performed during triangulation of pointer co-ordinate data to determine pointer contact locations relative to the touch surface.
  • the DSPs 82 in turn output this pointer co-ordinate data to the computer 60 (step 120 ).
  • Each DSP 82 also outputs a pointer position certainty estimate representing the degree of certainty that the DSP 82 has positively identified the actual pointer in the captured image.
  • the computer 60 which receives pointer co-ordinate and certainty estimate data from the DSPs 82 of both imaging assemblies 58 (steps 122 and 124 ), makes a decision as to whether to use the pointer co-ordinate data returned by one or both imaging assemblies based on the certainty estimates (step 126 ). Generally, the pointer co-ordinate data generated by both DSPs 82 is used by the computer 60 .
  • the certainty estimate associated on the pointer co-ordinate data generated by one of the DSPs 82 is below a threshold level, in this case 50%, representing a low degree of certainty that the pointer co-ordinate data is accurate, that pointer co-ordinate data is ignored and not used.
  • Triangulation is then performed using the collinearity equations referred earlier either using the pointer co-ordinate data from both imaging assemblies 58 (step 128 ) or using the pointer co-ordinate data from one imaging assembly 58 (step 130 ). Since the collinearity equations relate image position (x, y) to spatial position (X, Y, Z), two (x, y) positions, one from each digital camera 74 , are necessary to compute a unique (X, Y, Z) spatial position for the pointer. This yields four equations and three unknowns. The collinearity equations are rearranged to produce a linear least-squares problem, making triangulation an efficient procedure. Since the results of the triangulation in this case yield an (X, Y, Z) spatial position for the pointer, multiple pointers appearing within the fields of view of the imaging assemblies 58 can be tracked separately thereby to provide pointer disambiguation.
  • the triangulation results can be refined using a non-linear least-squares technique if desired.
  • imaging assemblies 58 that see the entire touch surface 54 in three-dimensions as a perspective view as well as its surrounding area provides advantages. For example, during image processing, pointers crossing the boundaries of the touch surface 54 can be recognized prior to contact on the touch surface. This information can be used by the DSPs 82 to limit image processing to pixels within the relevant pixel subset adjacent the boundary crossover points.
  • the touch system 50 also provides shadow/object discrimination. Generally, as a pointer is brought towards the touch surface 54 , one of the imaging assemblies 58 will see the pointer before the other. The imaging assembly seeing the pointer first can provide pointer information to the other imaging assembly identifying the region of its relevant pixel subset that should be examined to locate the pointer. This helps to increase the probability of locating the pointer accurately and quickly. Planar homography is used to relate the two digital camera focal planes to one another, allowing the pointer information to be effectively exchanged between the imaging assemblies 58 .
  • Another approach is to make use of a different relationship between the views of the imaging assemblies, through an entity known as the fundamental matrix, or the closely-related essential matrix.
  • the fundamental matrix translates this point into a line in the other digital camera image.
  • it is only necessary to search an image along this line (known as an epipolar line) to locate the corresponding point.
  • This approach has an advantage in that it severely limits the search region in the second digital camera view and helps to eliminate false positives.
  • the imaging assemblies 58 are shown as being disposed in front of the plane of the touch surface 54 by a distance equal to approximately 2.5 cm. As mentioned, this distance ensures that the bezel 56 does not obstruct the imaging assemblies' views of the entire touch surface 54 .
  • the distances over which the imaging assemblies 58 are disposed in front of the plane of the touch surface 54 can of course vary depending on the environment although, the distances have an impact on the size of the relevant pixel subset corresponding to the boundary of the touch surface that must be processed.
  • the imaging assemblies 58 are positioned so that, at a minimum, the four corners of the touch surface 54 are within their fields of view. FIGS.
  • 8 a to 8 d show the number of pixel rows in a captured image that must be processed for different distances assuming a touch surface 54 having an 84 diagonal dimension and a 4:3 aspect ratio. As will be appreciated, as the distance increases so do the number of pixel rows that require processing in captured images.
  • the vanishing points for the touch surface 54 can be determined as described in the publication entitled “Geometric Computation for Machine Vison”, Oxford University Press, Oxford 1993, authored by Kanatani.
  • planar homography as described in the publication entitled “Multiple View Geometry in Computer Vision”, Cambridge University Press, Cambridge 2001, authored by Hartley et al. or the three-point method as described in the publication entitled “Minimum Solutions for Orientations in Calibration and Orientation of Cameras in Computer Vision”, Springer-Verlag, New York 2001, authored by Wrobel can be used to calibrate the touch system.
  • the vanishing points corresponding to the two sets of parallel peripheral side edges of the touch surface may be used to define the external orientation parameters completely.
  • the vanishing points are finite, i.e., they lie within the bounds of a digital camera image and serve to define the rotation matrix R. If a unit of length is assumed, the digital camera positions can then be determined, completing the external orientation parameter determination.
  • the vanishing points can also be used to define the digital camera's focal length and principal point as described in the publication entitled “On Estimating Rotations”, T. U.
  • the two vanishing points define a three-dimensional co-ordinate system for the touch system, from which everything else follows.
  • the determined external orientation parameters be refined using a least-squares method.
  • the planar homography calibration approach relates points on the plane of the touch surface 54 to points on the image plane of the digital camera. By measuring the positions of a number of image points corresponding to certain scene points, it is possible to define the nine components of the homography matrix. Once this is done, the homography matrix can be decomposed into the rotation matrix R and a vector representing the digital camera's position in three-dimensional space. Using this calibration method requires some assumptions about the digital camera's internal orientation to be made. The need to make these assumptions can however be avoided by rewriting the homography matrix as another matrix entity known as the image of the absolute conic as described in the previously mentioned Hartley et al. publication. This matrix entity provides direct estimates of the focal length and principal point of the digital camera through its singular value decomposition.
  • the three-point calibration method makes use of basic trigonometry and the fact that three points define a plane. If the locations and distances between three points in a three-dimensional scene are known and form a triangle, then the image positions of these points may be used to define angles between the points. This information is sufficient to solve for the distances of the three points from the digital camera's optical center, thus giving the digital camera's position in space. Subsequent processing of the image points then provides an estimate of the rotation matrix R. As this method gives an orientation from a minimal data set, it can be used to initialize a least-squares method for refining the orientation, and hence to provide the initial guess input at step 104 during calibration.
  • the orientation of each digital camera in space can be determined based on an entity known as a “quaternion”.
  • a quaternion is a four-element vector with certain properties as described in the previously mentioned Foerstner publication. Quaternion elements take on only values between ⁇ 1 and 1, with one of the elements being constrained to be 1. This avoids problems associated with abrupt changes in value and assists greatly in the convergence using a least-squares approach.
  • some angle changes create difficulty such as for example when an angle changes from 359 degrees to 360 degrees, which is the same as 0 degrees.
  • FIG. 9 shows another embodiment of an apparatus for detecting a pointer relative to a touch surface 54 .
  • both digital cameras 174 are accommodated within a single housing 172 adjacent its opposite ends.
  • the housing 172 overlies the top edge of the bezel 56 and spans the touch screen 52 so that the digital cameras 174 are positioned adjacent the top corners of the touch screen.
  • FIG. 10 shows a digital camera arrangement for a large scale touch system.
  • digital cameras C 1 to C 8 are provided along the top of the touch surface 254 and look back at and across the touch surface.
  • digital cameras C 1 and C 8 are located at the top left and top right corners of the touch surface 254 .
  • Intermediate pairs of digital cameras C 2 and C 3 , C 4 and C 5 and C 6 and C 7 are located at spaced locations along the top of the touch surface 254 .
  • the fields of view of the digital cameras are shown by the dotted lines. As can be seen, the fields of view of the cameras overlap so that each location on the touch surface 254 falls within the fields of view of at least two digital cameras. This of course allows a pointer to be tracked across the entire touch surface 254 using triangulation in the same manner described above.
  • FIG. 11 shows yet another digital camera arrangement for a large scale touch system.
  • evenly spaced digital cameras C 1 to C 7 are positioned above the top edge of the touch surface 354 and look back at and across the touch surface.
  • the fields of view of the digital cameras are shown by the dotted lines and as can be seen, the fields of view of the digital cameras overlap so that each location on the touch surface falls within the fields of view of at least two digital cameras. Again this allows a pointer to be tracked across the entire touch surface 354 using triangulation in the same manner described above.
  • most locations on the touch surface 354 fall within the fields of view of more than two digital cameras allowing multiple triangulation results to be generated for each pointer contact. Depending on the pointer contact locations, different logic can be used to select the triangulation results to be used to determine the pointer contact location.
  • the position of pointer P on touch surface 354 can be calculated by triangulating pointer information derived from images captured by digital cameras C 1 and C 2 and possibly by triangulating pointer information derived from images captured by digital camera C 3 .
  • pointer information derived from images captured by digital cameras C 1 and C 3 and digital cameras C 2 and C 3 can be triangulated resulting in multiple triangulation results.
  • the multiple triangulation results can be averaged or processed according to other logic to yield a single pointer position. If digital camera C 3 is deemed to be too far from the pointer P, the result from the digital camera C 3 can be ignored.
  • pointer information derived from images captured by digital camera C 3 can be used to track the pointer to determine when the pointer reaches a certain proximity to the digital camera C 3 .
  • the pointer information derived from images captured by digital camera C 3 can be triangulated to determine the position of the pointer on the touch surface 354 .
  • FIGS. 13 b and 13 c show other positions of pointers on the touch surface 354 and the various triangulation results that can be derived from images captured by the digital cameras.
  • the imaging assemblies are able to self-calibrate, the imaging assemblies can be affixed basically to any surface to convert that surface to a touch surface.
  • the touch system 50 is described as including a computer communicating with the DSPs of the imaging assemblies and processing the pointer co-ordinate data using triangulation to determine the position of the pointer relative to the touch surface, other processing architectures can of course be used.
  • the DSP of one of the imaging assemblies may serve as the processor responsible for triangulating the pointer co-ordinate data.

Abstract

An apparatus for detecting a pointer relative to a touch surface includes at least two spaced imaging assemblies having overlapping fields of view encompassing the touch surface. The imaging assemblies see the touch surface in three-dimensions as a perspective view. The imaging assemblies acquire overlapping images from different locations. A processor receives and processes image data generated by at least one of the imaging assemblies to determine the location of the pointer relative to the touch surface.

Description

  • This application is a continuation of U.S. patent application Ser. No. 10/836,536, filed May 5, 2004, now U.S. Pat. No. 7,492,357
  • FIELD OF THE INVENTION
  • The present invention relates generally to interactive input systems and in particular to an apparatus and method for detecting a pointer relative to a touch surface.
  • BACKGROUND OF THE INVENTION
  • Touch systems are well known in the art and typically include a touch screen having a touch surface on which contacts are made using a pointer in order to generate user input. Pointer contacts with the touch surface are detected and are used to generate corresponding output depending on areas of the touch surface where the pointer contacts are made. Common touch systems utilize analog resistive, electromagnetic, capacitive, acoustic or machine vision techniques to identify pointer contacts on the touch surface.
  • For example, International PCT Application No. PCT/CA01/00980 filed on Jul. 5, 2001 and published under No. WO 02/03316 on Jan. 10, 2002, assigned to SMART Technologies Inc., assignee of the present invention, discloses a camera-based touch system comprising a touch screen that includes a passive touch surface on which a computer-generated image is presented. A rectangular bezel or frame surrounds the touch surface and supports digital cameras at its corners. The digital cameras have overlapping fields of view that encompass and look generally across the plane of the touch surface. The digital cameras acquire images looking across the touch surface from different locations and generate image data. Image data acquired by the digital cameras is processed by digital signal processors to determine if a pointer exists in the captured image data. When it is determined that a pointer exists in the captured image data, the digital signal processors convey pointer characteristic data to a master controller, which in turn processes the pointer characteristic data to determine the location of the pointer in (x, y)-co-ordinates relative to the touch surface using triangulation. The pointer co-ordinate data is conveyed to a computer executing one or more applications programs. The computer uses the pointer co-ordinate data to update the computer-generated image that is presented on the touch surface. Pointer contacts on the touch surface can therefore be recorded as writing or drawing or used to control execution of applications programs executed by the computer.
  • Although the above touch system works extremely well, improvements in vision-based touch systems are continually being sought.
  • It is therefore an object of the present invention to provide a novel apparatus and method for detecting a pointer relative to a touch surface.
  • SUMMARY OF THE INVENTION
  • According to one aspect of the present invention there is provided an apparatus for detecting a pointer relative to a touch surface comprising at least two spaced imaging devices having overlapping fields of view encompassing the touch surface. The imaging devices see the touch surface in three-dimensions as a perspective view. The imaging devices acquire images from different locations. A processor receives and processes image data generated by at least one of the imaging devices to determine the location of the pointer relative to the touch surface.
  • Each imaging device is calibrated to establish the relationship between points (X, Y, Z) in its perspective view and points (x, y) in acquired images. Each imaging device generates pointer co-ordinate data when a pointer is captured in an acquired image. The processor triangulates the pointer co-ordinate data to determine the location of the pointer relative to the touch surface.
  • In one embodiment, the apparatus includes a pair of imaging devices with each imaging device being positioned adjacent a different corner of the touch surface. Each imaging device is spaced from and spaced in front of the touch surface. Each imaging device is positioned relative to the touch surface so that at a minimum the entire periphery of the touch surface is within its perspective view.
  • In one embodiment, during calibration, calibration points (X, Y, Z) on the touch surface and image points (x, y) corresponding to the calibration points are measured. Collinearity equations are solved using the measured calibration and image points to determine external and internal orientation parameters of the imaging devices. The collinearity equations are solved using a least-squares method. The calibrations points are at spaced location along the periphery of the touch surface such as at the corners and edge mid-points of the touch surface. In an alternative embodiment, the external orientation parameters of the imaging devices are determined using a vanishing point method. In yet another embodiment, the external and internal orientation parameters of the imaging devices are determined using planar homography. In still yet another embodiment, the external orientation parameters of the imaging devices are determined using a three-point method.
  • In one embodiment, each imaging device generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer of the acquired image. The certainty value is used by the processor to determine pointer co-ordinate data to be used to determine the position of the pointer relative to the touch surface. The processor ignores pointer co-ordinate data generated by the imaging device when the certainty value associated therewith is below a threshold level.
  • According to another aspect of the present invention there is provided a camera-based touch system comprising a generally rectangular passive touch surface on which contacts are made using a pointer. Camera devices are removably mounted adjacent at least two corners of the touch surface. Each of the camera devices has a field of view looking across and back towards the touch surface and is disposed in front of the plane of the touch surface. The fields of view of the camera devices overlap over the touch surface. The camera devices acquire images of the touch surface. A processor receives and processes the image data to determine the location of the pointer relative to the touch surface via triangulation.
  • According to yet another aspect of the present invention there is provided an apparatus for detecting a pointer relative to a generally rectangular touch surface comprising at least two spaced imaging devices having overlapping fields of view encompassing the touch surface. The imaging devices see the touch surface in three-dimensions as a perspective view with the perspective view at a minimum including the four corners of the touch surface. The imaging devices acquire overlapping images from different locations. A processor receives and processes image data generated by at least one of the imaging devices to determine the location of the pointer relative to the touch surface using triangulation.
  • The present invention provides advantages in that since the imaging devices see the touch surface in three-dimensions as a perspective view, the imaging devices see the entire touch surface as well as its surrounding area. As a result, during image processing it is not necessary to process the entire images captured by the imaging devices but rather only pixels corresponding to information within the boundaries of the touch surface. Noise and other aberrations occurring in areas outside of the touch surface can be disregarded. In addition, the three-dimensional perspective of the imaging devices allows the apparatus to be automatically calibrated and calibrated on a continuing basis without the need for user intervention. Furthermore, the three-dimensional perspective of the imaging devices allows (x, y, z) co-ordinates to be assigned to each pointer appearing within the fields of view of the imaging devices. Thus, the apparatus is able to disambiguate between multiple pointers contacting the touch surface.
  • The present invention also provides advantages in that since the imaging devices are portable, they can be used to turn basically any surface into a touch surface. The imaging devices simply need to extend forwardly of the surface a sufficient distance so that their fields of view looking back and across the touch surface see the corners of the surface and are not obstructed by any bezel or framing surrounding the surface. The use of portable imaging devices that see the touch surface in three-dimensions as a perspective view also supports arbitrarily large or curved touch surfaces.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • Embodiments of the present invention will now be described more fully with reference to the accompanying drawings in which:
  • FIG. 1 is a schematic illustration, partially in perspective, of an apparatus for detecting a pointer relative to a touch surface;
  • FIG. 2 is a schematic front plan view of the apparatus of FIG. 1;
  • FIG. 3 is a schematic side view of the apparatus of FIG. 1;
  • FIG. 4 is a perspective view of an imaging assembly forming part of the apparatus of FIG. 1;
  • FIG. 5 is a schematic block diagram of the imaging assembly of FIG. 4;
  • FIG. 6 is a flow chart showing the steps performed during calibration of the apparatus of FIG. 1;
  • FIG. 7 is a flow chart showing the steps performed during triangulation of pointer data extracted from acquired images to determine the location of a pointer contact on the touch surface;
  • FIG. 8 a to 8 d show the number of pixel rows in a captured image that must be processed for different spacings between an imaging assembly and the plane of the touch surface;
  • FIG. 9 is a perspective view of another embodiment of apparatus for detecting a pointer relative to a touch surface;
  • FIG. 10 is a front view of yet another embodiment of an apparatus for detecting a pointer relative to a touch surface;
  • FIG. 11 is still yet another embodiment of an apparatus for detecting a pointer relative to a touch surface; and
  • FIGS. 12 a to 12 c show different pointer contacts on the touch surface of the apparatus of FIG. 11.
  • DETAILED DESCRIPTION OF THE EMBODIMENTS
  • Turning now to FIGS. 1 to 3, an apparatus for detecting a pointer relative to a touch surface is shown and is generally identified by reference numeral 50. In this embodiment, apparatus 50 is in the form of a touch system and includes a touch screen 52 having a touch surface 54 on which pointer contacts are to be made. Touch screen 52 is a flat panel display such as for example, a plasma display, a high-definition television (HDTV), a liquid crystal display (LCD) or the like that presents an image on the touch surface 54. A bezel 56 surrounds the touch surface 54. A pair of imaging assemblies 58 is positioned adjacent the top corners of the touch screen 52. The imaging assemblies 58 communicate with each other as well as with a computer 60 executing one or more application programs. The computer 60 processes pointer co-ordinate data generated by the imaging assemblies 58 to determine the locations of pointer contacts made on the touch surface 54 and updates the image that is presented on the touch surface 54 in response to the pointer contacts. Pointer contacts on the touch surface 54 can therefore be recorded as writing or drawing or used to control execution of application programs executed by the computer 60.
  • Turning now to FIG. 4, one of the imaging assemblies 58 is better illustrated. As can be seen, the imaging assembly 58 includes a housing 72 shaped complimentary to the corner of the touch screen 52. Formations (not shown) on the housing 72 allow the imaging assembly 58 to be secured in position relative to the corner of the touch screen. The imaging assembly 58 can however be removed from the touch screen 52 making the imaging assembly 58 portable. A digital camera 74 is accommodated by the housing 72 and is positioned in front of the plane of the touch surface 54 by a distance equal to approximately 2.5 cm. This distance between the digital camera 74 and the plane of the touch surface 54 is selected so that the bezel 56 does not obstruct the field of view of the digital camera 74 allowing the digital camera to see the entire touch surface 54 while still permitting useful three-dimensional processing of image data. The digital camera 74 is also positioned above the touch surface 54 by a distance equal to approximately 10 cm. The digital camera 74 is aimed so that its field of view (FOV) looks back at and across the entire touch surface 54. Thus, the digital camera 74 sees the touch surface 54 including its four corners in three-dimensions as a perspective view. The optical axis of the digital camera 74 is in line with the top corner of the touch screen 52 and forms an angle equal to approximately 45 degrees with respect to the top edge of the touch screen 52.
  • Housing 72 also accommodates a circuit board on which the digital camera 74 and image processing circuitry 80 are mounted as shown in FIG. 5. As can be seen, the image processing circuitry 80 includes a digital signal processor (DSP) 82 coupled to the digital camera 74 and to static random access memory (SRAM) 84. The DSP 82 also communicates with the computer 60 over a data bus 86. The digital camera 74 is a high-resolution CMOS digital camera having a 640×480 pixel array such as that manufactured by National Semiconductor under model No. LM9618 and an associated lens 88. The DSP 82 is of the type manufactured by Analog Devices Inc. under model No. Blackfin BF 533 and includes a feature that allows image data to be moved from the digital camera 74 to the SRAM 84 quickly without interrupting DSP processing.
  • The general operation of the touch system 50 will firstly be described. When a pointer is brought into the fields of view of the digital cameras 74 and contacts the touch surface 54, each of the digital cameras 74 acquires an image including the pointer. The DSP 82 of each imaging assembly 58 moves the image data from the digital camera 74 to the SRAM 84 and then compares the image data with a mathematical model describing the boundary of the touch surface 54 as seen by the digital camera 74. This enables a pixel subset of the captured image including only relevant pixels to be processed. The DSP 82 of each imaging assembly 58 in turn processes the pixel subset to determine the (x, y) position of the pointer within the captured image. The imaging assemblies 58 in turn convey this pointer co-ordinate data to the computer 60, which uses the pointer co-ordinate data to calculate the (X, Y, Z) location of the pointer contact on the touch surface 54 using triangulation. The pointer location data is then either recorded by the computer 60 as writing or drawing or interpreted as a mouse or other event.
  • In order to enable pointer contacts relative to the touch surface 54 to be calculated accurately using triangulation, the touch system 50 needs to be calibrated so that either or both imaging assemblies 58 know how a particular point in a captured image relates to a physical three-dimensional location on the touch surface 54 (the “scene”). During calibration, a transformation that establishes the relationship between any point in the three-dimensional scene that each imaging assembly 58 observes and any point in a captured two-dimensional image is established. This step is necessary, since when the touch system 50 is set up, a priori information concerning the locations and orientations of the digital cameras 74 with respect to a touch surface 54 are not known.
  • The relationship between a point in a three-dimensional scene and its two-dimensional position in a captured image is summarized according to the collinearity equations as follows:
  • x = x 0 - Δ x - fR 1 R 3 y = y 0 - Δ y - fR 2 R 3
  • where:
  • (x, y) are the co-ordinates of a point in a captured image corresponding to a point (X, Y, Z) in the three-dimensional scene;
  • (x0, y0) are the co-ordinates of the principal point of the digital camera 74, the location at which the optical axis of the digital camera meets the focal plane of the digital camera with the optical axis being approximately normal to the focal plane;
  • x, Δy) represent distortion terms introduced due to the imperfect nature of the lens 88;
  • f is the digital camera focal length; and
  • (R1, R2, R3) are terms depending on point (X, Y, Z), the spatial location of the optical center of the digital camera (X0, Y0, Z0) and the orientation angles (ω, φ, κ) of the digital camera optical axis with respect to the three-dimensional co-ordinate system of the touch surface 54.
  • The above collinearity equations represent a pinhole model. Thus, each digital camera 74 is idealized as a very small aperture at the location of the digital camera's optical center (focal point), which is taken to be the position of the digital camera in three-dimensional space. The three-dimensional nature of the digital cameras' view is important in that it allows the digital cameras to see over the bezels 56, if the touch surface 54 is assumed to be planar allows the plane of the touch surface to be determined, allows a determination to be made at any point as to whether a pointer is in contact with the touch surface 54 or hovering above the touch surface 54, and allows the position of the pointer relative to the touch surface 54 to be determined.
  • The above collinearity equations express that a point in a three-dimensional scene with co-ordinates (X, Y, Z) projects into a two-dimensional image at point (x, y). In order to establish the transformation using the collinearity equations, the external orientation parameters (X0, Y0, Z0) and ω, φ, κ and the internal orientation parameters f, x0, y0 and Δx, Δy of the digital cameras 74 need to be determined.
  • The distortion of each lens 88 can be represented by terms relating specifically to both radial distortion and decentering distortion. Due to the relativity low quality of each lens 88 and captured image data, in the present embodiment, only the first-order radial distortion term is recorded. As a result the lens distortion terms can be expressed as:

  • Δx=(x−x 0)K 1((x−x 0)2+(y−y 0)2)

  • Δy=(y−y 0)K 1((x 1 −x 0)2+(y−y 0)2)
  • Thus, lens distortion can be summarized through parameter K1.
  • As will be appreciated, ten (10) parameters for each digital camera 74 need to be determined from the collinearity equations to calibrate each digital camera, namely:

  • X0, Y0, Z0, ω, φ, κ, f, x0, y0, K1
  • In the present embodiment, a self-calibration technique is used to calibrate the touch system 50. Self-calibration is based on the fact that if the three-dimensional positions of reference points on an object are known and the two-dimensional positions of the same points can be measured in one or more images of the object, these data completely specify the location of the imaging assembly capturing the image, the angular orientation of the imaging assembly and parameters relating to the lens of the imaging assembly.
  • The positions (X0, Y0, Z0) of the digital cameras 74 in three-dimensional space may be measured in absolute units (e.g., centimeters) or in relative terms by assuming a unit of length corresponding to a reference length such as for example the shorter dimension of the touch surface 54. Each digital camera's angular orientation is represented by the three angles ω, φ, κ allowing a rotation matrix R for each digital camera 74 to be defined. The rotation matrix R describes the transformation between the co-ordinate system of the three-dimensional scene and that of the digital camera. Calculating the focal length f, principal point (x0, y0), and lens distortion coefficient(s) for each digital camera 74 is not necessary if precise values for these digital camera parameters are known.
  • During self-calibration, it is assumed that the touch surface 54 corresponds to the X-Y plane, and that the Z axis is pointing outward from the touch surface 54 at an angle generally normal to the plane of the touch surface. If image positions (x, y) corresponding to a number of scene points (X, Y, Z) are measured from an image, and the positions of the scene points (X, Y, Z) are known (e.g., in centimeters), then the collinearity equations may be set up for each point and solved using a least-squares technique to enable the external and internal orientation parameters to be determined. The least-squares method is used due to the non-linear nature of the collinearity equation model.
  • In the present embodiment, eight (8) calibration points around the periphery of the touch surface 54 are chosen since by doing so yields sixteen (16) equations and ten (10) unknowns, which is sufficient for a good least-squares solution. In particular, the four corners and the mid-points along each side edge of the touch surface 54 are selected as the calibration points since the (X, Y) positions at these calibration points are easy to measure, provide reproducible calibration points, are easily located by users and at each of these (X, Y) positions Z=0 cm. The corresponding image points are defined by either measuring the positions of a pointer at the calibration points captured in an image, or by measuring the positions of markers at the calibration points in a captured image.
  • Turning now to FIG. 6, a flow chart illustrating the general steps performed during self-calibration is shown. Initially the (X, Y, Z) positions of the calibration points on the touch surface 54 are determined (step 100). Specifically, the positions of the calibration points on the touch surface 54 are determined by measuring the positions of the touch surface corners and the mid-points of its side edges relative to one of the touch surface corners that is designated to be at the co-ordinate origin. Alternatively, if the aspect ratio of the touch surface 54 is known, then the short dimension of the touch surface can be taken as a unit of length, allowing the relative positions of the corners and side edge mid-points to be determined. The (x, y) positions of the calibrations points in the images are then determined through image processing (step 102), with possible initial guess input from external image processing as will be described (step 104). A calibration solution is then computed, including the internal orientation parameters if desired (step 106). If the computed solution is based on initial guess input, a least-squares refinement is computed (step 108).
  • With the touch system 50 calibrated, the three-dimensional position of a pointer above or in contact with the touch surface 54 can be calculated via triangulation. During triangulation it is assumed that all of the camera orientation parameters are known and the pointer position (X, Y, Z) is to be determined given corresponding (x, y) measurements from the image captured either by one or both digital cameras 74. FIG. 7 is a flow chart illustrating the general steps performed during triangulation of pointer co-ordinate data to determine pointer contact locations relative to the touch surface. When images are captured by the imaging assemblies 58 including a pointer, the image data is processed by the DSPs 82 to determine the position of the pointer in each captured image in (x, y) co-ordinates. The DSPs 82 in turn output this pointer co-ordinate data to the computer 60 (step 120). Each DSP 82 also outputs a pointer position certainty estimate representing the degree of certainty that the DSP 82 has positively identified the actual pointer in the captured image. The computer 60, which receives pointer co-ordinate and certainty estimate data from the DSPs 82 of both imaging assemblies 58 (steps 122 and 124), makes a decision as to whether to use the pointer co-ordinate data returned by one or both imaging assemblies based on the certainty estimates (step 126). Generally, the pointer co-ordinate data generated by both DSPs 82 is used by the computer 60. If however, the certainty estimate associated on the pointer co-ordinate data generated by one of the DSPs 82 is below a threshold level, in this case 50%, representing a low degree of certainty that the pointer co-ordinate data is accurate, that pointer co-ordinate data is ignored and not used.
  • Triangulation is then performed using the collinearity equations referred earlier either using the pointer co-ordinate data from both imaging assemblies 58 (step 128) or using the pointer co-ordinate data from one imaging assembly 58 (step 130). Since the collinearity equations relate image position (x, y) to spatial position (X, Y, Z), two (x, y) positions, one from each digital camera 74, are necessary to compute a unique (X, Y, Z) spatial position for the pointer. This yields four equations and three unknowns. The collinearity equations are rearranged to produce a linear least-squares problem, making triangulation an efficient procedure. Since the results of the triangulation in this case yield an (X, Y, Z) spatial position for the pointer, multiple pointers appearing within the fields of view of the imaging assemblies 58 can be tracked separately thereby to provide pointer disambiguation.
  • When performing triangulation using pointer co-ordinate data from a single imaging assembly 58, it is assumed Z=0 (cm). In this case, one of the unknowns in the collinearity equations is eliminated. In other words, spatial position (X, Y) is determined from image position (x, y). Using images from a single imaging assembly 58 provides advantages in that the touch system 50 can still determine pointer contacts with the touch surface 54 even in instances where one of the imaging assemblies 58 is unable to see the pointer.
  • Once the triangulation results are available, the triangulation results can be refined using a non-linear least-squares technique if desired.
  • The use of imaging assemblies 58 that see the entire touch surface 54 in three-dimensions as a perspective view as well as its surrounding area provides advantages. For example, during image processing, pointers crossing the boundaries of the touch surface 54 can be recognized prior to contact on the touch surface. This information can be used by the DSPs 82 to limit image processing to pixels within the relevant pixel subset adjacent the boundary crossover points. The touch system 50 also provides shadow/object discrimination. Generally, as a pointer is brought towards the touch surface 54, one of the imaging assemblies 58 will see the pointer before the other. The imaging assembly seeing the pointer first can provide pointer information to the other imaging assembly identifying the region of its relevant pixel subset that should be examined to locate the pointer. This helps to increase the probability of locating the pointer accurately and quickly. Planar homography is used to relate the two digital camera focal planes to one another, allowing the pointer information to be effectively exchanged between the imaging assemblies 58.
  • Another approach is to make use of a different relationship between the views of the imaging assemblies, through an entity known as the fundamental matrix, or the closely-related essential matrix. Here, if the location of a point is known in one digital camera view, the fundamental matrix translates this point into a line in the other digital camera image. Thus, it is only necessary to search an image along this line (known as an epipolar line) to locate the corresponding point. This approach has an advantage in that it severely limits the search region in the second digital camera view and helps to eliminate false positives.
  • In the above-described embodiment, the imaging assemblies 58 are shown as being disposed in front of the plane of the touch surface 54 by a distance equal to approximately 2.5 cm. As mentioned, this distance ensures that the bezel 56 does not obstruct the imaging assemblies' views of the entire touch surface 54. The distances over which the imaging assemblies 58 are disposed in front of the plane of the touch surface 54 can of course vary depending on the environment although, the distances have an impact on the size of the relevant pixel subset corresponding to the boundary of the touch surface that must be processed. The imaging assemblies 58 are positioned so that, at a minimum, the four corners of the touch surface 54 are within their fields of view. FIGS. 8 a to 8 d show the number of pixel rows in a captured image that must be processed for different distances assuming a touch surface 54 having an 84 diagonal dimension and a 4:3 aspect ratio. As will be appreciated, as the distance increases so do the number of pixel rows that require processing in captured images.
  • Although a self-calibration technique has been described, other techniques can be used to calibrate the touch system 50. For example, the vanishing points for the touch surface 54 can be determined as described in the publication entitled “Geometric Computation for Machine Vison”, Oxford University Press, Oxford 1993, authored by Kanatani. Alternatively planar homography as described in the publication entitled “Multiple View Geometry in Computer Vision”, Cambridge University Press, Cambridge 2001, authored by Hartley et al. or the three-point method as described in the publication entitled “Minimum Solutions for Orientations in Calibration and Orientation of Cameras in Computer Vision”, Springer-Verlag, New York 2001, authored by Wrobel can be used to calibrate the touch system.
  • During the vanishing point method, advantage of the fact that the touch surface 54 is generally rectangular in shape is taken during computation of the external orientation parameters. In this case, by finding the peripheral edges of the touch surface 54 in a digital camera image, the vanishing points corresponding to the two sets of parallel peripheral side edges of the touch surface may be used to define the external orientation parameters completely. In the touch system, the vanishing points are finite, i.e., they lie within the bounds of a digital camera image and serve to define the rotation matrix R. If a unit of length is assumed, the digital camera positions can then be determined, completing the external orientation parameter determination. The vanishing points can also be used to define the digital camera's focal length and principal point as described in the publication entitled “On Estimating Rotations”, T. U. Munchen, 1999, authored by Foerstner. As will be appreciated, the two vanishing points define a three-dimensional co-ordinate system for the touch system, from which everything else follows. When using this approach it is preferred that the determined external orientation parameters be refined using a least-squares method.
  • The planar homography calibration approach relates points on the plane of the touch surface 54 to points on the image plane of the digital camera. By measuring the positions of a number of image points corresponding to certain scene points, it is possible to define the nine components of the homography matrix. Once this is done, the homography matrix can be decomposed into the rotation matrix R and a vector representing the digital camera's position in three-dimensional space. Using this calibration method requires some assumptions about the digital camera's internal orientation to be made. The need to make these assumptions can however be avoided by rewriting the homography matrix as another matrix entity known as the image of the absolute conic as described in the previously mentioned Hartley et al. publication. This matrix entity provides direct estimates of the focal length and principal point of the digital camera through its singular value decomposition.
  • The three-point calibration method makes use of basic trigonometry and the fact that three points define a plane. If the locations and distances between three points in a three-dimensional scene are known and form a triangle, then the image positions of these points may be used to define angles between the points. This information is sufficient to solve for the distances of the three points from the digital camera's optical center, thus giving the digital camera's position in space. Subsequent processing of the image points then provides an estimate of the rotation matrix R. As this method gives an orientation from a minimal data set, it can be used to initialize a least-squares method for refining the orientation, and hence to provide the initial guess input at step 104 during calibration.
  • Although the above-described calibration techniques make use of three angles to define the orientation of each digital camera 74 in space through a rotation matrix R, alternatives are available. For example rather than defining the rotation matrix R, the orientation of each digital camera in space can be determined based on an entity known as a “quaternion”. A quaternion is a four-element vector with certain properties as described in the previously mentioned Foerstner publication. Quaternion elements take on only values between −1 and 1, with one of the elements being constrained to be 1. This avoids problems associated with abrupt changes in value and assists greatly in the convergence using a least-squares approach. As will be appreciated, when measuring angles, some angle changes create difficulty such as for example when an angle changes from 359 degrees to 360 degrees, which is the same as 0 degrees.
  • FIG. 9 shows another embodiment of an apparatus for detecting a pointer relative to a touch surface 54. In this embodiment, both digital cameras 174 are accommodated within a single housing 172 adjacent its opposite ends. The housing 172 overlies the top edge of the bezel 56 and spans the touch screen 52 so that the digital cameras 174 are positioned adjacent the top corners of the touch screen.
  • Imaging assemblies that see the touch surface in three-dimensions as a perspective view can also be used in conjunction with large scale touch systems such as those described in U.S. patent application Ser. No. (not available) filed on Jan. 2, 2004 to Hill et al., assigned to SMART Technologies Inc., assignee of the present invention, the content of which is incorporated herein by reference. For example, FIG. 10 shows a digital camera arrangement for a large scale touch system. In this embodiment, digital cameras C1 to C8 are provided along the top of the touch surface 254 and look back at and across the touch surface. In particular, digital cameras C1 and C8 are located at the top left and top right corners of the touch surface 254. Intermediate pairs of digital cameras C2 and C3, C4 and C5 and C6 and C7 are located at spaced locations along the top of the touch surface 254. The fields of view of the digital cameras are shown by the dotted lines. As can be seen, the fields of view of the cameras overlap so that each location on the touch surface 254 falls within the fields of view of at least two digital cameras. This of course allows a pointer to be tracked across the entire touch surface 254 using triangulation in the same manner described above.
  • FIG. 11 shows yet another digital camera arrangement for a large scale touch system. In this embodiment, evenly spaced digital cameras C1 to C7 are positioned above the top edge of the touch surface 354 and look back at and across the touch surface. The fields of view of the digital cameras are shown by the dotted lines and as can be seen, the fields of view of the digital cameras overlap so that each location on the touch surface falls within the fields of view of at least two digital cameras. Again this allows a pointer to be tracked across the entire touch surface 354 using triangulation in the same manner described above. In this embodiment, most locations on the touch surface 354 fall within the fields of view of more than two digital cameras allowing multiple triangulation results to be generated for each pointer contact. Depending on the pointer contact locations, different logic can be used to select the triangulation results to be used to determine the pointer contact location.
  • For example, as shown in FIG. 12 a, the position of pointer P on touch surface 354 can be calculated by triangulating pointer information derived from images captured by digital cameras C1 and C2 and possibly by triangulating pointer information derived from images captured by digital camera C3. In this latter case pointer information derived from images captured by digital cameras C1 and C3 and digital cameras C2 and C3 can be triangulated resulting in multiple triangulation results. The multiple triangulation results can be averaged or processed according to other logic to yield a single pointer position. If digital camera C3 is deemed to be too far from the pointer P, the result from the digital camera C3 can be ignored. Alternatively, pointer information derived from images captured by digital camera C3 can be used to track the pointer to determine when the pointer reaches a certain proximity to the digital camera C3. When the pointer reaches a certain proximity to the digital camera C3, the pointer information derived from images captured by digital camera C3 can be triangulated to determine the position of the pointer on the touch surface 354.
  • FIGS. 13 b and 13 c show other positions of pointers on the touch surface 354 and the various triangulation results that can be derived from images captured by the digital cameras.
  • By pairing the digital cameras, curved and non-planar touch surfaces can be supported since the various pairs of digital cameras need only be responsible for viewing a portion of the touch surface.
  • As will be appreciated, since the imaging assemblies are able to self-calibrate, the imaging assemblies can be affixed basically to any surface to convert that surface to a touch surface.
  • Although the touch system 50 is described as including a computer communicating with the DSPs of the imaging assemblies and processing the pointer co-ordinate data using triangulation to determine the position of the pointer relative to the touch surface, other processing architectures can of course be used. For example, the DSP of one of the imaging assemblies may serve as the processor responsible for triangulating the pointer co-ordinate data.
  • Although preferred embodiments of the present invention have been described, those of skill in the art will appreciate that variations and modifications may be made without departing from the spirit and scope thereof as defined by the appended claims.

Claims (53)

1. An interactive input system comprising:
at least two imaging devices having overlapping fields of view of an input surface, each of said imaging devices comprising a lens having at least one form of distortion; and
processing structure determining at least one distortion parameter of each said lens during calibration.
2. An interactive input system according to claim 1 wherein said at least one form of distortion is radial distortion.
3. An interactive input system according to claim 1 wherein said at least one form of distortion is decentering distortion.
4. An interactive input system according to claim 1 wherein said at least one form of distortion is radial and decentering distortion.
5. An interactive input system according to claim 1 wherein the distortion of each said lens is expressed by:

Δx=(x−x 0)K 1((x−x 0)2+(y−y 0)2)

Δy=(y−y 0)K 1((x 1 −x 0)2+(y−y 0)2)
where:
x, Δy) represent radial and decentering distortion of the lens;
(x, y) are the co-ordinates of a point in an image captured by the imaging device corresponding to a point (X, Y, Z) in a three-dimensional scene;
(x0, y0) are the co-ordinates of the principal point of the imaging device, the location at which the optical axis of the imaging device meets the focal plane of the imaging device with the optical axis being approximately normal to the focal plane; and
K1 is a lens parameter.
6. An interactive input system according to claim 1 wherein said processing structure determines said at least one distortion parameter of each said lens during a self-calibration routine.
7. An interactive input system according to claim 6 wherein said at least one form of distortion is radial distortion.
8. An interactive input system according to claim 6 wherein said at least one form of distortion is decentering distortion.
9. An interactive input system according to claim 6 wherein said at least one form of distortion is radial and decentering distortion.
10. An interactive input system according to claim 6 wherein the distortion of each said lens is expressed by:

Δx=(x−x 0)K 1((x−x 0)2+(y−y 0)2)

Δy=(y−y 0)K 1((x 1 −x 0)2+(y−y 0)2)
where:
x, Δy) represent radial and decentering distortion of the lens;
(x, y) are the co-ordinates of a point in an image captured by the imaging device corresponding to a point (X, Y, Z) in a three-dimensional scene;
(x0, y0) are the co-ordinates of the principal point of the imaging device, the location at which the optical axis of the imaging device meets the focal plane of the imaging device with the optical axis being approximately normal to the focal plane; and
K1 is a lens parameter.
11. An interactive input system according to claim 1 wherein said input surface is one of planar, curved and non-planar.
12. An interactive input system according to claim 11 wherein said processing structure determines said at least one distortion parameter of each said lens during a self-calibration routine.
13. An interactive input system according to claim 12 wherein said at least one form of distortion is radial distortion.
14. An interactive input system according to claim 12 wherein said at least one form of distortion is decentering distortion.
15. An interactive input system according to claim 12 wherein said at least one form of distortion is radial and decentering distortion.
16. An interactive input system according to claim 12 wherein the distortion of each said lens is expressed by:

Δx=(x−x 0)K 1((x−x 0)2+(y−y 0)2)

Δy=(y−y 0)K 1((x 1 −x 0)2+(y−y 0)2)
where:
x, Δy) represent radial and decentering distortion of the lens;
(x, y) are the co-ordinates of a point in an image captured by the imaging device corresponding to a point (X, Y, Z) in a three-dimensional scene;
(x0, y0) are the co-ordinates of the principal point of the imaging device, the location at which the optical axis of the imaging device meets the focal plane of the imaging device with the optical axis being approximately normal to the focal plane; and
K1 is a lens parameter.
17. An interactive input system according to claim 1 wherein said input surface is any surface within the overlapping fields of view of said imaging devices.
18. An interactive input system according to claim 1 wherein each of said imaging devices sees said input surface in three-dimensions as a perspective view and wherein said processing structure further processes image data generated by at least one of said imaging devices to determine the location of a pointer relative to said input surface.
19. An interactive input system according to claim 18 wherein each imaging device is calibrated to establish the relationship between points (X, Y, Z) in its perspective view and points (x, y) in acquired images, each imaging device generating pointer co-ordinate data when a pointer is captured in an acquired image.
20. An interactive input system according to claim 19 wherein said processing structure triangulates the pointer co-ordinate data to determine the location of the pointer relative to said input surface.
21. An interactive input system according to claim 20 wherein each imaging device is positioned relative to said input surface so that at a minimum the entire periphery of the input surface is within its field of view.
22. An apparatus according to claim 21 wherein said input surface is bordered by a bezel.
23. An interactive input system according to claim 20 wherein said processing structure determines said at least one distortion parameter of each said lens during a self-calibration routine.
24. An interactive input system according to claim 23 wherein said at least one form of distortion is radial distortion.
25. An interactive input system according to claim 23 wherein said at least one form of distortion is decentering distortion.
26. An interactive input system according to claim 23 wherein said at least one form of distortion is radial and decentering distortion.
27. An interactive input system according to claim 23 wherein the distortion of each said lens is expressed by:

Δx=(x−x 0)K 1((x−x 0)2+(y−y 0)2)

Δy=(y−y 0)K 1((x 1 −x 0)2+(y−y 0)2)
where:
x, Δy) represent radial and decentering distortion of the lens;
(x, y) are the co-ordinates of a point in an image captured by the imaging device corresponding to a point (X, Y, Z) in the three-dimensional scene;
(x0, y0) are the co-ordinates of the principal point of the imaging device, the location at which the optical axis of the imaging device meets the focal plane of the imaging device with the optical axis being approximately normal to the focal plane; and
K1 is a lens parameter.
28. An interactive input system according to claim 23 wherein said input surface is one of planar, curved and non-planar.
29. An interactive input system according to claim 20 wherein each imaging device generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer in the acquired image.
30. An interactive input system according to claim 29 wherein said certainty value is used by said processing structure to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said input surface.
31. An interactive input system according to claim 30 wherein said processing structure ignores pointer co-ordinate data generated by said imaging device when the certainty value associated therewith is below a threshold level.
32. An interactive input system according to claim 20 wherein the imaging device that detects a pointer in its acquired image first communicates data to the other imaging device to assist that imaging device to detect the pointer in its acquired image.
33. An interactive input system according to claim 32 wherein each imaging device also generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer in the acquired image.
34. An interactive input system according to claim 33 wherein said certainty value is used by said processing structure to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said input surface.
35. An interactive input system according to claim 34 wherein said processing structure ignores pointer co-ordinate data generated by said imaging device when the certainty value associated therewith is below a threshold level.
36. An interactive input system according to claim 20 wherein each imaging device processes a subset of pixels in each acquired image.
37. An interactive input system according to claim 17 wherein said imaging devices are portable.
38. A camera-based interactive input system comprising:
a touch surface on which contacts are made using a pointer;
camera devices looking at said touch surface from different vantages and having overlapping fields, said camera devices acquiring images of said touch surface, each of said camera devices having an imperfect lens; and
processing structure receiving and processing said image data to determine the location of said pointer relative to said touch surface via triangulation, said processing structure compensating for image distortion as a result of the imperfect lens of each said camera device.
39. A camera-based interactive input system according to claim 38 wherein said processing structure determines at least one distortion parameter of each said lens during a self-calibration routine.
40. A camera-based interactive input system according to claim 39 wherein said at least one form of distortion is radial distortion.
41. A camera-based interactive input system according to claim 40 wherein said at least one form of distortion is decentering distortion.
42. A camera-based interactive input system according to claim 40 wherein said at least one form of distortion is radial and decentering distortion.
43. A camera-based interactive input system according to claim 40 wherein the distortion of each said lens is expressed by:

Δx=(x−x 0)K 1((x−x 0)2+(y−y 0)2)

Δy=(y−y 0)K 1((x 1 −x 0)2+(y−y 0)2)
where:
x, Δy) represent radial and decentering distortion of the lens;
(x, y) are the co-ordinates of a point in an image captured by the camera device corresponding to a point (X, Y, Z) in a three-dimensional scene;
(x0, y0) are the co-ordinates of the principal point of the camera device, the location at which the optical axis of the camera device meets the focal plane of the camera device with the optical axis being approximately normal to the focal plane; and
K1 is a lens parameter.
44. A camera-based interactive input system according to claim 39 wherein said touch surface is one of planar, curved and non-planar.
45. A camera-based interactive input system according to claim 39 wherein each camera device generates a certainty value representing the degree of certainty that the camera device has positively identified the pointer in the acquired image.
46. A camera-based interactive input system according to claim 45 wherein said certainty value is used by said processing structure to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface.
47. A camera-based interactive input system according to claim 46 wherein said processing structure ignores pointer co-ordinate data generated by said camera device when the certainty value associated therewith is below a threshold level.
48. A camera-based interactive input system according to claim 39 wherein the camera device that detects a pointer in its acquired image first communicates data to the other camera device to assist that camera device to detect the pointer in its acquired image.
49. A camera-based interactive input system according to claim 48 wherein each camera device also generates a certainty value representing the degree of certainty that the camera device has positively identified the pointer in the acquired image.
50. A camera-based interactive input system according to claim 49 wherein said certainty value is used by said processing structure to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface.
51. A camera-based interactive input system according to claim 50 wherein said processing structure ignores pointer co-ordinate data generated by said camera device when the certainty value associated therewith is below a threshold level.
52. A camera-based interactive input system according to claim 39 wherein each camera device processes a subset of pixels in each acquired image.
53. A camera-based interactive input system according to claim 44 wherein said camera devices are portable.
US12/369,904 2004-05-05 2009-02-12 Apparatus and method for detecting a pointer relative to a touch surface Abandoned US20090146972A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/369,904 US20090146972A1 (en) 2004-05-05 2009-02-12 Apparatus and method for detecting a pointer relative to a touch surface

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/838,536 US7492357B2 (en) 2004-05-05 2004-05-05 Apparatus and method for detecting a pointer relative to a touch surface
US12/369,904 US20090146972A1 (en) 2004-05-05 2009-02-12 Apparatus and method for detecting a pointer relative to a touch surface

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/838,536 Continuation US7492357B2 (en) 2004-05-05 2004-05-05 Apparatus and method for detecting a pointer relative to a touch surface

Publications (1)

Publication Number Publication Date
US20090146972A1 true US20090146972A1 (en) 2009-06-11

Family

ID=35239003

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/838,536 Active 2025-11-25 US7492357B2 (en) 2004-05-05 2004-05-05 Apparatus and method for detecting a pointer relative to a touch surface
US12/369,904 Abandoned US20090146972A1 (en) 2004-05-05 2009-02-12 Apparatus and method for detecting a pointer relative to a touch surface

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/838,536 Active 2025-11-25 US7492357B2 (en) 2004-05-05 2004-05-05 Apparatus and method for detecting a pointer relative to a touch surface

Country Status (6)

Country Link
US (2) US7492357B2 (en)
EP (2) EP1766501A4 (en)
JP (1) JP5122948B2 (en)
CN (1) CN101019096B (en)
CA (1) CA2564262A1 (en)
WO (1) WO2005106775A1 (en)

Cited By (44)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070236454A1 (en) * 2003-10-09 2007-10-11 Smart Technologies, Inc. Apparatus For Determining The Location Of A Pointer Within A Region Of Interest
US20090278795A1 (en) * 2008-05-09 2009-11-12 Smart Technologies Ulc Interactive Input System And Illumination Assembly Therefor
US20100083109A1 (en) * 2008-09-29 2010-04-01 Smart Technologies Ulc Method for handling interactions with multiple users of an interactive input system, and interactive input system executing the method
US20100079409A1 (en) * 2008-09-29 2010-04-01 Smart Technologies Ulc Touch panel for an interactive input system, and interactive input system incorporating the touch panel
US20100079493A1 (en) * 2008-09-29 2010-04-01 Smart Technologies Ulc Method for selecting and manipulating a graphical object in an interactive input system, and interactive input system executing the method
US20100201812A1 (en) * 2009-02-11 2010-08-12 Smart Technologies Ulc Active display feedback in interactive input systems
US20100207911A1 (en) * 2003-02-14 2010-08-19 Next Holdings Limited Touch screen Signal Processing With Single-Point Calibration
US20100265202A1 (en) * 2000-07-05 2010-10-21 Smart Technologies Ulc Passive touch system and method of detecting user input
WO2011003205A1 (en) * 2009-07-10 2011-01-13 Smart Technologies Ulc Disambiguating pointers by imaging multiple touch-input zones
US20110050650A1 (en) * 2009-09-01 2011-03-03 Smart Technologies Ulc Interactive input system with improved signal-to-noise ratio (snr) and image capture method
US20110069019A1 (en) * 2009-07-08 2011-03-24 Smart Technologies Ulc Method for manipulating a graphic widget in a three-dimensional environment displayed on a touch panel of an interactive input system
US20110095989A1 (en) * 2009-10-23 2011-04-28 Smart Technologies Ulc Interactive input system and bezel therefor
US20110169748A1 (en) * 2010-01-11 2011-07-14 Smart Technologies Ulc Method for handling user input in an interactive input system, and interactive input system executing the method
US20110234638A1 (en) * 2003-09-16 2011-09-29 Smart Technologies Ulc Gesture recognition method and touch system incorporating the same
WO2011120130A1 (en) * 2010-04-01 2011-10-06 Smart Technologies Ulc Multi-pointer disambiguation by combining image and acceleration data
US20110285669A1 (en) * 2010-05-21 2011-11-24 Lassesson Kristian Electronic Devices Including Interactive Displays Implemented Using Cameras and Related Methods and Computer Program Products
US8089462B2 (en) 2004-01-02 2012-01-03 Smart Technologies Ulc Pointer tracking across multiple overlapping coordinate input sub-regions defining a generally contiguous input region
US8094137B2 (en) 2007-07-23 2012-01-10 Smart Technologies Ulc System and method of detecting contact on a display
WO2012006716A1 (en) * 2010-07-12 2012-01-19 Smart Technologies Ulc Interactive input system and method
US8115753B2 (en) 2007-04-11 2012-02-14 Next Holdings Limited Touch screen system with hover and click input methods
US8120596B2 (en) 2004-05-21 2012-02-21 Smart Technologies Ulc Tiled touch system
US8149221B2 (en) 2004-05-07 2012-04-03 Next Holdings Limited Touch panel display system with illumination and detection provided from a single edge
US20120179994A1 (en) * 2011-01-12 2012-07-12 Smart Technologies Ulc Method for manipulating a toolbar on an interactive input system and interactive input system executing the method
US8228304B2 (en) 2002-11-15 2012-07-24 Smart Technologies Ulc Size/scale orientation determination of a pointer in a camera-based touch system
US8274496B2 (en) 2004-04-29 2012-09-25 Smart Technologies Ulc Dual mode touch systems
US8289299B2 (en) 2003-02-14 2012-10-16 Next Holdings Limited Touch screen signal processing
US20120293557A1 (en) * 2011-05-18 2012-11-22 Chip Goal Electronics Corporation, R.O.C. Object tracking apparatus, interactive image display system using object tracking apparatus, and methods thereof
US8339378B2 (en) 2008-11-05 2012-12-25 Smart Technologies Ulc Interactive input system with multi-angle reflector
US8384693B2 (en) 2007-08-30 2013-02-26 Next Holdings Limited Low profile touch panel systems
US8405637B2 (en) 2008-01-07 2013-03-26 Next Holdings Limited Optical position sensing system and optical position sensor assembly with convex imaging window
US8432377B2 (en) 2007-08-30 2013-04-30 Next Holdings Limited Optical touchscreen with improved illumination
US8456451B2 (en) 2003-03-11 2013-06-04 Smart Technologies Ulc System and method for differentiating between pointers used to contact touch surface
US8456447B2 (en) 2003-02-14 2013-06-04 Next Holdings Limited Touch screen signal processing
WO2013113101A1 (en) * 2012-02-02 2013-08-08 Smart Technologies Ulc Interactive input system and method of detecting objects
WO2013142958A1 (en) * 2012-03-30 2013-10-03 Smart Technologies Ulc Method for generally continuously calibrating an interactive input system
US8902193B2 (en) 2008-05-09 2014-12-02 Smart Technologies Ulc Interactive input system and bezel therefor
US20150103054A1 (en) * 2013-10-14 2015-04-16 Touchjet Pte. Ltd. Photoelectric touch assembly, photoelectric touch method and projector with touch function
US20150185896A1 (en) * 2013-12-28 2015-07-02 Paul J. Gwin Virtual and configurable touchscreens
US9140540B2 (en) 2012-12-07 2015-09-22 Ricoh Company, Ltd. Coordinate detector and electronic information board system
US9160979B1 (en) * 2011-05-27 2015-10-13 Trimble Navigation Limited Determining camera position for a photograph having a displaced center of projection
US20160209985A1 (en) * 2015-01-20 2016-07-21 Wistron Corporation Optical touch device and touch detecting method using the same
US9442607B2 (en) 2006-12-04 2016-09-13 Smart Technologies Inc. Interactive input system and method
US20170060255A1 (en) * 2015-08-26 2017-03-02 Samsung Electronics Co., Ltd. Object detection apparatus and object detection method thereof
US10185445B2 (en) 2013-10-14 2019-01-22 Touchjet Pte. Ltd. Determining touch signals from interactions with a reference plane proximate to a display surface

Families Citing this family (66)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4052498B2 (en) 1999-10-29 2008-02-27 株式会社リコー Coordinate input apparatus and method
JP2001184161A (en) 1999-12-27 2001-07-06 Ricoh Co Ltd Method and device for inputting information, writing input device, method for managing written data, method for controlling display, portable electronic writing device, and recording medium
US7492357B2 (en) * 2004-05-05 2009-02-17 Smart Technologies Ulc Apparatus and method for detecting a pointer relative to a touch surface
US7593593B2 (en) 2004-06-16 2009-09-22 Microsoft Corporation Method and system for reducing effects of undesired signals in an infrared imaging system
US20060152482A1 (en) * 2005-01-07 2006-07-13 Chauncy Godwin Virtual interface and control device
US7911444B2 (en) 2005-08-31 2011-03-22 Microsoft Corporation Input method for surface of interactive display
US7630002B2 (en) * 2007-01-05 2009-12-08 Microsoft Corporation Specular reflection reduction using multiple cameras
US8212857B2 (en) * 2007-01-26 2012-07-03 Microsoft Corporation Alternating light sources to reduce specular reflection
US8264468B1 (en) 2007-06-19 2012-09-11 Imaging Systems Technology, Inc. Touch system for blue screen
US8330730B1 (en) 2007-09-04 2012-12-11 Imaging Systems Technology, Inc. Calibrating of interactive touch system for image compositing
US8487881B2 (en) * 2007-10-17 2013-07-16 Smart Technologies Ulc Interactive input system, controller therefor and method of controlling an appliance
US20090128495A1 (en) * 2007-11-20 2009-05-21 Microsoft Corporation Optical input device
US20090213093A1 (en) * 2008-01-07 2009-08-27 Next Holdings Limited Optical position sensor using retroreflection
US20090207144A1 (en) * 2008-01-07 2009-08-20 Next Holdings Limited Position Sensing System With Edge Positioning Enhancement
CN101266530B (en) * 2008-04-04 2012-12-12 中国海洋大学 Large-screen three-dimensional measuring touch screen
US20090277697A1 (en) * 2008-05-09 2009-11-12 Smart Technologies Ulc Interactive Input System And Pen Tool Therefor
US20090278794A1 (en) * 2008-05-09 2009-11-12 Smart Technologies Ulc Interactive Input System With Controlled Lighting
AU2009253801A1 (en) * 2008-06-05 2009-12-10 Smart Technologies Ulc Multiple pointer ambiguity and occlusion resolution
US20110074738A1 (en) * 2008-06-18 2011-03-31 Beijing Irtouch Systems Co., Ltd. Touch Detection Sensing Apparatus
AU2009291462A1 (en) * 2008-09-15 2010-03-18 Smart Technologies Ulc Touch input with image sensor and signal processor
CN102160018B (en) * 2008-09-15 2014-09-24 惠普开发有限公司 Touchscreen display with plural cameras
EP2353069B1 (en) * 2008-10-02 2013-07-03 Next Holdings Limited Stereo optical sensors for resolving multi-touch in a touch detection system
US20100207912A1 (en) * 2009-02-13 2010-08-19 Arima Lasers Corp. Detection module and an optical detection system comprising the same
US20100229090A1 (en) * 2009-03-05 2010-09-09 Next Holdings Limited Systems and Methods for Interacting With Touch Displays Using Single-Touch and Multi-Touch Gestures
US7751671B1 (en) * 2009-03-25 2010-07-06 Next Holdings Limited Optical touchscreens comprising removably connected optical members
TWI399677B (en) * 2009-03-31 2013-06-21 Arima Lasers Corp Optical detection apparatus and method
US8884925B2 (en) * 2009-04-05 2014-11-11 Radion Engineering Co. Ltd. Display system and method utilizing optical sensors
JP2010282463A (en) * 2009-06-05 2010-12-16 Newcom Inc Touch panel device
TWI399676B (en) * 2009-06-30 2013-06-21 Pixart Imaging Inc Object detection calibration system of an optical touch screen and method thereof
KR100931520B1 (en) * 2009-08-11 2009-12-14 (주)누리봄 Image display apparatus for detecting a position
US20110096034A1 (en) * 2009-10-23 2011-04-28 Sonix Technology Co., Ltd. Optical touch-sensing display
KR101097992B1 (en) * 2009-11-05 2011-12-26 주식회사 스마트센스테크놀러지 The pointing device
CN102713794A (en) * 2009-11-24 2012-10-03 奈克斯特控股公司 Methods and apparatus for gesture recognition mode control
US20110199387A1 (en) * 2009-11-24 2011-08-18 John David Newton Activating Features on an Imaging Device Based on Manipulations
WO2011069148A1 (en) * 2009-12-04 2011-06-09 Next Holdings Limited Methods and systems for position detection using an interactive volume
US20110176082A1 (en) * 2010-01-18 2011-07-21 Matthew Allard Mounting Members For Touch Sensitive Displays
WO2011089538A1 (en) * 2010-01-25 2011-07-28 Naveen Chawla A stereo-calibration-less multiple-camera human-tracking system for human-computer 3d interaction
JP2011175543A (en) * 2010-02-25 2011-09-08 Sanyo Electric Co Ltd Indicator detection device and touch panel
US20110234542A1 (en) * 2010-03-26 2011-09-29 Paul Marson Methods and Systems Utilizing Multiple Wavelengths for Position Detection
JP2011210188A (en) * 2010-03-30 2011-10-20 Sony Corp Image processing apparatus, method of displaying image, image display program, and recording medium having image display program recorded thereon
US9092125B2 (en) 2010-04-08 2015-07-28 Avaya Inc. Multi-mode touchscreen user interface for a multi-state touchscreen device
US8338725B2 (en) 2010-04-29 2012-12-25 Au Optronics Corporation Camera based touch system
KR20120005903A (en) * 2010-07-09 2012-01-17 삼성전자주식회사 Multipurpose sensing device and electronic equipment including the same
US8269750B2 (en) * 2010-08-13 2012-09-18 Omnivision Technologies, Inc. Optical position input system and method
WO2012025159A1 (en) * 2010-08-27 2012-03-01 Brainlab Ag Multiple-layer pointing position determination on a medical display
CN102446033A (en) * 2010-10-12 2012-05-09 太瀚科技股份有限公司 Optical touch location system and method
US9019239B2 (en) * 2010-11-29 2015-04-28 Northrop Grumman Systems Corporation Creative design systems and methods
FR2976093B1 (en) * 2011-06-01 2013-08-16 Thales Sa OPTICAL TRANSMITTER AND RECEIVER TOUCH SYSTEM
US9274615B2 (en) 2011-11-11 2016-03-01 Pixart Imaging Inc. Interactive input system and method
TWI446250B (en) * 2011-12-05 2014-07-21 Wistron Corp Portable optical touch system and operating method thereof
CN102778976B (en) * 2012-02-07 2016-03-30 北京京东方光电科技有限公司 Touch point method for determining position and touch-screen on a kind of touch-screen
TWI471778B (en) * 2012-03-12 2015-02-01 Wintek Corp Touch device and touch sensing method thereof
TWI475446B (en) * 2012-04-24 2015-03-01 Wistron Corp Optical touch control system and capture signal adjusting method thereof
KR20140026252A (en) * 2012-08-22 2014-03-05 동우 화인켐 주식회사 Pointing display device
US9560343B2 (en) 2012-11-23 2017-01-31 Samsung Electronics Co., Ltd. Apparatus and method for calibrating multi-layer three-dimensional (3D) display
KR20150104183A (en) * 2013-01-07 2015-09-14 타마고 홀딩 에스에이 Panoramic camera
CN103488352B (en) * 2013-08-27 2016-09-14 合肥工业大学 The coordinate capture of infrared multi-point touch screen and correction algorithm
CN103743342B (en) * 2013-12-31 2017-04-05 温康纳(常州)机械制造有限公司 Synchronous relief visual identifying system
DE102014000436A1 (en) * 2014-01-16 2015-07-16 Mbda Deutschland Gmbh Method for determining the position data of a target object in a reference system and method for guiding an aircraft, which is preferably designed as a missile
JP2015225570A (en) * 2014-05-29 2015-12-14 富士通テン株式会社 Operation device
EP2985678B1 (en) * 2014-08-11 2019-04-24 Alpine Electronics, Inc. Position detection system and method of calibrating a position detection system
JP6464624B2 (en) * 2014-09-12 2019-02-06 株式会社リコー Image processing system, image processing apparatus, method, and program
US10146331B2 (en) 2014-11-28 2018-12-04 Ricoh Company, Ltd. Information processing system for transforming coordinates of a position designated by a pointer in a virtual image to world coordinates, information processing apparatus, and method of transforming coordinates
TWI564773B (en) * 2015-08-05 2017-01-01 緯創資通股份有限公司 Optical touch system and optical touch apparatus thereof
WO2019147612A1 (en) 2018-01-25 2019-08-01 Neonode Inc. Polar coordinate sensor
US11651516B2 (en) 2020-02-20 2023-05-16 Sony Group Corporation Multiple view triangulation with improved robustness to observation errors

Citations (100)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3025406A (en) * 1959-02-05 1962-03-13 Flightex Fabrics Inc Light screen for ballistic uses
US3860754A (en) * 1973-05-07 1975-01-14 Univ Illinois Light beam position encoder apparatus
US4144449A (en) * 1977-07-08 1979-03-13 Sperry Rand Corporation Position detection apparatus
US4243879A (en) * 1978-04-24 1981-01-06 Carroll Manufacturing Corporation Touch panel with ambient light sampling
US4247767A (en) * 1978-04-05 1981-01-27 Her Majesty The Queen In Right Of Canada, As Represented By The Minister Of National Defence Touch sensitive computer input device
US4507557A (en) * 1983-04-01 1985-03-26 Siemens Corporate Research & Support, Inc. Non-contact X,Y digitizer using two dynamic ram imagers
US4811004A (en) * 1987-05-11 1989-03-07 Dale Electronics, Inc. Touch panel system and method for using same
US4893120A (en) * 1986-11-26 1990-01-09 Digital Electronics Corporation Touch panel using modulated light
US4990901A (en) * 1987-08-25 1991-02-05 Technomarket, Inc. Liquid crystal display touch screen having electronics on one side
US5097516A (en) * 1991-02-28 1992-03-17 At&T Bell Laboratories Technique for illuminating a surface with a gradient intensity line of light to achieve enhanced two-dimensional imaging
US5179369A (en) * 1989-12-06 1993-01-12 Dale Electronics, Inc. Touch panel and method for controlling same
US5196836A (en) * 1991-06-28 1993-03-23 International Business Machines Corporation Touch panel display
US5196835A (en) * 1988-09-30 1993-03-23 International Business Machines Corporation Laser touch panel reflective surface aberration cancelling
US5317140A (en) * 1992-11-24 1994-05-31 Dunthorn David I Diffusion-assisted position location particularly for visual pen detection
US5483603A (en) * 1992-10-22 1996-01-09 Advanced Interconnection Technology System and method for automatic optical inspection
US5483261A (en) * 1992-02-14 1996-01-09 Itu Research, Inc. Graphical input controller and method with rear screen image detection
US5484966A (en) * 1993-12-07 1996-01-16 At&T Corp. Sensing stylus position using single 1-D image sensor
US5490655A (en) * 1993-09-16 1996-02-13 Monger Mounts, Inc. Video/data projector and monitor ceiling/wall mount
US5502568A (en) * 1993-03-23 1996-03-26 Wacom Co., Ltd. Optical position detecting unit, optical coordinate input unit and optical position detecting method employing a pattern having a sequence of 1's and 0's
US5591945A (en) * 1995-04-19 1997-01-07 Elo Touchsystems, Inc. Acoustic touch position sensor using higher order horizontally polarized shear wave propagation
US5594469A (en) * 1995-02-21 1997-01-14 Mitsubishi Electric Information Technology Center America Inc. Hand gesture machine control system
US5594502A (en) * 1993-01-20 1997-01-14 Elmo Company, Limited Image reproduction apparatus
US5729704A (en) * 1993-07-21 1998-03-17 Xerox Corporation User-directed method for operating on an object-based model data structure through a second contextual image
US5734375A (en) * 1995-06-07 1998-03-31 Compaq Computer Corporation Keyboard-compatible optical determination of object's position
US5914783A (en) * 1997-03-24 1999-06-22 Mistubishi Electric Information Technology Center America, Inc. Method and apparatus for detecting the location of a light source
US6031531A (en) * 1998-04-06 2000-02-29 International Business Machines Corporation Method and system in a graphical user interface for facilitating cursor object movement for physically challenged computer users
US6179426B1 (en) * 1999-03-03 2001-01-30 3M Innovative Properties Company Integrated front projection system
US6188388B1 (en) * 1993-12-28 2001-02-13 Hitachi, Ltd. Information presentation apparatus and information display apparatus
US6191773B1 (en) * 1995-04-28 2001-02-20 Matsushita Electric Industrial Co., Ltd. Interface apparatus
US6208329B1 (en) * 1996-08-13 2001-03-27 Lsi Logic Corporation Supplemental mouse button emulation system, method and apparatus for a coordinate based data input device
US6208330B1 (en) * 1997-03-07 2001-03-27 Canon Kabushiki Kaisha Coordinate input apparatus and its control method
US6335724B1 (en) * 1999-01-29 2002-01-01 Ricoh Company, Ltd. Method and device for inputting coordinate-position and a display board system
US6337681B1 (en) * 1991-10-21 2002-01-08 Smart Technologies Inc. Projection display system with pressure sensing at screen, and computer assisted alignment implemented by applying pressure at displayed calibration marks
US6339748B1 (en) * 1997-11-11 2002-01-15 Seiko Epson Corporation Coordinate input system and display apparatus
US20020008692A1 (en) * 1998-07-30 2002-01-24 Katsuyuki Omura Electronic blackboard system
US20020015159A1 (en) * 2000-08-04 2002-02-07 Akio Hashimoto Position detection device, position pointing device, position detecting method and pen-down detecting method
US6346966B1 (en) * 1997-07-07 2002-02-12 Agilent Technologies, Inc. Image acquisition system for machine vision applications
US6352351B1 (en) * 1999-06-30 2002-03-05 Ricoh Company, Ltd. Method and apparatus for inputting coordinates
US6353434B1 (en) * 1998-09-08 2002-03-05 Gunze Limited Input coordinate transformation apparatus for converting coordinates input from a coordinate input device into coordinates in a display coordinate system for displaying images on a display
US6359612B1 (en) * 1998-09-30 2002-03-19 Siemens Aktiengesellschaft Imaging system for displaying image information that has been acquired by means of a medical diagnostic imaging device
US6362468B1 (en) * 1999-06-10 2002-03-26 Saeilo Japan, Inc. Optical unit for detecting object and coordinate input apparatus using same
US20020036617A1 (en) * 1998-08-21 2002-03-28 Timothy R. Pryor Novel man machine interfaces and applications
US20020048027A1 (en) * 1993-05-24 2002-04-25 Alf Pettersen Method and system for geometry measurements
US20030001825A1 (en) * 1998-06-09 2003-01-02 Katsuyuki Omura Coordinate position inputting/detecting device, a method for inputting/detecting the coordinate position, and a display board system
US6504532B1 (en) * 1999-07-15 2003-01-07 Ricoh Company, Ltd. Coordinates detection apparatus
US6507339B1 (en) * 1999-08-23 2003-01-14 Ricoh Company, Ltd. Coordinate inputting/detecting system and a calibration method therefor
US6512838B1 (en) * 1999-09-22 2003-01-28 Canesta, Inc. Methods for enhancing performance and data acquired from three-dimensional image systems
US20030025951A1 (en) * 2001-07-27 2003-02-06 Pollard Stephen Bernard Paper-to-computer interfaces
US6518600B1 (en) * 2000-11-17 2003-02-11 General Electric Company Dual encapsulation for an LED
US6517266B2 (en) * 2001-05-15 2003-02-11 Xerox Corporation Systems and methods for hand-held printing on a surface or medium
US6522830B2 (en) * 1993-11-30 2003-02-18 Canon Kabushiki Kaisha Image pickup apparatus
US6529189B1 (en) * 2000-02-08 2003-03-04 International Business Machines Corporation Touch screen stylus with IR-coupled selection buttons
US20030043116A1 (en) * 2001-06-01 2003-03-06 Gerald Morrison Calibrating camera offsets to facilitate object Position determination using triangulation
US20030046401A1 (en) * 2000-10-16 2003-03-06 Abbott Kenneth H. Dynamically determing appropriate computer user interfaces
US6531999B1 (en) * 2000-07-13 2003-03-11 Koninklijke Philips Electronics N.V. Pointing direction calibration in video conferencing and other camera-based system applications
US6530664B2 (en) * 1999-03-03 2003-03-11 3M Innovative Properties Company Integrated front projection system with enhanced dry erase screen configuration
US6532006B1 (en) * 1999-01-29 2003-03-11 Ricoh Company, Ltd. Coordinates input device, coordinates input method, a display board system
US20030063073A1 (en) * 2001-10-03 2003-04-03 Geaghan Bernard O. Touch panel system and method for distinguishing multiple touch inputs
US20030086603A1 (en) * 2001-09-07 2003-05-08 Distortion Graphics, Inc. System and method for transforming graphical images
US20040001144A1 (en) * 2002-06-27 2004-01-01 Mccharles Randy Synchronization of camera images in camera-based touch system to enhance position determination of fast moving objects
US6674424B1 (en) * 1999-10-29 2004-01-06 Ricoh Company, Ltd. Method and apparatus for inputting information including coordinate data
US20040012573A1 (en) * 2000-07-05 2004-01-22 Gerald Morrison Passive touch system and method of detecting user input
US6683584B2 (en) * 1993-10-22 2004-01-27 Kopin Corporation Camera display system
US20040021633A1 (en) * 2002-04-06 2004-02-05 Rajkowski Janusz Wiktor Symbol encoding apparatus and method
US6690397B1 (en) * 2000-06-05 2004-02-10 Advanced Neuromodulation Systems, Inc. System for regional data association and presentation and method for the same
US6690363B2 (en) * 2000-06-19 2004-02-10 Next Holdings Limited Touch panel display system
US6690357B1 (en) * 1998-10-07 2004-02-10 Intel Corporation Input device using scanning sensors
US20040031779A1 (en) * 2002-05-17 2004-02-19 Cahill Steven P. Method and system for calibrating a laser processing system and laser marking system utilizing same
US20040032401A1 (en) * 2002-08-19 2004-02-19 Fujitsu Limited Touch panel device
US20040046749A1 (en) * 1996-10-15 2004-03-11 Nikon Corporation Image recording and replay apparatus
US20040051709A1 (en) * 2002-05-31 2004-03-18 Eit Co., Ltd. Apparatus for controlling the shift of virtual space and method and program for controlling same
US6710770B2 (en) * 2000-02-11 2004-03-23 Canesta, Inc. Quasi-three-dimensional method and apparatus to detect and localize interaction of user-object and virtual transfer device
US20040070674A1 (en) * 2002-10-15 2004-04-15 Foote Jonathan T. Method, apparatus, and system for remotely annotating a target
US20040233461A1 (en) * 1999-11-12 2004-11-25 Armstrong Brian S. Methods and apparatus for measuring orientation and distance
US6864882B2 (en) * 2000-05-24 2005-03-08 Next Holdings Limited Protected touch panel display system
US20050052427A1 (en) * 2003-09-10 2005-03-10 Wu Michael Chi Hung Hand gesture interaction with touch surface
US20050057524A1 (en) * 2003-09-16 2005-03-17 Hill Douglas B. Gesture recognition method and touch system incorporating the same
US20060012579A1 (en) * 2004-07-14 2006-01-19 Canon Kabushiki Kaisha Coordinate input apparatus and its control method
US20060022962A1 (en) * 2002-11-15 2006-02-02 Gerald Morrison Size/scale and orientation determination of a pointer in a camera-based touch system
US20060028456A1 (en) * 2002-10-10 2006-02-09 Byung-Geun Kang Pen-shaped optical mouse
US7002555B1 (en) * 1998-12-04 2006-02-21 Bayer Innovation Gmbh Display comprising touch panel
US7007236B2 (en) * 2001-09-14 2006-02-28 Accenture Global Services Gmbh Lab window collaboration
US20070019103A1 (en) * 2005-07-25 2007-01-25 Vkb Inc. Optical apparatus for virtual interface projection and sensing
US7176904B2 (en) * 2001-03-26 2007-02-13 Ricoh Company, Limited Information input/output apparatus, information input/output control method, and computer product
US7187489B2 (en) * 1999-10-05 2007-03-06 Idc, Llc Photonic MEMS and structures
US7190496B2 (en) * 2003-07-24 2007-03-13 Zebra Imaging, Inc. Enhanced environment visualization using holographic stereograms
US20080029691A1 (en) * 2006-08-03 2008-02-07 Han Jefferson Y Multi-touch sensing display through frustrated total internal reflection
US7330184B2 (en) * 2002-06-12 2008-02-12 Smart Technologies Ulc System and method for recognizing connector gestures
US7333095B1 (en) * 2006-07-12 2008-02-19 Lumio Inc Illumination for optical touch panel
US7333094B2 (en) * 2006-07-12 2008-02-19 Lumio Inc. Optical touch screen
US20080055262A1 (en) * 2006-08-31 2008-03-06 Au Optronics Corp. Liquid crystal display with a liquid crystal touch panel having photo-sensing elements
US20080055267A1 (en) * 2006-09-01 2008-03-06 Au Optronics Corp. Touch-control liquid crystal display background of the invention
US20080062140A1 (en) * 2006-06-09 2008-03-13 Apple Inc. Touch screen liquid crystal display
US20080062149A1 (en) * 2003-05-19 2008-03-13 Baruch Itzhak Optical coordinate input device comprising few elements
US20080068352A1 (en) * 2004-02-17 2008-03-20 Smart Technologies Inc. Apparatus for detecting a pointer within a region of interest
US7479949B2 (en) * 2006-09-06 2009-01-20 Apple Inc. Touch screen device, method, and graphical user interface for determining commands by applying heuristics
US7479982B2 (en) * 2002-07-03 2009-01-20 Topcon Corporation Device and method of measuring data for calibration, program for measuring data for calibration, program recording medium readable with computer, and image data processing device
US7492357B2 (en) * 2004-05-05 2009-02-17 Smart Technologies Ulc Apparatus and method for detecting a pointer relative to a touch surface
US20090058832A1 (en) * 2007-08-30 2009-03-05 John Newton Low Profile Touch Panel Systems
US20090058833A1 (en) * 2007-08-30 2009-03-05 John Newton Optical Touchscreen with Improved Illumination

Family Cites Families (76)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4558313A (en) 1981-12-31 1985-12-10 International Business Machines Corporation Indicator to data processing interface
JPS61262917A (en) 1985-05-17 1986-11-20 Alps Electric Co Ltd Filter for photoelectric touch panel
DE3616490A1 (en) 1985-05-17 1986-11-27 Alps Electric Co Ltd OPTICAL COORDINATE INPUT DEVICE
US4822145A (en) 1986-05-14 1989-04-18 Massachusetts Institute Of Technology Method and apparatus utilizing waveguide and polarized light for display of dynamic images
JPS6375918A (en) 1986-09-19 1988-04-06 Alps Electric Co Ltd Coordinate input device
US4782328A (en) 1986-10-02 1988-11-01 Product Development Services, Incorporated Ambient-light-responsive touch screen data input method and system
US4746770A (en) 1987-02-17 1988-05-24 Sensor Frame Incorporated Method and apparatus for isolating and manipulating graphic objects on computer video monitor
US5109435A (en) 1988-08-08 1992-04-28 Hughes Aircraft Company Segmentation method for use against moving objects
US5130794A (en) 1990-03-29 1992-07-14 Ritchey Kurtis J Panoramic display system
JPH05189137A (en) * 1992-01-16 1993-07-30 Sumitomo Heavy Ind Ltd Command input device for computer
GB9201949D0 (en) 1992-01-30 1992-03-18 Jenkin Michael Large-scale,touch-sensitive video display
US5880411A (en) 1992-06-08 1999-03-09 Synaptics, Incorporated Object position detector with edge motion feature and gesture recognition
JP3244798B2 (en) 1992-09-08 2002-01-07 株式会社東芝 Moving image processing device
US5982352A (en) 1992-09-18 1999-11-09 Pryor; Timothy R. Method for providing human input to a computer
US5359155A (en) 1993-03-25 1994-10-25 Tiger Scientific Corp. Illumination apparatus for a digitizer tablet
GB2286100A (en) 1994-01-19 1995-08-02 Ibm Touch-sensitive display apparatus
US5577733A (en) 1994-04-08 1996-11-26 Downing; Dennis L. Targeting system
US5771039A (en) 1994-06-06 1998-06-23 Ditzik; Richard J. Direct view display device integration techniques
US5737740A (en) 1994-06-27 1998-04-07 Numonics Apparatus and method for processing electronic documents
US5638092A (en) 1994-12-20 1997-06-10 Eng; Tommy K. Cursor control system
JPH08179888A (en) 1994-12-21 1996-07-12 Hitachi Ltd Input device for large screen display
US5554828A (en) 1995-01-03 1996-09-10 Texas Instruments Inc. Integration of pen-based capability into a field emission device system
US5736686A (en) 1995-03-01 1998-04-07 Gtco Corporation Illumination apparatus for a digitizer tablet with improved light panel
JP3436828B2 (en) 1995-05-08 2003-08-18 株式会社リコー Image processing device
US5786810A (en) 1995-06-07 1998-07-28 Compaq Computer Corporation Method of determining an object's position and associated apparatus
US5825352A (en) 1996-01-04 1998-10-20 Logitech, Inc. Multiple fingers contact sensing method for emulating mouse buttons and mouse operations on a touch sensor pad
JPH09190284A (en) 1996-01-11 1997-07-22 Canon Inc Information processor and information processing method
TW394879B (en) 1996-02-09 2000-06-21 Sega Enterprises Kk Graphics processing system and its data input device
US6002808A (en) 1996-07-26 1999-12-14 Mitsubishi Electric Information Technology Center America, Inc. Hand gesture control system
US5936615A (en) 1996-09-12 1999-08-10 Digital Equipment Corporation Image-based touchscreen
US5819201A (en) 1996-09-13 1998-10-06 Magellan Dis, Inc. Navigation system with vehicle service information
AU5156198A (en) 1996-10-29 1998-05-22 Xeotron Corporation Optical device utilizing optical waveguides and mechanical light-switches
US6061177A (en) 1996-12-19 2000-05-09 Fujimoto; Kenneth Noboru Integrated computer display and graphical input apparatus and method
US6252989B1 (en) 1997-01-07 2001-06-26 Board Of The Regents, The University Of Texas System Foveated image coding system and method for image bandwidth reduction
US5914709A (en) 1997-03-14 1999-06-22 Poa Sana, Llc User input device for a computer system
JP3876942B2 (en) 1997-06-13 2007-02-07 株式会社ワコム Optical digitizer
US6161066A (en) 1997-08-18 2000-12-12 The Texas A&M University System Advanced law enforcement and response technology
US6243074B1 (en) 1997-08-29 2001-06-05 Xerox Corporation Handedness detection for a physical manipulatory grammar
US6072494A (en) 1997-10-15 2000-06-06 Electric Planet, Inc. Method and apparatus for real-time gesture recognition
TW449709B (en) 1997-11-17 2001-08-11 Hewlett Packard Co A method for distinguishing a contact input
US6310610B1 (en) 1997-12-04 2001-10-30 Nortel Networks Limited Intelligent touch display
EP2256605B1 (en) 1998-01-26 2017-12-06 Apple Inc. Method and apparatus for integrating manual input
AU2439399A (en) * 1998-02-09 1999-08-23 Haim Azaria Video camera computer touch screen system
JP2000105671A (en) 1998-05-11 2000-04-11 Ricoh Co Ltd Coordinate input and detecting device, and electronic blackboard system
US6064354A (en) 1998-07-01 2000-05-16 Deluca; Michael Joseph Stereoscopic user interface method and apparatus
US6972753B1 (en) 1998-10-02 2005-12-06 Semiconductor Energy Laboratory Co., Ltd. Touch panel, display device provided with touch panel and electronic equipment provided with display device
US6545669B1 (en) 1999-03-26 2003-04-08 Husam Kinawi Object-drag continuity between discontinuous touch-screens
JP2000298544A (en) * 1999-04-12 2000-10-24 Matsushita Electric Ind Co Ltd Input/output device and its method
TW459192B (en) 1999-06-25 2001-10-11 Toshiba Corp Electronic apparatus and electronic system provided with the same
US6275214B1 (en) * 1999-07-06 2001-08-14 Karl C. Hansen Computer presentation system and method with optical tracking of wireless pointer
JP3905670B2 (en) 1999-09-10 2007-04-18 株式会社リコー Coordinate input detection apparatus, information storage medium, and coordinate input detection method
JP4057200B2 (en) 1999-09-10 2008-03-05 株式会社リコー Coordinate input device and recording medium for coordinate input device
WO2001023933A1 (en) 1999-09-29 2001-04-05 Nikon Corporation Projection optical system
JP2001184161A (en) * 1999-12-27 2001-07-06 Ricoh Co Ltd Method and device for inputting information, writing input device, method for managing written data, method for controlling display, portable electronic writing device, and recording medium
JP3934846B2 (en) 2000-03-06 2007-06-20 株式会社リコー Coordinate input / detection device, electronic blackboard system, light receiving element positional deviation correction method, and storage medium
JP2001265516A (en) 2000-03-16 2001-09-28 Ricoh Co Ltd Coordinate input device
JP2001282445A (en) 2000-03-31 2001-10-12 Ricoh Co Ltd Coordinate input/detecting device and information display input device
DE60140909D1 (en) * 2000-07-05 2010-02-04 Smart Technologies Ulc Method for a camera-based touch system
US7466843B2 (en) * 2000-07-07 2008-12-16 Pryor Timothy R Multi-functional control and entertainment systems
US6747663B2 (en) 2000-08-24 2004-06-08 Sun Microsystems, Inc. Interpolating sample values from known triangle vertex values
JP3793014B2 (en) 2000-10-03 2006-07-05 キヤノン株式会社 Electron source manufacturing apparatus, electron source manufacturing method, and image forming apparatus manufacturing method
US6774889B1 (en) 2000-10-24 2004-08-10 Microsoft Corporation System and method for transforming an ordinary computer monitor screen into a touch screen
JP2002196874A (en) * 2000-12-27 2002-07-12 Ntt Docomo Inc Device and method for inputting handwritten data, personal certification device and its method
US7030861B1 (en) 2001-02-10 2006-04-18 Wayne Carl Westerman System and method for packing multi-touch gestures onto a hand
JP4551018B2 (en) 2001-04-05 2010-09-22 富士通株式会社 Image combiner
JP2003173237A (en) 2001-09-28 2003-06-20 Ricoh Co Ltd Information input-output system, program and storage medium
JP3920067B2 (en) * 2001-10-09 2007-05-30 株式会社イーアイティー Coordinate input device
JP2003167669A (en) 2001-11-22 2003-06-13 Internatl Business Mach Corp <Ibm> Information processor, program, and coordinate input method
US6628216B2 (en) 2002-02-13 2003-09-30 Intersil Americas Inc. Calibration of resistor ladder using difference measurement and parallel resistive correction
JP3763409B2 (en) * 2002-03-27 2006-04-05 独立行政法人理化学研究所 3D position input device
JP2004013314A (en) * 2002-06-04 2004-01-15 Fuji Xerox Co Ltd Position measuring input support device
US20030226968A1 (en) * 2002-06-10 2003-12-11 Steve Montellese Apparatus and method for inputting data
JP2004038528A (en) * 2002-07-03 2004-02-05 Saeilo Japan Inc Optical coordinate detecting device
JP2004046464A (en) * 2002-07-10 2004-02-12 Nippon Telegr & Teleph Corp <Ntt> Apparatus and method for estimating three-dimensional position of mobile object, program, and recording medium thereof
US6972401B2 (en) 2003-01-30 2005-12-06 Smart Technologies Inc. Illuminated bezel and touch system incorporating the same
US20080129707A1 (en) * 2004-07-27 2008-06-05 Pryor Timothy R Method and apparatus employing multi-functional controls and displays

Patent Citations (106)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3025406A (en) * 1959-02-05 1962-03-13 Flightex Fabrics Inc Light screen for ballistic uses
US3860754A (en) * 1973-05-07 1975-01-14 Univ Illinois Light beam position encoder apparatus
US4144449A (en) * 1977-07-08 1979-03-13 Sperry Rand Corporation Position detection apparatus
US4247767A (en) * 1978-04-05 1981-01-27 Her Majesty The Queen In Right Of Canada, As Represented By The Minister Of National Defence Touch sensitive computer input device
US4243879A (en) * 1978-04-24 1981-01-06 Carroll Manufacturing Corporation Touch panel with ambient light sampling
US4507557A (en) * 1983-04-01 1985-03-26 Siemens Corporate Research & Support, Inc. Non-contact X,Y digitizer using two dynamic ram imagers
US4893120A (en) * 1986-11-26 1990-01-09 Digital Electronics Corporation Touch panel using modulated light
US4811004A (en) * 1987-05-11 1989-03-07 Dale Electronics, Inc. Touch panel system and method for using same
US4990901A (en) * 1987-08-25 1991-02-05 Technomarket, Inc. Liquid crystal display touch screen having electronics on one side
US5196835A (en) * 1988-09-30 1993-03-23 International Business Machines Corporation Laser touch panel reflective surface aberration cancelling
US5179369A (en) * 1989-12-06 1993-01-12 Dale Electronics, Inc. Touch panel and method for controlling same
US5097516A (en) * 1991-02-28 1992-03-17 At&T Bell Laboratories Technique for illuminating a surface with a gradient intensity line of light to achieve enhanced two-dimensional imaging
US5196836A (en) * 1991-06-28 1993-03-23 International Business Machines Corporation Touch panel display
US20080042999A1 (en) * 1991-10-21 2008-02-21 Martin David A Projection display system with pressure sensing at a screen, a calibration system corrects for non-orthogonal projection errors
US6337681B1 (en) * 1991-10-21 2002-01-08 Smart Technologies Inc. Projection display system with pressure sensing at screen, and computer assisted alignment implemented by applying pressure at displayed calibration marks
US5483261A (en) * 1992-02-14 1996-01-09 Itu Research, Inc. Graphical input controller and method with rear screen image detection
US5483603A (en) * 1992-10-22 1996-01-09 Advanced Interconnection Technology System and method for automatic optical inspection
US5317140A (en) * 1992-11-24 1994-05-31 Dunthorn David I Diffusion-assisted position location particularly for visual pen detection
US5594502A (en) * 1993-01-20 1997-01-14 Elmo Company, Limited Image reproduction apparatus
US5502568A (en) * 1993-03-23 1996-03-26 Wacom Co., Ltd. Optical position detecting unit, optical coordinate input unit and optical position detecting method employing a pattern having a sequence of 1's and 0's
US20020048027A1 (en) * 1993-05-24 2002-04-25 Alf Pettersen Method and system for geometry measurements
US5729704A (en) * 1993-07-21 1998-03-17 Xerox Corporation User-directed method for operating on an object-based model data structure through a second contextual image
US5490655A (en) * 1993-09-16 1996-02-13 Monger Mounts, Inc. Video/data projector and monitor ceiling/wall mount
US6683584B2 (en) * 1993-10-22 2004-01-27 Kopin Corporation Camera display system
US6522830B2 (en) * 1993-11-30 2003-02-18 Canon Kabushiki Kaisha Image pickup apparatus
US5484966A (en) * 1993-12-07 1996-01-16 At&T Corp. Sensing stylus position using single 1-D image sensor
US6188388B1 (en) * 1993-12-28 2001-02-13 Hitachi, Ltd. Information presentation apparatus and information display apparatus
US5594469A (en) * 1995-02-21 1997-01-14 Mitsubishi Electric Information Technology Center America Inc. Hand gesture machine control system
US5591945A (en) * 1995-04-19 1997-01-07 Elo Touchsystems, Inc. Acoustic touch position sensor using higher order horizontally polarized shear wave propagation
US6191773B1 (en) * 1995-04-28 2001-02-20 Matsushita Electric Industrial Co., Ltd. Interface apparatus
US5734375A (en) * 1995-06-07 1998-03-31 Compaq Computer Corporation Keyboard-compatible optical determination of object's position
US6208329B1 (en) * 1996-08-13 2001-03-27 Lsi Logic Corporation Supplemental mouse button emulation system, method and apparatus for a coordinate based data input device
US20040046749A1 (en) * 1996-10-15 2004-03-11 Nikon Corporation Image recording and replay apparatus
US6208330B1 (en) * 1997-03-07 2001-03-27 Canon Kabushiki Kaisha Coordinate input apparatus and its control method
US5914783A (en) * 1997-03-24 1999-06-22 Mistubishi Electric Information Technology Center America, Inc. Method and apparatus for detecting the location of a light source
US6346966B1 (en) * 1997-07-07 2002-02-12 Agilent Technologies, Inc. Image acquisition system for machine vision applications
US6339748B1 (en) * 1997-11-11 2002-01-15 Seiko Epson Corporation Coordinate input system and display apparatus
US6031531A (en) * 1998-04-06 2000-02-29 International Business Machines Corporation Method and system in a graphical user interface for facilitating cursor object movement for physically challenged computer users
US20030001825A1 (en) * 1998-06-09 2003-01-02 Katsuyuki Omura Coordinate position inputting/detecting device, a method for inputting/detecting the coordinate position, and a display board system
US20020008692A1 (en) * 1998-07-30 2002-01-24 Katsuyuki Omura Electronic blackboard system
US20020036617A1 (en) * 1998-08-21 2002-03-28 Timothy R. Pryor Novel man machine interfaces and applications
US6353434B1 (en) * 1998-09-08 2002-03-05 Gunze Limited Input coordinate transformation apparatus for converting coordinates input from a coordinate input device into coordinates in a display coordinate system for displaying images on a display
US6359612B1 (en) * 1998-09-30 2002-03-19 Siemens Aktiengesellschaft Imaging system for displaying image information that has been acquired by means of a medical diagnostic imaging device
US6690357B1 (en) * 1998-10-07 2004-02-10 Intel Corporation Input device using scanning sensors
US7002555B1 (en) * 1998-12-04 2006-02-21 Bayer Innovation Gmbh Display comprising touch panel
US6532006B1 (en) * 1999-01-29 2003-03-11 Ricoh Company, Ltd. Coordinates input device, coordinates input method, a display board system
US6335724B1 (en) * 1999-01-29 2002-01-01 Ricoh Company, Ltd. Method and device for inputting coordinate-position and a display board system
US6530664B2 (en) * 1999-03-03 2003-03-11 3M Innovative Properties Company Integrated front projection system with enhanced dry erase screen configuration
US6179426B1 (en) * 1999-03-03 2001-01-30 3M Innovative Properties Company Integrated front projection system
US6362468B1 (en) * 1999-06-10 2002-03-26 Saeilo Japan, Inc. Optical unit for detecting object and coordinate input apparatus using same
US6352351B1 (en) * 1999-06-30 2002-03-05 Ricoh Company, Ltd. Method and apparatus for inputting coordinates
US6504532B1 (en) * 1999-07-15 2003-01-07 Ricoh Company, Ltd. Coordinates detection apparatus
US6507339B1 (en) * 1999-08-23 2003-01-14 Ricoh Company, Ltd. Coordinate inputting/detecting system and a calibration method therefor
US6512838B1 (en) * 1999-09-22 2003-01-28 Canesta, Inc. Methods for enhancing performance and data acquired from three-dimensional image systems
US7187489B2 (en) * 1999-10-05 2007-03-06 Idc, Llc Photonic MEMS and structures
US6674424B1 (en) * 1999-10-29 2004-01-06 Ricoh Company, Ltd. Method and apparatus for inputting information including coordinate data
US20040233461A1 (en) * 1999-11-12 2004-11-25 Armstrong Brian S. Methods and apparatus for measuring orientation and distance
US6529189B1 (en) * 2000-02-08 2003-03-04 International Business Machines Corporation Touch screen stylus with IR-coupled selection buttons
US6710770B2 (en) * 2000-02-11 2004-03-23 Canesta, Inc. Quasi-three-dimensional method and apparatus to detect and localize interaction of user-object and virtual transfer device
US6864882B2 (en) * 2000-05-24 2005-03-08 Next Holdings Limited Protected touch panel display system
US6690397B1 (en) * 2000-06-05 2004-02-10 Advanced Neuromodulation Systems, Inc. System for regional data association and presentation and method for the same
US6690363B2 (en) * 2000-06-19 2004-02-10 Next Holdings Limited Touch panel display system
US20040012573A1 (en) * 2000-07-05 2004-01-22 Gerald Morrison Passive touch system and method of detecting user input
US20060034486A1 (en) * 2000-07-05 2006-02-16 Gerald Morrison Passive touch system and method of detecting user input
US20070002028A1 (en) * 2000-07-05 2007-01-04 Smart Technologies, Inc. Passive Touch System And Method Of Detecting User Input
US6531999B1 (en) * 2000-07-13 2003-03-11 Koninklijke Philips Electronics N.V. Pointing direction calibration in video conferencing and other camera-based system applications
US6714311B2 (en) * 2000-08-04 2004-03-30 Xiroku Inc. Position detection device, position pointing device, position detecting method and pen-down detecting method
US20020015159A1 (en) * 2000-08-04 2002-02-07 Akio Hashimoto Position detection device, position pointing device, position detecting method and pen-down detecting method
US20030046401A1 (en) * 2000-10-16 2003-03-06 Abbott Kenneth H. Dynamically determing appropriate computer user interfaces
US6518600B1 (en) * 2000-11-17 2003-02-11 General Electric Company Dual encapsulation for an LED
US7176904B2 (en) * 2001-03-26 2007-02-13 Ricoh Company, Limited Information input/output apparatus, information input/output control method, and computer product
US6517266B2 (en) * 2001-05-15 2003-02-11 Xerox Corporation Systems and methods for hand-held printing on a surface or medium
US20030043116A1 (en) * 2001-06-01 2003-03-06 Gerald Morrison Calibrating camera offsets to facilitate object Position determination using triangulation
US20030025951A1 (en) * 2001-07-27 2003-02-06 Pollard Stephen Bernard Paper-to-computer interfaces
US20030086603A1 (en) * 2001-09-07 2003-05-08 Distortion Graphics, Inc. System and method for transforming graphical images
US7007236B2 (en) * 2001-09-14 2006-02-28 Accenture Global Services Gmbh Lab window collaboration
US20030063073A1 (en) * 2001-10-03 2003-04-03 Geaghan Bernard O. Touch panel system and method for distinguishing multiple touch inputs
US20040021633A1 (en) * 2002-04-06 2004-02-05 Rajkowski Janusz Wiktor Symbol encoding apparatus and method
US20040031779A1 (en) * 2002-05-17 2004-02-19 Cahill Steven P. Method and system for calibrating a laser processing system and laser marking system utilizing same
US7015418B2 (en) * 2002-05-17 2006-03-21 Gsi Group Corporation Method and system for calibrating a laser processing system and laser marking system utilizing same
US20040051709A1 (en) * 2002-05-31 2004-03-18 Eit Co., Ltd. Apparatus for controlling the shift of virtual space and method and program for controlling same
US7330184B2 (en) * 2002-06-12 2008-02-12 Smart Technologies Ulc System and method for recognizing connector gestures
US20040001144A1 (en) * 2002-06-27 2004-01-01 Mccharles Randy Synchronization of camera images in camera-based touch system to enhance position determination of fast moving objects
US7184030B2 (en) * 2002-06-27 2007-02-27 Smart Technologies Inc. Synchronization of cameras in camera-based touch system to enhance position determination of fast moving objects
US7479982B2 (en) * 2002-07-03 2009-01-20 Topcon Corporation Device and method of measuring data for calibration, program for measuring data for calibration, program recording medium readable with computer, and image data processing device
US20040032401A1 (en) * 2002-08-19 2004-02-19 Fujitsu Limited Touch panel device
US20060028456A1 (en) * 2002-10-10 2006-02-09 Byung-Geun Kang Pen-shaped optical mouse
US20040070674A1 (en) * 2002-10-15 2004-04-15 Foote Jonathan T. Method, apparatus, and system for remotely annotating a target
US20060022962A1 (en) * 2002-11-15 2006-02-02 Gerald Morrison Size/scale and orientation determination of a pointer in a camera-based touch system
US20080062149A1 (en) * 2003-05-19 2008-03-13 Baruch Itzhak Optical coordinate input device comprising few elements
US7190496B2 (en) * 2003-07-24 2007-03-13 Zebra Imaging, Inc. Enhanced environment visualization using holographic stereograms
US20050052427A1 (en) * 2003-09-10 2005-03-10 Wu Michael Chi Hung Hand gesture interaction with touch surface
US20050057524A1 (en) * 2003-09-16 2005-03-17 Hill Douglas B. Gesture recognition method and touch system incorporating the same
US20080068352A1 (en) * 2004-02-17 2008-03-20 Smart Technologies Inc. Apparatus for detecting a pointer within a region of interest
US7492357B2 (en) * 2004-05-05 2009-02-17 Smart Technologies Ulc Apparatus and method for detecting a pointer relative to a touch surface
US20060012579A1 (en) * 2004-07-14 2006-01-19 Canon Kabushiki Kaisha Coordinate input apparatus and its control method
US20070019103A1 (en) * 2005-07-25 2007-01-25 Vkb Inc. Optical apparatus for virtual interface projection and sensing
US20080062140A1 (en) * 2006-06-09 2008-03-13 Apple Inc. Touch screen liquid crystal display
US7333094B2 (en) * 2006-07-12 2008-02-19 Lumio Inc. Optical touch screen
US7333095B1 (en) * 2006-07-12 2008-02-19 Lumio Inc Illumination for optical touch panel
US20080029691A1 (en) * 2006-08-03 2008-02-07 Han Jefferson Y Multi-touch sensing display through frustrated total internal reflection
US20080055262A1 (en) * 2006-08-31 2008-03-06 Au Optronics Corp. Liquid crystal display with a liquid crystal touch panel having photo-sensing elements
US20080055267A1 (en) * 2006-09-01 2008-03-06 Au Optronics Corp. Touch-control liquid crystal display background of the invention
US7479949B2 (en) * 2006-09-06 2009-01-20 Apple Inc. Touch screen device, method, and graphical user interface for determining commands by applying heuristics
US20090058832A1 (en) * 2007-08-30 2009-03-05 John Newton Low Profile Touch Panel Systems
US20090058833A1 (en) * 2007-08-30 2009-03-05 John Newton Optical Touchscreen with Improved Illumination

Cited By (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8203535B2 (en) 2000-07-05 2012-06-19 Smart Technologies Ulc Passive touch system and method of detecting user input
US20100265202A1 (en) * 2000-07-05 2010-10-21 Smart Technologies Ulc Passive touch system and method of detecting user input
US8378986B2 (en) 2000-07-05 2013-02-19 Smart Technologies Ulc Passive touch system and method of detecting user input
US8055022B2 (en) 2000-07-05 2011-11-08 Smart Technologies Ulc Passive touch system and method of detecting user input
US8228304B2 (en) 2002-11-15 2012-07-24 Smart Technologies Ulc Size/scale orientation determination of a pointer in a camera-based touch system
US8508508B2 (en) * 2003-02-14 2013-08-13 Next Holdings Limited Touch screen signal processing with single-point calibration
US8466885B2 (en) 2003-02-14 2013-06-18 Next Holdings Limited Touch screen signal processing
US8456447B2 (en) 2003-02-14 2013-06-04 Next Holdings Limited Touch screen signal processing
US20100207911A1 (en) * 2003-02-14 2010-08-19 Next Holdings Limited Touch screen Signal Processing With Single-Point Calibration
US8289299B2 (en) 2003-02-14 2012-10-16 Next Holdings Limited Touch screen signal processing
US8456451B2 (en) 2003-03-11 2013-06-04 Smart Technologies Ulc System and method for differentiating between pointers used to contact touch surface
US8325134B2 (en) 2003-09-16 2012-12-04 Smart Technologies Ulc Gesture recognition method and touch system incorporating the same
US20110234638A1 (en) * 2003-09-16 2011-09-29 Smart Technologies Ulc Gesture recognition method and touch system incorporating the same
US8456418B2 (en) 2003-10-09 2013-06-04 Smart Technologies Ulc Apparatus for determining the location of a pointer within a region of interest
US20070236454A1 (en) * 2003-10-09 2007-10-11 Smart Technologies, Inc. Apparatus For Determining The Location Of A Pointer Within A Region Of Interest
US8089462B2 (en) 2004-01-02 2012-01-03 Smart Technologies Ulc Pointer tracking across multiple overlapping coordinate input sub-regions defining a generally contiguous input region
US8576172B2 (en) 2004-01-02 2013-11-05 Smart Technologies Ulc Pointer tracking across multiple overlapping coordinate input sub-regions defining a generally contiguous input region
US8274496B2 (en) 2004-04-29 2012-09-25 Smart Technologies Ulc Dual mode touch systems
US8149221B2 (en) 2004-05-07 2012-04-03 Next Holdings Limited Touch panel display system with illumination and detection provided from a single edge
US8120596B2 (en) 2004-05-21 2012-02-21 Smart Technologies Ulc Tiled touch system
US9442607B2 (en) 2006-12-04 2016-09-13 Smart Technologies Inc. Interactive input system and method
US8115753B2 (en) 2007-04-11 2012-02-14 Next Holdings Limited Touch screen system with hover and click input methods
US8094137B2 (en) 2007-07-23 2012-01-10 Smart Technologies Ulc System and method of detecting contact on a display
US8432377B2 (en) 2007-08-30 2013-04-30 Next Holdings Limited Optical touchscreen with improved illumination
US8384693B2 (en) 2007-08-30 2013-02-26 Next Holdings Limited Low profile touch panel systems
US8405636B2 (en) 2008-01-07 2013-03-26 Next Holdings Limited Optical position sensing system and optical position sensor assembly
US8405637B2 (en) 2008-01-07 2013-03-26 Next Holdings Limited Optical position sensing system and optical position sensor assembly with convex imaging window
US20090278795A1 (en) * 2008-05-09 2009-11-12 Smart Technologies Ulc Interactive Input System And Illumination Assembly Therefor
US8902193B2 (en) 2008-05-09 2014-12-02 Smart Technologies Ulc Interactive input system and bezel therefor
US20100079493A1 (en) * 2008-09-29 2010-04-01 Smart Technologies Ulc Method for selecting and manipulating a graphical object in an interactive input system, and interactive input system executing the method
US8810522B2 (en) 2008-09-29 2014-08-19 Smart Technologies Ulc Method for selecting and manipulating a graphical object in an interactive input system, and interactive input system executing the method
US20100083109A1 (en) * 2008-09-29 2010-04-01 Smart Technologies Ulc Method for handling interactions with multiple users of an interactive input system, and interactive input system executing the method
US20100079409A1 (en) * 2008-09-29 2010-04-01 Smart Technologies Ulc Touch panel for an interactive input system, and interactive input system incorporating the touch panel
US8339378B2 (en) 2008-11-05 2012-12-25 Smart Technologies Ulc Interactive input system with multi-angle reflector
US20100201812A1 (en) * 2009-02-11 2010-08-12 Smart Technologies Ulc Active display feedback in interactive input systems
US8416206B2 (en) 2009-07-08 2013-04-09 Smart Technologies Ulc Method for manipulating a graphic widget in a three-dimensional environment displayed on a touch panel of an interactive input system
US20110069019A1 (en) * 2009-07-08 2011-03-24 Smart Technologies Ulc Method for manipulating a graphic widget in a three-dimensional environment displayed on a touch panel of an interactive input system
EP2452255A4 (en) * 2009-07-10 2013-12-25 Smart Technologies Ulc Disambiguating pointers by imaging multiple touch-input zones
WO2011003205A1 (en) * 2009-07-10 2011-01-13 Smart Technologies Ulc Disambiguating pointers by imaging multiple touch-input zones
CN102549536A (en) * 2009-07-10 2012-07-04 智能技术无限责任公司 Disambiguating pointers by imaging multiple touch-input zones
US8692768B2 (en) 2009-07-10 2014-04-08 Smart Technologies Ulc Interactive input system
EP2452255A1 (en) * 2009-07-10 2012-05-16 SMART Technologies ULC Disambiguating pointers by imaging multiple touch-input zones
US20110050650A1 (en) * 2009-09-01 2011-03-03 Smart Technologies Ulc Interactive input system with improved signal-to-noise ratio (snr) and image capture method
US8902195B2 (en) 2009-09-01 2014-12-02 Smart Technologies Ulc Interactive input system with improved signal-to-noise ratio (SNR) and image capture method
US20110095989A1 (en) * 2009-10-23 2011-04-28 Smart Technologies Ulc Interactive input system and bezel therefor
US8502789B2 (en) 2010-01-11 2013-08-06 Smart Technologies Ulc Method for handling user input in an interactive input system, and interactive input system executing the method
US20110169748A1 (en) * 2010-01-11 2011-07-14 Smart Technologies Ulc Method for handling user input in an interactive input system, and interactive input system executing the method
WO2011120130A1 (en) * 2010-04-01 2011-10-06 Smart Technologies Ulc Multi-pointer disambiguation by combining image and acceleration data
US20110285669A1 (en) * 2010-05-21 2011-11-24 Lassesson Kristian Electronic Devices Including Interactive Displays Implemented Using Cameras and Related Methods and Computer Program Products
WO2012006716A1 (en) * 2010-07-12 2012-01-19 Smart Technologies Ulc Interactive input system and method
US20120179994A1 (en) * 2011-01-12 2012-07-12 Smart Technologies Ulc Method for manipulating a toolbar on an interactive input system and interactive input system executing the method
US20120293557A1 (en) * 2011-05-18 2012-11-22 Chip Goal Electronics Corporation, R.O.C. Object tracking apparatus, interactive image display system using object tracking apparatus, and methods thereof
US9160979B1 (en) * 2011-05-27 2015-10-13 Trimble Navigation Limited Determining camera position for a photograph having a displaced center of projection
WO2013113101A1 (en) * 2012-02-02 2013-08-08 Smart Technologies Ulc Interactive input system and method of detecting objects
US9323322B2 (en) 2012-02-02 2016-04-26 Smart Technologies Ulc Interactive input system and method of detecting objects
WO2013142958A1 (en) * 2012-03-30 2013-10-03 Smart Technologies Ulc Method for generally continuously calibrating an interactive input system
US9360966B2 (en) 2012-03-30 2016-06-07 Smart Technologies Ulc Method for generally continuously calibrating an interactive input system
US9140540B2 (en) 2012-12-07 2015-09-22 Ricoh Company, Ltd. Coordinate detector and electronic information board system
US20150103054A1 (en) * 2013-10-14 2015-04-16 Touchjet Pte. Ltd. Photoelectric touch assembly, photoelectric touch method and projector with touch function
US10185445B2 (en) 2013-10-14 2019-01-22 Touchjet Pte. Ltd. Determining touch signals from interactions with a reference plane proximate to a display surface
US9317150B2 (en) * 2013-12-28 2016-04-19 Intel Corporation Virtual and configurable touchscreens
US20150185896A1 (en) * 2013-12-28 2015-07-02 Paul J. Gwin Virtual and configurable touchscreens
US20160209985A1 (en) * 2015-01-20 2016-07-21 Wistron Corporation Optical touch device and touch detecting method using the same
US9772718B2 (en) * 2015-01-20 2017-09-26 Wistron Corporation Optical touch device and touch detecting method using the same
US20170060255A1 (en) * 2015-08-26 2017-03-02 Samsung Electronics Co., Ltd. Object detection apparatus and object detection method thereof

Also Published As

Publication number Publication date
US7492357B2 (en) 2009-02-17
CN101019096A (en) 2007-08-15
CA2564262A1 (en) 2005-11-10
CN101019096B (en) 2012-04-18
EP2562622A2 (en) 2013-02-27
JP5122948B2 (en) 2013-01-16
WO2005106775A1 (en) 2005-11-10
US20050248539A1 (en) 2005-11-10
JP2007536652A (en) 2007-12-13
EP2562622A3 (en) 2013-08-07
EP1766501A1 (en) 2007-03-28
EP1766501A4 (en) 2008-07-16

Similar Documents

Publication Publication Date Title
US7492357B2 (en) Apparatus and method for detecting a pointer relative to a touch surface
US7256772B2 (en) Auto-aligning touch system and method
US6919880B2 (en) Calibrating camera offsets to facilitate object position determination using triangulation
US20050088424A1 (en) Passive touch system and method of detecting user input
US9442607B2 (en) Interactive input system and method
JP2007129709A (en) Method for calibrating imaging device, method for calibrating imaging system including arrangement of imaging devices, and imaging system
CN108989785B (en) Naked eye 3D display method, device, terminal and medium based on human eye tracking
CN110930463B (en) Method and device for calibrating internal reference of monitoring camera and electronic equipment
CN113034612B (en) Calibration device, method and depth camera
CN111798521A (en) Calibration method, calibration device, storage medium and electronic equipment
US20130331145A1 (en) Measuring system for mobile three dimensional imaging system
CN113763478A (en) Unmanned vehicle camera calibration method, device, equipment, storage medium and system
EP3825649A1 (en) Ranging camera
EP4054187A1 (en) Calibration method of a portable electronic device
CN114593688B (en) Three-dimensional measurement method and device based on AR (augmented reality) glasses, AR glasses and storage medium
Peer et al. Where physically is the optical center?
CN113194173A (en) Depth data determination method and device and electronic equipment
CN113834278A (en) Food material data analysis method and device, storage equipment and storage medium
US8724090B2 (en) Position estimation system

Legal Events

Date Code Title Description
AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING INC., NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNORS:SMART TECHNOLOGIES ULC;SMART TECHNOLOGIES INC.;REEL/FRAME:030935/0848

Effective date: 20130731

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNORS:SMART TECHNOLOGIES ULC;SMART TECHNOLOGIES INC.;REEL/FRAME:030935/0879

Effective date: 20130731

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: RELEASE OF ABL SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040711/0956

Effective date: 20161003

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: RELEASE OF ABL SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040711/0956

Effective date: 20161003

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: RELEASE OF TERM LOAN SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040713/0123

Effective date: 20161003

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: RELEASE OF TERM LOAN SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040713/0123

Effective date: 20161003

AS Assignment

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040798/0077

Effective date: 20161003

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040798/0077

Effective date: 20161003

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040819/0306

Effective date: 20161003

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040819/0306

Effective date: 20161003