USRE40153E1 - Multi-touch system and method for emulating modifier keys via fingertip chords - Google Patents

Multi-touch system and method for emulating modifier keys via fingertip chords Download PDF

Info

Publication number
USRE40153E1
USRE40153E1 US11/140,529 US14052905A USRE40153E US RE40153 E1 USRE40153 E1 US RE40153E1 US 14052905 A US14052905 A US 14052905A US RE40153 E USRE40153 E US RE40153E
Authority
US
United States
Prior art keywords
modifier
chord
zone
touch
zones
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime, expires
Application number
US11/140,529
Inventor
Wayne Carl Westerman
John Greer Elias
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Apple Inc
Original Assignee
Apple Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US09/681,178 external-priority patent/US6570557B1/en
Application filed by Apple Inc filed Critical Apple Inc
Priority to US11/140,529 priority Critical patent/USRE40153E1/en
Assigned to APPLE INC. reassignment APPLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: FINGERWORKS, INC.
Application granted granted Critical
Publication of USRE40153E1 publication Critical patent/USRE40153E1/en
Adjusted expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048Indexing scheme relating to G06F3/048
    • G06F2203/04808Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen

Definitions

  • the present invention pertains to multi-touch input systems and methods, and more particularly to a mixture of chord keying, gesture recognition and touch typing techniques.
  • chord keying schemes that map each letter of the alphabet or even shorthand word parts to a different finger combination. This allows chord keyboards to have a reduced number of keys, often limited to a home row of keys. This in turn reduces finger travel and potentially speeds typing.
  • Some schemes like U.S. Pat. No. 5,281,966 to Walsh, adopt a mapping that is sensibly organized so as to be easy to learn and remember, while others, such as U.S. Pat. No. 5,642,108 to Gopher et al., emphasize long-term keying performance by assigning the most frequently entered letters of the alphabet to those finger combinations that are quickest and easiest to perform. In U.S. Pat. No.
  • McClound discloses a scheme for communicating with three-finger chords.
  • a touch of the index finger on one of the nine regions of a small selector pad can be modified by thumb and/or middle finger presses on switch pads adjacent to the selector pad.
  • Non-chordic touch typing on surfaces that provide limited tactile feedback presents its own difficulties. If the typist is not careful, the hands or individual fingers tend to drift out of alignment with the key layout, or more particularly with the home row of keys where hands normally rest. Reaching for punctuation and modifier keys located on the periphery of QWERTY computer keyboard layouts exacerbates this drift. Though the Shift modifier key is not particularly far from the home row keys, the direction of pinky motion needed to reach Shift strongly pulls the other fingertips off their alignment with home row. Since the Shift modifier key must be reached so frequently to capitalize words, even typists using mechanical keyboards have long complained about the awkward pinky twist and ulnar deviation at the wrist necessary to hold it down. Accurately, hitting the Shift keys becomes, if anything, more awkward on a relatively smooth surface that does not give like a mechanical key.
  • modifier keys such as Shift, Ctrl, and Alt are often allocated to the thumbs (e.g. U.S. Pat. No. 5,642,108 to Gophert et al. and U.S. Pat. No. 5,689,253 to Hargreaves et al.) or to palm presses, as in U.S. Pat. No. 5,017,030 to Crews.
  • thumbs e.g. U.S. Pat. No. 5,642,108 to Gophert et al. and U.S. Pat. No. 5,689,253 to Hargreaves et al.
  • palm presses as in U.S. Pat. No. 5,017,030 to Crews.
  • this invention introduces four-fingertip modifier chords to eliminate the hand twist and reach traditionally required to activate modifier keys. Simultaneously dropping the four long fingertips of a hand into a modifier zone on or near the home row keys applies the Shift modifier to subsequent typing or pointing input so long as any finger from the modifier chord remains touching the surface. Typically, then, the modifier will apply to activity by the opposite hand, but the present invention also lets a hand modify its own typing, thereby allowing capitalization of whole words, if at least one of its modifier chord fingertips remains touching as others lift to strike nearby keys.
  • the four-fingertips (excluding the thumb) chord is preferred for this role because it is the easiest to drop and hold on the surface besides the five-finger chord, which must be reserved for hand resting.
  • the modifier press signal is not sent to the host computer immediately upon detection of the modifier chord touchdown.
  • the modifier press will only be sent, commiting the modifier, upon detection of modifiable input activity by other than the thumb of the modifying hand.
  • Modifiable input activity can include any user action that produces a keypress, pointing, dragging, clicking or other command for the host computer, but does not typically include resting touches that cause no signals to be output. Any touch by the modifying hand's thumb detected before commit will immediately cancel the modifier chord, effectively turning it into a hand resting chord.
  • thumb touches after the modifier press or commit need not permanently cancel the modifier.
  • these thumb touches represent editing keys such as Space or BackSpace keypresses
  • the Shift modifier signal may release temporarily while the thumb key is transmitted since the typist is most likely just erasing or putting a normal space between two capitalized words.
  • Restricting the Shift modifier chord to a zone along home row encourages typists to return their hands to the home typing position. Furthermore, this allows a Ctrl modifier zone to be established along the row of keys above home row, an easy stretch from home row.
  • a third modifier zone can be established along the row of keys below home row for rarer modifiers such as Alt, Windows, Open Apple, or Meta. Even a fourth modifier zone is possible approximately two key rows below home row. Note that all of these modifier zones can be reached through straight flexion or extension of the fingers from their home row position-absolutely no twisting or rotation of the wrist or fingers is necessary.
  • multiple modifiers are activated by the same hand simultaneously when the fingertips of the modifier chord are clearly distributed into different modifier zones.
  • the vertical offset of each finger is measured relative to the home row key the finger normally rests upon.
  • Accidental activation of a multiple zones is prevented by checking for a minimum interval between the vertical offsets of fingertips in different zones. If this condition is not met, the average of the vertical fingertip offsets is used to choose a single modifier zone. Distributing the fingertips into different zones does imply some finger twisting, but does not cause as much hand drift as reaching for multiple modifier keys on the periphery of the key layout.
  • different modifiers can be activated in a manner independent of any zones or the overall hand position on the surface. Rather, they are distinguished by different horizontal and vertical separations between the four fingertips performing the chord. Shift, for example, might be activated by the normal relaxed placement of four fingertips in a row with about 2 cm (3 ⁇ 4′′) separating the fingertip centers. Ctrl would then be activated by placing the fingertips stretched along the row with an average 3 cm (11 ⁇ 8′′) separation between them.
  • a third modifier could be activated by splitting the fingertips vertically into two rows a couple cm apart. This aspect of the invention is most useful for non-typing situations where hand motions are not focused around a default position along home row. With this aspect, a hand can, for instance, apply different modifiers to mouse clicking activity on the opposite hand without having to reposition itself within certain modifier zones.
  • a primary objective of the present invention is to provide an apparatus capable of detecting four-fingertip modifier chords that obviate the awkward pinky or thumb reaches previously needed to strike and hold modifier keys.
  • Another objective of this invention is to prevent spurious modifier chord activation when the user is slowly relaxing into a hand resting chord, but does not initially have the thumb on the surface.
  • a further objective of this invention is to allow use of the Shift modifier chord for capitalization across words without applying the Shift modifier to intervening Space or BackSpace key activations by thumbs.
  • Yet another objective of this invention is to establish different modifier zones across, above, and below the home row of keys that can be utilized to apply different modifiers.
  • a further objective of this invention is to support simultaneous activation of multiple modifiers with the same hand when the fingertips of the modifier chord are clearly distributed among different modifier zones.
  • Another objective of this invention is to support selection of different modifiers from the relative arrangement of fingertips within a modifier chord rather than their placement within any particular zone on the surface.
  • FIG. 1 is a block level diagram of a multi-touch system with typing, chord motion, and modifier chord recognition according to a preferred embodiment of the present invention.
  • FIG. 2 is a diagram of a split QWERTY key layout with prior art modifier keys that are struck by the pinky or thumb.
  • FIG. 3 is a diagram of a split QWERTY key layout with a different chord modifier zone overlapping each row of keys according to a preferred embodiment of the present invention.
  • FIG. 4 is a data flow data diagram showing the relation to the processes and data that the present invention uses to detect and apply modifier chords in its preferred embodiment.
  • FIG. 5 illustrates the contents of the touch data structure used to store touch location, identity and timing and, when formed into a chain, represent a sequence of touches.
  • FIG. 6 illustrates the contents of the data structure used to keep track of a modifier chord's state through its life cycle of being canceled or applicable and eventually lifted.
  • FIG. 7 illustrates the configuration parameters used in the preferred embodiment to represent a single modifier zone.
  • FIG. 8 is a flowchart of the process that detects new modifier chords.
  • FIG. 9 is a flowchart detailing computation of fingertip vertical offsets from the default fingertip locations.
  • FIG. 10 is a flowchart of the preferred embodiment of the process that detects which modifier zone(s) a chord is selecting.
  • FIG. 11 contains touching timing diagrams showing preferred system responses to thumb touches during modifier chord performance.
  • FIG. 12 contains touch timing diagrams that demonstrate sustain of a modifier chord throughout typing touches by the modifier chord fingertips.
  • FIG. 13 contains touch timing diagrams for cases in which a modifier chord should be allowed to apply to touch activity that actually occurs slightly before its touchdown or after its liftoff.
  • FIG. 14 is a flowchart of the process that detects modifier chord cancellation and/or liftoff according to the present invention.
  • FIG. 15 is a flowchart of the process that searches for modifier chords or keys applicable to modifiable input activity at a given time.
  • FIG. 16 shows several distinct fingertip arrangements that can be configured to select and apply different modifiers in an alternative to modifier zones.
  • FIG. 17 illustrates alternative configuration parameters that can encode a template for a recognizable arrangement of fingers within a modifier chord.
  • FIG. 18 is a flowchart of the alternative modifier selection process that finds the fingertip arrangement template that most closely matches the fingertip arrangement within a performed chord irrespective of absolute hand position on the surface.
  • the typing recognition methods of this invention are utilized within a multi-touch system like that shown in FIG. 1 .
  • the sensor scanning hardware 6 detects touches by fingers 2 on the touch surface 4 .
  • the proximity image formation 8 and contact tracking and identification 10 modules determine the touch timing and surface coordinates and report these to the typing recognizer 12 .
  • the typing recognizer decides which keys the user intended to press and tells the host communications interface 16 to send those keys to the host computer 18 .
  • the chord motion recognizer module 14 that interprets lateral sliding of multiple fingers as pointing or gesture input and effectively disables the typing recognizer for such touches.
  • the synchronization detector 13 searches for simultaneous presses or releases of multiple fingers, thereby aiding in detection of chord slides, chord taps, resting hands, and, for the purposes of this invention, modifier chords.
  • Prior art embodiments of all modules in FIG. 1 except the MODCHORD subdivisions 15 are described in related U.S. pat. app. Ser. No. 09/236,513 by Westerman and Elias. That application is incorporated herein by reference in its entirety. It discloses techniques for detection of chord taps and generation of single commands or button clicks therefrom. It also discloses methods to detect and apply conventional modifier key touches.
  • modifier chord recognition method could be utilized with any sensing device that accurately reports the lateral position of multiple fingertips on a surface.
  • the modifier chord recognition software need not reside within the sensing device. It could just as easily execute within the host computer system, or the host computer system and sensing device might be combined such that the same microprocessor executes finger tracking, modifier chord recognition, and user application software.
  • some keyboard interfacing protocols use edge-signaling of key activation state while others use level-signaling.
  • keyboards with the legacy PS/2 interface for IBM-compatible PCs will transmit a press keycode only upon initial activation of a modifier keyswitch and will send a corresponding release keycode immediately after the finger lifts off the switch.
  • the press/release keycodes are only transmitted at edges or transitions in the state of the keyswitch.
  • Keyboards communicating via the more recent USB (Universal Serial Bus) protocol use level-signaling: as long as a keyswitch is depressed, the keyboard regularly and repeatedly sends the corresponding keycode to the host computer. There are no distinct press and release keycodes.
  • edge-signaling press/release terminology throughout to describe transmission of modifiers to the host computer 18 , but it will be apparent to those of ordinary skill in the art how any edge-signaling implementation of the host communications interface 16 can be converted to level-signaling and remain well within the scope of this invention.
  • the key layout illustrated in FIG. 2 exemplifies prior art placement of modifier keys 20 , 21 , and 22 such that they can be reached by the thumb or pinky fingers.
  • This diagram shows a QWERTY layout split into left 23 and right 24 halves such that Ctrl and Alt modifier keys 22 can be placed between the split halves within easy reach of either thumb.
  • Non-split layouts are more likely to place Ctrl and Alt modifier keys 21 on the bottom row of the layout where they are most likely to be operated by the pinkies.
  • most layouts keep the Shift modifier keys 20 diagonally below the home row key (‘A’ or ‘;’) that the pinky normally rests upon. Though these Shift keys 20 are not that far from home row, the pinky motion needed to reach and hold them down is particularly awkward.
  • FIG. 3 shows the chord modifier zones 30 - 34 and 35 - 37 of the present invention as different hatches ranging across each row of the key layout.
  • the different modifier zones would be indicated on the touch surface by different background colors rather than different hatches.
  • simultaneous touchdown of four fingertips within one of these zones will cause subsequent touch activity to be modified just as if a modifier key was being held as long as the modifier chord is not fully lifted or canceled by an improper thumb touch.
  • a hand resting on the home row ‘ASDF’ or ‘JKL;’ keys can activate any of its modifier zones by pure finger flexion or extension, eliminating the awkward twists needed to reach conventional modifier keys.
  • modifier zones are not horizontally restricted to the ‘ASDF’ or ‘JKL;’ key columns, so the typist need not worry about horizontal hand alignment when performing a modifier chord.
  • the key layout improved with modifier zones may retain the conventional Shift modifier keys 20 near the pinky for the convenience of novices who have not yet learned to perform modifier chords, but experience with the invention has shown that typists rarely use these conventional modifier keys once they have learned the more convenient modifier chords. Since capitalization is needed so frequently, the modifier zones 31 and 36 along the left and right home row of keys are preferably assigned the Left Shift and Right Shift modifiers.
  • the modifier zones 30 and 35 above home row are preferably assigned the Left Ctrl and Right Ctrl modifiers, and the zones 32 and 37 directly below home row are preferably assigned the Left Alt and Right Alt modifiers.
  • Modifier zones 33 and 38 can be assigned any remaining modifiers such as Meta, Diamond, Windows, or Open apple supported by the host computer 18 's particular operating system.
  • modifier zones can be established just as easily over Dvorak, non-split, or non-English key layouts.
  • the modifier zones could alternatively be assigned to Asian language modifiers like Kana or Kanji and remain well within the scope of this invention.
  • modifier chord recognition is split into three processes as shown in FIG. 4 .
  • Process 56 detailed in FIG. 8 , detects new modifier chords by monitoring the incoming touch sequence 50 for touchdowns by the four fingertips of one hand that are simultaneous and also lie within one of the zones or arrangements specified by the configuration data 54 . This process only needs to execute each time a new finger touch is detected and appended to the touch sequence, not for image frames from the sensor scanner 6 that contain no new surface contacts.
  • the incoming touch sequence 50 consists of a chain of touch data structures 80 (detailed in FIG. 5 ) ordered by touchdown times 85 .
  • the configuration data structures 100 and 520 that store the modifier zones or arrangements 54 are detailed in FIG. 7 and FIG. 17 .
  • process 56 will allocate a corresponding modifier state data structure 90 (detailed in FIG. 6 ) and appended it to the modifier chord state chain 58 .
  • the liftoff/cancellation detection process 60 (detailed in FIG. 14 ) checks each new image frame or sensor array scan for absence of all four fingertips or presence of the thumb from any hand with a modifier chord pending in the state chain 58 . Upon detection of one of these conditions, the liftoff/cancellation process 60 uses feedback path 68 into the modifier state chain 58 to record either the chord liftoff time 94 or chord cancellation 96 . It may also send corresponding modifier release signals to the host computer 18 through the host communication interface 16 .
  • the third process 62 searches the modifier state chain 58 for uncancelled modifier chords roughly coincident with modifiable input activity. If this process finds any applicable modifier chords whose press signals have not already been sent, it will cause the host computer interface 16 to send their modifier press signals to the host computer 18 .
  • This search process 62 is typically called by the typing recognizer 12 , synchronization detector 13 , or chord motion recognizer 14 with the timestamp 64 of the modifiable input activity right before such activity itself is transmitted through the host communication interface 16 .
  • FIG. 5 lists basic parameters needed for each touch data structure 80 to support detection of new modifier chords 56 .
  • a ring or chain of such data structures ordered by touchdown time 85 represents a touch sequence by one or more fingers. Since palm touches are to be ignored by all recognition processes, they can be left out of the touch sequence 50 .
  • Each touch data structure 80 must contain the touch's x and y surface coordinates 82 as reported by the touch sensors 6 . These should estimate the center of the touch, which for proximity or pressure sensors is typically computed as the centroid of fingertip flesh contacting the surface. The y offset from default finger location 84 will be computed in FIG. 9 to improve the accuracy of modifier zone selections.
  • each touch data structure should have a copy of the hand and finger identity 84 estimated for the touch by the contact tracking and identification module 10 .
  • the touch data should also include the finger touchdown time 85 , also reported by the contact tracking and identification module 10 .
  • the touch liftoff time 86 is a useful indicator of individual modifier key release
  • the modifier chord liftoff time 94 is not derived from the individual liftoff times 86 of the touches originally forming the chord. This is because in its preferred embodiment the present invention beneficially allows individual fingertips of a modifier chord to lift temporarily and touch again to type while other modifier chord fingertips remain touching, as is shown in FIG. 12 .
  • These temporary finger lifts will establish the touch liftoff times 86 for the original touches forming the chord but may not be indicative of final chord liftoff since the temporarily lifted fingers may touch again to sustain the chord while other modifier fingertips lift to type, and so on.
  • FIG. 6 lists the parameters that keep track of a modifier chord's state 90 once it is initially detected by process 56 .
  • the modifier type bits 91 encodes which modifiers, e.g. Shift, Ctrl, or Alt, the chord's fingertip placement has selected. The contents of these bits will be determined by either the modifier zone selection process of FIG. 10 or the fingertip arrangement selection process of FIG. 18 .
  • the hand field 92 indicates which hand performed the modifier chord.
  • the chord touchdown time 93 is set by the new chord detection process 56 as the minimum of the touchdown times 85 of the original four fingertip touches firming the chord. As discussed previously, the chord liftoff time 94 should not be computed as the maximum of the original touch liftoff times 86 . Instead, the liftoff and cancellation check process 60 , FIG.
  • chord liftoff time 94 is set as the first time after chord detection that none of the four fingertips from the chord are found to be touching the surface.
  • the canceled flag 96 is set by the liftoff and cancellation check process 60 if the chord fingertips slide substantially or a thumb from the chord's hand touches before the chord as committed. When set, this flag causes the applicable modifier search process 62 to ignore the modifier chord.
  • the committed flag 97 is set once a modifier chord is actually applied by the applicable modifier search process 62 to outgoing typing or clicking activity. Once set, this flag prevents the chord from being canceled except when fully lifted.
  • FIG. 7 shows the data structure 100 and parameters used to configure each modifier zone.
  • the modifier type bitmask 102 is a set of flags indicating which of the Shift, Ctrl, Alt or other modifiers are assigned to the zone. Note that this bitmask implementation allows multiple modifiers, such as Shift Ctrl, to be assigned to the same zone if the user so desires.
  • the hand field 104 can restrict the zone to the left 23 or right 24 half of the key layout, but no other horizontal alignment restriction is necessary.
  • the min 106 and the max 108 Y offset from home row determine the vertical range of the modifier zone. Note that no extra configuration parameters are necessary to arch the zones along each row, should the key rows be so arched, because the zone selection process of FIG. 10 will compare these Y offsets with each fingertip's vertical offset relative to the position of the home row key the finger normally rests upon, also know as the fingertip's default position. Thus the modifier zone will automatically arch to match any arch in the home row key locations.
  • the new modifier chord detection process is shown in FIG. 8 .
  • This process begins whenever a new finger touch is detected by the contact tracking and identification module 10 .
  • Step 150 increments the latest touch index n and stores the touch's parameters 80 at the n th location of the touch sequence array T[ ].
  • Steps 152 , 154 , and 156 scan backward m touches in the touch sequence trying to find the largest synchronized subsequence that includes the new touch T[n].
  • Decision diamond 154 judges synchronization by testing whether the m th previous touch T[n ⁇ m] contacted the surface within a synchronization interval of about 60 milliseconds of the new touch T[n].
  • the typing recognizer 12 should not generate signals to the host corresponding to an individual touch over a key until sufficient time has passed without subsequent touches on the same hand that this synchronization detection loop can be certain that the touch is not synchronized with later touches. Thus the typing recognizer 12 must delay key output about 60 ms from finger touchdown or be prepared to erase or undo keys from touches later found to be part of a chord.
  • decision diamond 154 finds a previous touch too old to be synchronized with T[n], it passes on the largest synchronized subsequence as T[n ⁇ m+1] . . . T[n].
  • Decision diamond 158 then examines the finger and hand identity 84 of each synchronized touch looking for a combination of identifies from one hand that matches any combination allowed for modifier chords. In the preferred embodiment, only the 4 fingertip combination, index, middle, ring and pinky, excluding the thumb, is used for modifier chords. To prevent duplicate detection of the same modifier chord, decision diamond 158 must require the newest touch T[n] to be one of the modifier chord fingertips. Otherwise, any synchronized touches intervening from the hand opposite a modifier chord combination do not affect the modifier chord, but the modifier chord may eventually apply its modifiers to them as in touch 370 of FIG. 13 . If the synchronized subsequence does not contain a modifier chord combination from either hand, the process returns 160 until the next new touch warrants a renewed detection attempt.
  • block 162 forms a touch array MT[ ] indexed by finger identity containing only the modifier chord touches. As further described in FIG. 9 , block 162 also computes each touch's vertical offset from its corresponding home row key or default location. Block 164 , further described in FIG. 10 , checks whether the chord has been performed within any of the established modifier zones. In an alternative embodiment, block 164 may check the arrangement of fingertips within the chord as further described in FIG. 18 . Decision diamond 170 ends new modifier detection through step 168 if the chord matches none of the established zones or arrangements.
  • step 172 allocates a new modifier state 90 , setting the modifier type 91 according to that of the selected zones or arrangement, setting the modifier hand 92 to be detected chord's hand identity, and setting the chord touchdown time 93 as the minimum of the touchdown times 85 of the synchronized touches forming the chord, and setting the chord liftoff time 94 to 0 pending full liftoff detection.
  • Step 172 also appends this new modifier state to the state chain 58 that may already contain state from other chord or key modifiers being held by the opposite hand.
  • New modifier detection returns at step 176 , and responsibility for canceling the modifier chord or applying it to modifiable input activity passes to processes 60 and 62 .
  • FIG. 9 shows the details of vertical offset computation block 162 .
  • Step 200 starts the sync index s at the oldest synchronized touch and loops 210 through the synchronized subsequence.
  • Decision diamond 204 separates out the modifier chord touches from any opposite hand touches.
  • step 206 stores each modifier chord touch into the touch array MT[ ] in order of their finger identity 84 .
  • Step 208 computes each modifier touch's vertical offset from the default or resting location of its corresponding finger. For key layouts such as 23 and 24 with an arch across the home row keys, these default locations should be the locations of the ‘ASDF’ and ‘JKL;’ keys. These vertical offsets help the modifier zone matching process 164 efficiently compensate for the natural arch across the fingertips.
  • Decision diamond 202 breaks the loop and returns through 212 once the whole synchronized subsequence has been processed.
  • FIG. 10 discloses the preferred embodiment of block 164 , a search for modifier zones 100 that the modifier chord fingertips lie within.
  • the array Z[ ] of configured zones is assumed to contain only zones for the given modifier chord's hand, and these zones are assumed to be ordered within the array from farthest below home row to farthest above.
  • the loop of steps 250 - 276 will attempt to accumulate modifier types from each zone that any of the modifier chord fingertips lies within. If, however, decision diamond 262 finds that the fingertips are not clearly spaced across zones, i.e. if two or more fingertips are vertically bunched together yet straddling the border between two zones, it will direct the loop of steps 280 - 288 to find the single zone that the average of the fingertip offsets falls within.
  • Step 250 clears selected_mods, the variable whose bits will accumulate the selected modifier types.
  • Step 252 clears the modifier zone index i.
  • Step 254 initializes the max vertical offset for fingertips found in the i th zone, Z[i].found_ymax, to the zone's minimum vertical boundary Z[i].range_ymin.
  • Step 256 initialized the modifier fingertip index f to 2, representing the index finger.
  • Decision diamond 260 checks whether touch MT[f]'s vertical offset from default is within zone Z[i]'s boundaries. If not, decision diamond 270 checks whether all fingertip touches including the pinky have been examined, and if not step 272 advances the fingertip index.
  • step 264 If touch MT[f] is within Z[i]'s vertical range, decision diamond 262 checks whether it is vertically separated by at least zone_safety_sep from any touches already found in the zone below. zone_safety_sep should be set to about 1 cm or 3 ⁇ 8′′. If the touch is clearly separated form any in the zone below, step 264 will bump up Z[i].found_ymax as necessary. Step 266 accumulates the modifier types 91 assigned to Z[i] into the selected_mods bits with a bitwise OR operation. Once all fingertips have been checked against the i th zone, decision diamond 274 will check whether all zones have been tested, and if not advance to the next zone through step 276 . If the loop 254 - 276 gets through all zones without decision diamond 262 finding a fingertip separation violation, step 278 will return the accumulated modifier types selected_mods to decision diamond 170 of FIG. 8 .
  • step 284 computes from MT[ ]the average avg_yoffset of all four fingertips' vertical offsets.
  • Decision diamond 286 checks whether this avg_yoffset is within the range of zone Z[i]. If not, step 290 advances the zone index to the next zone until either decision diamond 292 finds all zones have been exhausted or decision diamond 286 finds a matching zone. Assuming avg_yoffset falls within the vertical range of one of the zones, step 288 assigns selected_mods the modifier type(s) of that zone, and step 278 returns these. Note that step 278 will return zero if the fingertips are not within range of any zone.
  • FIGS. 11-13 These timing diagrams display the touchdown (falling edge) and liftoff (rising edge) timings for particular fingers 84 from both hands, where LF 5 denotes the left pinky, LF 1 the left thumb, RF 2 the right index finger, and so on. Fingers not shown in an example can be assumed to be lifted throughout. All synchronized chord touches 300 are assumed to fall within a Shift modifier zone. The vertical dotted lines 93 and 94 identify the chord touchdown time and liftoff time, respectively.
  • Keys transmitted to the host by the typing recognizer 12 in response to asynchronous touches are denoted by the key's symbol encircled.
  • the key symbol is only capitalized if the Shift modifier chord applies to it.
  • the slight space drawn between each touchdown and the corresponding key character circle simply indicates that the typing recognizer must wait about 60 ms after each touchdown before sending a keypress to ensure that the touch will not be part of a chord.
  • the left and right ends of SHIFT ellipses 318 , 319 , 338 , 339 , 348 , 378 , and 388 in the MODIFIER 66 row demarcate the modifier press and release signals that should be sent to the host computer.
  • the bottom two rows of each timing diagram indicate setting of the modifier state canceled flag 96 or committed flag 97 in response to certain touches.
  • FIG 11 A shows typing before, during, and after performance of a left hand Shift chord.
  • the initial ‘a’ key touch 310 clearly precedes modifier chord touchdown 93 and so should not be capitalized. Notice that the Shift modifier is not sent to the host, committing the chord 469 , until the ‘J’ touch 311 needs to be sent to the host computer, well after modifier chord touchdown. Committing of modifier chords should be thus delayed until transmission of subsequent modifiable activity (the ‘J’ key touch in this case) to allow for the possibility that the 4 fingertips will begin sliding, suggesting that the typist is actually trying to scroll, or be supplemented with a thumb touch, suggesting the typist is just resting the hand sloppily.
  • This subsequent-touch-activity-dependence of modifier chord committing is unique and novel, as the chord and key taps of the related art commit either on liftoff or sufficient touching time of the tapping fingers themselves, while chord slides for pointing and command gestures commit upon significant lateral motion of the involved fingers.
  • the ‘I’ touch 312 is also capitalized, but the Shift modifier is temporarily released while the BackSpace key touch 302 is sent to the host, and then Shift is pressed again at 319 in time for the ‘U’ key touch 313 .
  • Decision channel 463 of FIG. 15 will handle such temporarily release of thumb editing keys such as BackSpace and Space under the assumption that typists usually intend to make quick, unshifted edits but are too lazy to lift and retouch the Shift chord before and after the thumb key activations.
  • the ‘n’ key touch 314 clearly follows modifier chord liftoff, and therefore should not be capitalized.
  • FIG. 11B demonstrates the desired system behavior when the modifying hand's thumb touches down before typing or clicking activity amongst the other fingers has a chance to commit the modifier chord.
  • the touches 320 and 323 before and after modifier chord performance do not get capitalized.
  • the ‘i’ and ‘n’ key touches 321 and 322 concurrent with the modifier chord do not get capitalized either because the thumb touch 302 preceding them causes the modifier chord to be canceled 404 .
  • Whether the thumb touch lands properly on and activates the BackSpace key or whether it lifts back off before the ‘i’ and ‘n’ touches is not important.
  • the thumb touch 302 is quite likely the result of sloppy hand resting, and should cancel the modifier chord lest subsequent typing or clicking be shifted unintentionally and have to be undone.
  • Decision diamond 402 of FIG. 14 will implement cancellation due to resting thumb touches.
  • FIGS. 12A and 12B demonstrate how a modifier chord can be sustained through typing by its fingers on its half of the key layout so that whole words can be capitalized from a single modifier chord touchdown.
  • the ‘K’key touch 330 on the opposite hand quickly commits 469 the modifier chord and the ‘I’ touch 331 follows.
  • the typist need only lift the middle and ring fingertips of the modifier chord and set them down 332 and 333 on those keys one at a time.
  • the typing recognizer must be configured to generate keys from asynchronous touches without waiting for touch liftoff for this to work well, as at least one of the resumed touches 332 and 333 will need to sustain one of the resumed touches 332 and 333 will need to sustain the modifier chord while the pinky and index of the modifier chord temporarily lift to touch the ‘A’ 335 and ‘T’ 336 keys.
  • Chord liftoff 94 does not register and cause release of the modifier 349 until the first moment when none of the modifier fingertips are touching.
  • the right thumb Space key touch 334 invokes temporary modifier release as in FIG. 11 A.
  • FIG. 12B is similar to FIG. 12A except that the modifier chord is performed by the right hand and committed 469 by a temporary liftoff and ‘P’ touchback 340 of one of its own pinky fingertip.
  • Touches 341 - 343 walk through the other modifier chord fingertips ending with a ‘T’ touch 344 by the opposite hand to spell ‘POINT’. This demonstrates that a modifier chord need not always be committed by the opposite hand, which can be useful in situations where only one hand is available for interaction with the touch surface.
  • FIGS. 13A and 13B demonstrate cases where it is helpful for the system to tolerate some inaccuracies in the timing between modifier chord touches and modifiable key touches. The same techniques also apply to individual modifier key touches on a surface.
  • the touch 370 actually precedes the modifier chord touchdowns 300 but is roughly synchronous with them. Assuming the modifier chord is detected before the key from touch 370 gets sent to the host, the modifier chord should apply to and capitalize the ‘T’.
  • Such handling of key touches that very slightly precede modifier chord touches is important for typists who get in the habit of simultaneously touching both the modifier chord and the opposite hand's key to be capitalized.
  • the system will implement it through the sync_slack term of decision diamond 456 in FIG. 15 .
  • the ‘T’ is capitalized again for touch 371 but not for ‘t’ touch 372 , which follows chord liftoff.
  • FIG. 13B shows a brief tap of a modifier chord without any coincident key touches.
  • ‘T’ touch 382 does follow shortly after chord liftoff. This suggests the typist was intending to capitalize a single key but, in performing the modifier chord so quickly, synchronized it poorly with the key touch 382 on the opposite hand.
  • FIG. 14 details process 60 , thumb cancellation and chord liftoff detection.
  • Step 400 denotes that whenever any finger touchdown or liftoff is detected on the surface, this process must be repeated for each modifier chord state 90 in the state chain 58 .
  • the combination of decision diamonds 401 and 402 causes the chord to be canceled in step 404 if a thumb touch by the modifier chord's hand is detected before the chord's committed flag gets set.
  • Decision diamond 402 may also decide to cancel a chord if excessive sliding is detected amongst the chord's fingertips. If no such canceling thumb touch is detected, decision diamond 406 checks whether chord liftoff has previously been detected. If not, decision diamond 408 checks the latest sensor image frame 52 for any touches by the modifier chord fingertips.
  • step 410 sets the chord liftoff time 94 to the timestamp of the latest sensor image. If liftoff was just detected and decision diamond 412 finds that the modifier types 91 for this modifier state MS are currently pressed for the host, step 414 instructs the host communications interface 16 to send the corresponding modifier release signals. Steps 404 and 410 , if executed, thus constitute the feedback path 68 from this process 60 to the modifier state chain 58 .
  • FIG. 15 details the search for applicable modifiers 62 that should always execute just before the typing recognizer 12 , synchronization detector 13 , or chord motion recognizer 14 transmit any typing, pointing, or clicking signals to the host computer 18 .
  • These modules will pass to the search process the modifiable tstamp parameter, set to the current system time for pointing activity but otherwise to the original touchdown time of the key or click chord touches.
  • Step 450 clears the applicable_mods variable, which will accumulate the bitmasks of the types of modifiers found applicable.
  • Step 452 initializes the modifier state reference MS to the most recent modifier state 90 in the chain 58 .
  • Decision diamond 456 causes canceled 404 chords to be excluded from the applicable modifier search. At this point the modifier state for a canceled chord can, if memory resources are scarce, be entirely removed from the state chain 58 .
  • decision diamond 456 ensures that modifiable_tstamp does not precede the chord touchdown time 93 by more than a sync_slack of about 60 ms. If the chord more or less precedes the modifiable activity, step 460 establishes a 0 ms default lift_slack. If the modifier chord has only been touching briefly and is not yet committed, decision diamond 462 will cause step 464 to establish a lift_slack of about 100-150 ms, allowing for late modifiable touches as in FIG. 13 B.
  • decision diamond 463 checks whether the modifiable activity is a thumb keypress, and if so, should prevent the modifier chord from applying at least if the chord is a Shift modifier. This will also cause temporary Shift modifier release during thumb key transmission as in FIG. 11 A and FIG. 12 A. Note that this-check is not done for briefly touching chords that have not committed yet so as to allow the typist to perform, for example, a single Shift BackSpace macro command using a modifier chord from the hand opposite the thumb key.
  • step 469 will set the modifier state's committed flag 97 , and step 468 will accumulate the modifier types selected by the applicable modifier state.
  • step 469 constitutes the main feedback path 70 from the applicable modifier search process 62 to the modifier state chain 58 .
  • Decision diamond 471 checks whether any older modifier states that might be applicable are left in the state chain 58 , and if so step 470 continues the search on the previous modifier state. Once the entire state chain has been examined, the process returns at step 472 the modifier types from the modifier chords found applicable. The host computer interface 16 will compare these newly applicable modifier types to a bitmask of the modifier types already pressed and send additional modifier press or release signals to the host computer as necessary.
  • chord modifier recognition method could easily be adapted to keyboards with conventional mechanical keyswitches, assuming the keyswitch matrix was wired such that the keyswitch scanner circuitry could reliably distinguish coincident presses of four home row keys forming the chord as well as one other key to be modified.
  • such an adaptation would not be notably advantageous because the effort needed to press four mechanical homerow keyswitches simultaneously is just about as straining as the awkward pinky reach for a conventional Shift key.
  • modifier chords are utilized with a proximity sensing multi-touch surface as in the preferred embodiment, holding the chord requires no effort giving modifier chords significant advantage over the pinky reach for a conventional Shift key.
  • FIG. 16 shows six sets of chord touches that are distinguishable not by their finger combination or alignment with respect to the key layout, but by the relative arrangement of the four fingertips index 502 through pinky 505 within the chord. The arrangements shown would most likely arise from right hand chord performances; left hand chord performances would produce the mirror image of these arrangements. Each arrangement can be assigned a different modifier type to allow a wide range of modifier selections anywhere on the touch surface.
  • FIG. 16A is most relaxed and is thus preferred for the Shift modifier.
  • the arrangement of FIG. 16B is preferably mapped to Ctrl, FIG. 16E to Alt, FIG. 16C to Ctrl Shift, and the remaining arrangements to any other modifiers useful on a particular host computer.
  • the crosshairs 500 indicate the centroid of each fingertip arrangement and will be used as the coordinate origin when storing the arrangements as templates.
  • FIG. 17 shows a configuration data structure 520 suitable for storing fingertip arrangement templates and associating them with particular modifier types.
  • configuration data structure 520 is an alternative to the modifier zone embodiment 100 of configuration data 54 . It has a modifier type bitmask 102 and modifier hand source 104 equivalent to the modifier zone data structure 100 .
  • Each data structure 520 encodes one of the fingertip arrangements from FIG. 16 or the like as a template of four x offsets 526 and four y offsets 528 from the chord center 500 . Encoding them as offsets from the centroid 500 of the 4 fingertip locations will help template matching in FIG. 18 be neutral to absolute hand position on the surface.
  • the pattern matching process of FIG. 18 selects the template arrangement that best matches the measured arrangement of fingertips performing a modifier chord.
  • This arrangement matching process is thus an alternative to FIG. 10 's modifier zone selection embodiment of block 164 .
  • An array AR[ ] of the template data structure 520 constitutes the data 54 .
  • Step 550 clears selected mods, the variable whose bits will hold the selected modifier types.
  • Step 552 initializes min_sumdist 2 , the variable that will hold the lowest match error of evaluated templates, to a very large number.
  • Step 555 compares the average or centroid, avg_fpos, of the four fingertip locations from the MT[ ] modifier touch array.
  • Step 554 clears the arrangement template index i.
  • Step 556 clears the squared error sum_dist 2 for the current template AR[i].
  • Step 558 initializes the finger identity index to 2 indicating the index finger.
  • Steps 560 - 564 then loop through to the pinky accumulating the squared error between each fingertip's offset from chord center and the corresponding template point's offset from chord center.
  • Decision diamond 566 checks whether the squared error for template AR[i] make it the best match so far, and if so step 568 updates min_sumdist 2 and step 570 stores template AR[i]'s modifier type 522 as the modifier selected so far.
  • Decision diamond 572 determines whether all template arrangements have been tested, and if not step 574 advances the index i to the next template. Once all arrangements have been tested, step 576 will return the modifier type 522 of the template 520 which most closely matches the measured arrangement of modifier touches MT[ ].
  • the four fingertip combination is both easiest to perform and by far most compatible with the typing, hand resting, and pointing activities that can also take place on a multi-touch surface.
  • Using the thumb and three fingertips is a bit more awkward and would necessitate cancellation when the pinky, or in general, a fifth digit touched the surface before commit by other modifiable input activity.
  • Using three fingertips for modifier chords would necessitate cancellation when either the thumb or a fourth fingertip from the modifier chord hand touched the surface before commit by modifiable activity on the opposite hand.
  • Three fingertips modifier chords would also be incompatible with drag operations that are preferably assigned to three-fingertips on a multi-touch surface.

Abstract

A multi-touch system is disclosed that recognizes simultaneous touchdown of four fingers on, above, or below the home row of keys as a modifier chord and applies modifiers such as Shift, Ctrl, or Alt to subsequent touch activity until none of the chord fingertips remain touching. Touches by the thumb of the modifier chord hand that occur before any modifiable typing or clicking activity cause the modifier chord to be canceled and reinterpreted as hand resting. The Shift modifier may be released temporarily during thumb keypresses that are intermixed with typing of capitalized characters. Distributing the modifier chord touches across different zones or key rows selects multiple modifiers. In an alternative embodiment, different modifiers can be selected with different arrangements of the fingers relative to one another within the chord, irrespective of absolute hand alignment with the touch surface.

Description

BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention pertains to multi-touch input systems and methods, and more particularly to a mixture of chord keying, gesture recognition and touch typing techniques.
2. The Related Art
The primary use of chords, or simultaneous finger presses, within the data entry art has been in chord keying schemes that map each letter of the alphabet or even shorthand word parts to a different finger combination. This allows chord keyboards to have a reduced number of keys, often limited to a home row of keys. This in turn reduces finger travel and potentially speeds typing. Some schemes, like U.S. Pat. No. 5,281,966 to Walsh, adopt a mapping that is sensibly organized so as to be easy to learn and remember, while others, such as U.S. Pat. No. 5,642,108 to Gopher et al., emphasize long-term keying performance by assigning the most frequently entered letters of the alphabet to those finger combinations that are quickest and easiest to perform. In U.S. Pat. No. 5,808,567, McClound discloses a scheme for communicating with three-finger chords. In this system, a touch of the index finger on one of the nine regions of a small selector pad can be modified by thumb and/or middle finger presses on switch pads adjacent to the selector pad.
The recent development of multiple-touch sensitive surfaces that lack the restrictions of distinct mechanical keys warrants a reexamination of chording schemes. Direct adaptation of the chord keying schemes cited above to a multi-touch surface certainly seems feasible, but may not be desirable. U.S. Pat. No. 5,825,352 to Bisset et al. describes a touchpad with row and column electrodes that produces pointing in response to single finger motion and dragging in response to two finger motion. U.S. Pat. No. 6,107,997, Ure utilizes the touch sensor array of U.S. Pat. No. 5,194,862 and interprets single finger motions as pointing while interpreting various placements of a 2-finger chord on a grid as key entry. In U.S. application Ser. No. 09/236,513, however, Westerman and Elias take yet another approach, interpreting asynchronous touches on a multi-touch surface (MTS) as conventional single-finger typing while interpreting motions initiated by chords as pointing, clicking, and other gesture commands. We prefer this approach for the following reasons: learning a few new chords for graphical manipulation is much easier than learning a slew of new chords for typing the whole alphabet, and graphical manipulation seems a better use of chords in today's graphics intensive computing environment. In dictation situations where greater text entry speeds are needed than can be achieved with non-chordic keying, adopting a continuous speech recognition system for text entry is becoming more practical than learning a chord keying technique.
Non-chordic touch typing on surfaces that provide limited tactile feedback presents its own difficulties. If the typist is not careful, the hands or individual fingers tend to drift out of alignment with the key layout, or more particularly with the home row of keys where hands normally rest. Reaching for punctuation and modifier keys located on the periphery of QWERTY computer keyboard layouts exacerbates this drift. Though the Shift modifier key is not particularly far from the home row keys, the direction of pinky motion needed to reach Shift strongly pulls the other fingertips off their alignment with home row. Since the Shift modifier key must be reached so frequently to capitalize words, even typists using mechanical keyboards have long complained about the awkward pinky twist and ulnar deviation at the wrist necessary to hold it down. Accurately, hitting the Shift keys becomes, if anything, more awkward on a relatively smooth surface that does not give like a mechanical key.
In the related ergonomic and chord keyboard art exemplified by FIG. 2 modifier keys such as Shift, Ctrl, and Alt are often allocated to the thumbs (e.g. U.S. Pat. No. 5,642,108 to Gophert et al. and U.S. Pat. No. 5,689,253 to Hargreaves et al.) or to palm presses, as in U.S. Pat. No. 5,017,030 to Crews. However, for a multi-touch surface, reaching the thumb for modifier keys poses the same drift exacerbation problems as reaching by the pinky, and palm touches should be ignored to encourage hand resting. Thus there exists a need in the multi-touch and chord keying art for alternative methods to activate modifier keys without drawing any fingers away from the row.
BRIEF SUMMARY OF THE INVENTION
In its primary aspect, this invention introduces four-fingertip modifier chords to eliminate the hand twist and reach traditionally required to activate modifier keys. Simultaneously dropping the four long fingertips of a hand into a modifier zone on or near the home row keys applies the Shift modifier to subsequent typing or pointing input so long as any finger from the modifier chord remains touching the surface. Typically, then, the modifier will apply to activity by the opposite hand, but the present invention also lets a hand modify its own typing, thereby allowing capitalization of whole words, if at least one of its modifier chord fingertips remains touching as others lift to strike nearby keys. The four-fingertips (excluding the thumb) chord is preferred for this role because it is the easiest to drop and hold on the surface besides the five-finger chord, which must be reserved for hand resting.
Since the four-fingertip chord is also preferred for window scrolling, and since it is often a prelude to dropping the thumb into the full five-finger hand resting chord, the present invention takes special precautions to prevent accidental modifier activation. The modifier press signal is not sent to the host computer immediately upon detection of the modifier chord touchdown. The modifier press will only be sent, commiting the modifier, upon detection of modifiable input activity by other than the thumb of the modifying hand. Modifiable input activity can include any user action that produces a keypress, pointing, dragging, clicking or other command for the host computer, but does not typically include resting touches that cause no signals to be output. Any touch by the modifying hand's thumb detected before commit will immediately cancel the modifier chord, effectively turning it into a hand resting chord. Such thumb touches after the modifier press or commit need not permanently cancel the modifier. However, if these thumb touches represent editing keys such as Space or BackSpace keypresses, the Shift modifier signal may release temporarily while the thumb key is transmitted since the typist is most likely just erasing or putting a normal space between two capitalized words.
Restricting the Shift modifier chord to a zone along home row encourages typists to return their hands to the home typing position. Furthermore, this allows a Ctrl modifier zone to be established along the row of keys above home row, an easy stretch from home row. A third modifier zone can be established along the row of keys below home row for rarer modifiers such as Alt, Windows, Open Apple, or Meta. Even a fourth modifier zone is possible approximately two key rows below home row. Note that all of these modifier zones can be reached through straight flexion or extension of the fingers from their home row position-absolutely no twisting or rotation of the wrist or fingers is necessary.
According to the present invention, multiple modifiers are activated by the same hand simultaneously when the fingertips of the modifier chord are clearly distributed into different modifier zones. To compensate for the natural arch in a row of fingertips, the vertical offset of each finger is measured relative to the home row key the finger normally rests upon. Accidental activation of a multiple zones is prevented by checking for a minimum interval between the vertical offsets of fingertips in different zones. If this condition is not met, the average of the vertical fingertip offsets is used to choose a single modifier zone. Distributing the fingertips into different zones does imply some finger twisting, but does not cause as much hand drift as reaching for multiple modifier keys on the periphery of the key layout.
In an alternative embodiment of this invention, different modifiers can be activated in a manner independent of any zones or the overall hand position on the surface. Rather, they are distinguished by different horizontal and vertical separations between the four fingertips performing the chord. Shift, for example, might be activated by the normal relaxed placement of four fingertips in a row with about 2 cm (¾″) separating the fingertip centers. Ctrl would then be activated by placing the fingertips stretched along the row with an average 3 cm (1⅛″) separation between them. A third modifier could be activated by splitting the fingertips vertically into two rows a couple cm apart. This aspect of the invention is most useful for non-typing situations where hand motions are not focused around a default position along home row. With this aspect, a hand can, for instance, apply different modifiers to mouse clicking activity on the opposite hand without having to reposition itself within certain modifier zones.
A primary objective of the present invention is to provide an apparatus capable of detecting four-fingertip modifier chords that obviate the awkward pinky or thumb reaches previously needed to strike and hold modifier keys.
Another objective of this invention is to prevent spurious modifier chord activation when the user is slowly relaxing into a hand resting chord, but does not initially have the thumb on the surface.
A further objective of this invention is to allow use of the Shift modifier chord for capitalization across words without applying the Shift modifier to intervening Space or BackSpace key activations by thumbs.
Yet another objective of this invention is to establish different modifier zones across, above, and below the home row of keys that can be utilized to apply different modifiers.
A further objective of this invention is to support simultaneous activation of multiple modifiers with the same hand when the fingertips of the modifier chord are clearly distributed among different modifier zones.
Another objective of this invention is to support selection of different modifiers from the relative arrangement of fingertips within a modifier chord rather than their placement within any particular zone on the surface.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block level diagram of a multi-touch system with typing, chord motion, and modifier chord recognition according to a preferred embodiment of the present invention.
FIG. 2 is a diagram of a split QWERTY key layout with prior art modifier keys that are struck by the pinky or thumb.
FIG. 3 is a diagram of a split QWERTY key layout with a different chord modifier zone overlapping each row of keys according to a preferred embodiment of the present invention.
FIG. 4 is a data flow data diagram showing the relation to the processes and data that the present invention uses to detect and apply modifier chords in its preferred embodiment.
FIG. 5 illustrates the contents of the touch data structure used to store touch location, identity and timing and, when formed into a chain, represent a sequence of touches.
FIG. 6 illustrates the contents of the data structure used to keep track of a modifier chord's state through its life cycle of being canceled or applicable and eventually lifted.
FIG. 7 illustrates the configuration parameters used in the preferred embodiment to represent a single modifier zone.
FIG. 8 is a flowchart of the process that detects new modifier chords.
FIG. 9 is a flowchart detailing computation of fingertip vertical offsets from the default fingertip locations.
FIG. 10 is a flowchart of the preferred embodiment of the process that detects which modifier zone(s) a chord is selecting.
FIG. 11 contains touching timing diagrams showing preferred system responses to thumb touches during modifier chord performance.
FIG. 12 contains touch timing diagrams that demonstrate sustain of a modifier chord throughout typing touches by the modifier chord fingertips.
FIG. 13 contains touch timing diagrams for cases in which a modifier chord should be allowed to apply to touch activity that actually occurs slightly before its touchdown or after its liftoff.
FIG. 14 is a flowchart of the process that detects modifier chord cancellation and/or liftoff according to the present invention.
FIG. 15 is a flowchart of the process that searches for modifier chords or keys applicable to modifiable input activity at a given time.
FIG. 16 shows several distinct fingertip arrangements that can be configured to select and apply different modifiers in an alternative to modifier zones.
FIG. 17 illustrates alternative configuration parameters that can encode a template for a recognizable arrangement of fingers within a modifier chord.
FIG. 18 is a flowchart of the alternative modifier selection process that finds the fingertip arrangement template that most closely matches the fingertip arrangement within a performed chord irrespective of absolute hand position on the surface.
DETAILED DESCRIPTION OF THE INVENTION
In the preferred embodiment, the typing recognition methods of this invention are utilized within a multi-touch system like that shown in FIG. 1. The sensor scanning hardware 6 detects touches by fingers 2 on the touch surface 4. The proximity image formation 8 and contact tracking and identification 10 modules determine the touch timing and surface coordinates and report these to the typing recognizer 12. The typing recognizer decides which keys the user intended to press and tells the host communications interface 16 to send those keys to the host computer 18. The chord motion recognizer module 14 that interprets lateral sliding of multiple fingers as pointing or gesture input and effectively disables the typing recognizer for such touches. The synchronization detector 13 searches for simultaneous presses or releases of multiple fingers, thereby aiding in detection of chord slides, chord taps, resting hands, and, for the purposes of this invention, modifier chords. Prior art embodiments of all modules in FIG. 1 except the MODCHORD subdivisions 15 are described in related U.S. pat. app. Ser. No. 09/236,513 by Westerman and Elias. That application is incorporated herein by reference in its entirety. It discloses techniques for detection of chord taps and generation of single commands or button clicks therefrom. It also discloses methods to detect and apply conventional modifier key touches. However, unlike the present invention, that application does not teach recognition of modifier chords nor associated techniques for selecting modifier types according to the zone a chord falls within or the template arrangement best matching its finger arrangement, canceling the chord in response to thumb resting, committing a modifier chord upon reception of subsequent modifiable input activity, nor sustain of modifier signals through typing by both hands until no fingers from the modifier chord remain touching. The detection 56, cancellation 60, and application 62 processes embodying the improvements of the present invention are widely distributed across the typing recognizer 12, synchronization detector 13, and chord motion recognizer 14 modules so as to efficiently apply modifier chords to the other input activities that these modules recognize. Thus these modifier recognition processes are collectively represented in FIG. 1 by the MODCHORD subdivision 15 of each module. The exact relation of the modifier chord recognition processes to each larger recognition module will be made apparent from the following detailed description. U.S. pat. app. Ser. No. 09/681,146 by Westerman further describes an improved typing recognizer 12 that compares touch geometry to key sequence candidate geometry, but its improvements do not bear directly on modifier chord recognition.
Those skilled in the art will recognize that the modifier chord recognition method disclosed herein could be utilized with any sensing device that accurately reports the lateral position of multiple fingertips on a surface. Likewise, the modifier chord recognition software need not reside within the sensing device. It could just as easily execute within the host computer system, or the host computer system and sensing device might be combined such that the same microprocessor executes finger tracking, modifier chord recognition, and user application software. Those with ordinary skill in the art will also be aware that some keyboard interfacing protocols use edge-signaling of key activation state while others use level-signaling. For instance, keyboards with the legacy PS/2 interface for IBM-compatible PCs will transmit a press keycode only upon initial activation of a modifier keyswitch and will send a corresponding release keycode immediately after the finger lifts off the switch. Thus the press/release keycodes are only transmitted at edges or transitions in the state of the keyswitch. Keyboards communicating via the more recent USB (Universal Serial Bus) protocol use level-signaling: as long as a keyswitch is depressed, the keyboard regularly and repeatedly sends the corresponding keycode to the host computer. There are no distinct press and release keycodes. This disclosure will use the edge-signaling, press/release terminology throughout to describe transmission of modifiers to the host computer 18, but it will be apparent to those of ordinary skill in the art how any edge-signaling implementation of the host communications interface 16 can be converted to level-signaling and remain well within the scope of this invention.
The key layout illustrated in FIG. 2 exemplifies prior art placement of modifier keys 20, 21, and 22 such that they can be reached by the thumb or pinky fingers. This diagram shows a QWERTY layout split into left 23 and right 24 halves such that Ctrl and Alt modifier keys 22 can be placed between the split halves within easy reach of either thumb. Non-split layouts are more likely to place Ctrl and Alt modifier keys 21 on the bottom row of the layout where they are most likely to be operated by the pinkies. Whether split or not, most layouts keep the Shift modifier keys 20 diagonally below the home row key (‘A’ or ‘;’) that the pinky normally rests upon. Though these Shift keys 20 are not that far from home row, the pinky motion needed to reach and hold them down is particularly awkward.
In contrast to prior art FIG. 2, FIG. 3 shows the chord modifier zones 30-34 and 35-37 of the present invention as different hatches ranging across each row of the key layout. In an actual product, the different modifier zones would be indicated on the touch surface by different background colors rather than different hatches. According to the preferred embodiment of the present invention, simultaneous touchdown of four fingertips within one of these zones will cause subsequent touch activity to be modified just as if a modifier key was being held as long as the modifier chord is not fully lifted or canceled by an improper thumb touch. Notice that a hand resting on the home row ‘ASDF’ or ‘JKL;’ keys can activate any of its modifier zones by pure finger flexion or extension, eliminating the awkward twists needed to reach conventional modifier keys. Notice also that the modifier zones are not horizontally restricted to the ‘ASDF’ or ‘JKL;’ key columns, so the typist need not worry about horizontal hand alignment when performing a modifier chord. The key layout improved with modifier zones may retain the conventional Shift modifier keys 20 near the pinky for the convenience of novices who have not yet learned to perform modifier chords, but experience with the invention has shown that typists rarely use these conventional modifier keys once they have learned the more convenient modifier chords. Since capitalization is needed so frequently, the modifier zones 31 and 36 along the left and right home row of keys are preferably assigned the Left Shift and Right Shift modifiers. The modifier zones 30 and 35 above home row are preferably assigned the Left Ctrl and Right Ctrl modifiers, and the zones 32 and 37 directly below home row are preferably assigned the Left Alt and Right Alt modifiers. Modifier zones 33 and 38 can be assigned any remaining modifiers such as Meta, Diamond, Windows, or Open apple supported by the host computer 18's particular operating system. Those skilled in the alternative key layout arts will realize that modifier zones can be established just as easily over Dvorak, non-split, or non-English key layouts. Also, those skilled in the foreign keyboard arts will realize that the modifier zones could alternatively be assigned to Asian language modifiers like Kana or Kanji and remain well within the scope of this invention.
In the preferred embodiment of the present invention, modifier chord recognition is split into three processes as shown in FIG. 4. Process 56, detailed in FIG. 8, detects new modifier chords by monitoring the incoming touch sequence 50 for touchdowns by the four fingertips of one hand that are simultaneous and also lie within one of the zones or arrangements specified by the configuration data 54. This process only needs to execute each time a new finger touch is detected and appended to the touch sequence, not for image frames from the sensor scanner 6 that contain no new surface contacts. The incoming touch sequence 50 consists of a chain of touch data structures 80 (detailed in FIG. 5) ordered by touchdown times 85. The configuration data structures 100 and 520 that store the modifier zones or arrangements 54 are detailed in FIG. 7 and FIG. 17. If it detects a new modifier chord, process 56 will allocate a corresponding modifier state data structure 90 (detailed in FIG. 6 ) and appended it to the modifier chord state chain 58. The liftoff/cancellation detection process 60 (detailed in FIG. 14 ) checks each new image frame or sensor array scan for absence of all four fingertips or presence of the thumb from any hand with a modifier chord pending in the state chain 58. Upon detection of one of these conditions, the liftoff/cancellation process 60 uses feedback path 68 into the modifier state chain 58 to record either the chord liftoff time 94 or chord cancellation 96. It may also send corresponding modifier release signals to the host computer 18 through the host communication interface 16. The third process 62 searches the modifier state chain 58 for uncancelled modifier chords roughly coincident with modifiable input activity. If this process finds any applicable modifier chords whose press signals have not already been sent, it will cause the host computer interface 16 to send their modifier press signals to the host computer 18. This search process 62 is typically called by the typing recognizer 12, synchronization detector 13, or chord motion recognizer 14 with the timestamp 64 of the modifiable input activity right before such activity itself is transmitted through the host communication interface 16.
FIG. 5 lists basic parameters needed for each touch data structure 80 to support detection of new modifier chords 56. A ring or chain of such data structures ordered by touchdown time 85 represents a touch sequence by one or more fingers. Since palm touches are to be ignored by all recognition processes, they can be left out of the touch sequence 50. Each touch data structure 80 must contain the touch's x and y surface coordinates 82 as reported by the touch sensors 6. These should estimate the center of the touch, which for proximity or pressure sensors is typically computed as the centroid of fingertip flesh contacting the surface. The y offset from default finger location 84 will be computed in FIG. 9 to improve the accuracy of modifier zone selections. To help determine whether a touch sequence represents the four fingertips of a modifier chord, each touch data structure should have a copy of the hand and finger identity 84 estimated for the touch by the contact tracking and identification module 10. To help detect synchronization of multiple touches, the touch data should also include the finger touchdown time 85, also reported by the contact tracking and identification module 10. While the touch liftoff time 86 is a useful indicator of individual modifier key release, the modifier chord liftoff time 94 is not derived from the individual liftoff times 86 of the touches originally forming the chord. This is because in its preferred embodiment the present invention beneficially allows individual fingertips of a modifier chord to lift temporarily and touch again to type while other modifier chord fingertips remain touching, as is shown in FIG. 12. These temporary finger lifts will establish the touch liftoff times 86 for the original touches forming the chord but may not be indicative of final chord liftoff since the temporarily lifted fingers may touch again to sustain the chord while other modifier fingertips lift to type, and so on.
FIG. 6 lists the parameters that keep track of a modifier chord's state 90 once it is initially detected by process 56. The modifier type bits 91 encodes which modifiers, e.g. Shift, Ctrl, or Alt, the chord's fingertip placement has selected. The contents of these bits will be determined by either the modifier zone selection process of FIG. 10 or the fingertip arrangement selection process of FIG. 18. The hand field 92 indicates which hand performed the modifier chord. The chord touchdown time 93 is set by the new chord detection process 56 as the minimum of the touchdown times 85 of the original four fingertip touches firming the chord. As discussed previously, the chord liftoff time 94 should not be computed as the maximum of the original touch liftoff times 86. Instead, the liftoff and cancellation check process 60, FIG. 14, will set the chord liftoff time 94 as the first time after chord detection that none of the four fingertips from the chord are found to be touching the surface. The canceled flag 96 is set by the liftoff and cancellation check process 60 if the chord fingertips slide substantially or a thumb from the chord's hand touches before the chord as committed. When set, this flag causes the applicable modifier search process 62 to ignore the modifier chord. The committed flag 97 is set once a modifier chord is actually applied by the applicable modifier search process 62 to outgoing typing or clicking activity. Once set, this flag prevents the chord from being canceled except when fully lifted.
FIG. 7 shows the data structure 100 and parameters used to configure each modifier zone. Here, the modifier type bitmask 102 is a set of flags indicating which of the Shift, Ctrl, Alt or other modifiers are assigned to the zone. Note that this bitmask implementation allows multiple modifiers, such as Shift Ctrl, to be assigned to the same zone if the user so desires. The hand field 104 can restrict the zone to the left 23 or right 24 half of the key layout, but no other horizontal alignment restriction is necessary. The min 106 and the max 108 Y offset from home row determine the vertical range of the modifier zone. Note that no extra configuration parameters are necessary to arch the zones along each row, should the key rows be so arched, because the zone selection process of FIG. 10 will compare these Y offsets with each fingertip's vertical offset relative to the position of the home row key the finger normally rests upon, also know as the fingertip's default position. Thus the modifier zone will automatically arch to match any arch in the home row key locations.
The new modifier chord detection process is shown in FIG. 8. This process begins whenever a new finger touch is detected by the contact tracking and identification module 10. Step 150 increments the latest touch index n and stores the touch's parameters 80 at the n th location of the touch sequence array T[ ]. Steps 152, 154, and 156 scan backward m touches in the touch sequence trying to find the largest synchronized subsequence that includes the new touch T[n]. Decision diamond 154 judges synchronization by testing whether the m th previous touch T[n−m] contacted the surface within a synchronization interval of about 60 milliseconds of the new touch T[n]. Note that the typing recognizer 12 should not generate signals to the host corresponding to an individual touch over a key until sufficient time has passed without subsequent touches on the same hand that this synchronization detection loop can be certain that the touch is not synchronized with later touches. Thus the typing recognizer 12 must delay key output about 60 ms from finger touchdown or be prepared to erase or undo keys from touches later found to be part of a chord. Once decision diamond 154 finds a previous touch too old to be synchronized with T[n], it passes on the largest synchronized subsequence as T[n−m+1] . . . T[n].
Decision diamond 158 then examines the finger and hand identity 84 of each synchronized touch looking for a combination of identifies from one hand that matches any combination allowed for modifier chords. In the preferred embodiment, only the 4 fingertip combination, index, middle, ring and pinky, excluding the thumb, is used for modifier chords. To prevent duplicate detection of the same modifier chord, decision diamond 158 must require the newest touch T[n] to be one of the modifier chord fingertips. Otherwise, any synchronized touches intervening from the hand opposite a modifier chord combination do not affect the modifier chord, but the modifier chord may eventually apply its modifiers to them as in touch 370 of FIG. 13. If the synchronized subsequence does not contain a modifier chord combination from either hand, the process returns 160 until the next new touch warrants a renewed detection attempt.
Assuming a modifier chord combination from one hand is found within the synchronized subsequence, block 162 forms a touch array MT[ ] indexed by finger identity containing only the modifier chord touches. As further described in FIG. 9, block 162 also computes each touch's vertical offset from its corresponding home row key or default location. Block 164, further described in FIG. 10, checks whether the chord has been performed within any of the established modifier zones. In an alternative embodiment, block 164 may check the arrangement of fingertips within the chord as further described in FIG. 18. Decision diamond 170 ends new modifier detection through step 168 if the chord matches none of the established zones or arrangements. Otherwise, step 172 allocates a new modifier state 90, setting the modifier type 91 according to that of the selected zones or arrangement, setting the modifier hand 92 to be detected chord's hand identity, and setting the chord touchdown time 93 as the minimum of the touchdown times 85 of the synchronized touches forming the chord, and setting the chord liftoff time 94 to 0 pending full liftoff detection. Step 172 also appends this new modifier state to the state chain 58 that may already contain state from other chord or key modifiers being held by the opposite hand. New modifier detection returns at step 176, and responsibility for canceling the modifier chord or applying it to modifiable input activity passes to processes 60 and 62.
FIG. 9 shows the details of vertical offset computation block 162. Step 200 starts the sync index s at the oldest synchronized touch and loops 210 through the synchronized subsequence. Decision diamond 204 separates out the modifier chord touches from any opposite hand touches. For the convenience of the modifier zone or arrangement matching process 164, step 206 stores each modifier chord touch into the touch array MT[ ] in order of their finger identity 84. Step 208 computes each modifier touch's vertical offset from the default or resting location of its corresponding finger. For key layouts such as 23 and 24 with an arch across the home row keys, these default locations should be the locations of the ‘ASDF’ and ‘JKL;’ keys. These vertical offsets help the modifier zone matching process 164 efficiently compensate for the natural arch across the fingertips. Decision diamond 202 breaks the loop and returns through 212 once the whole synchronized subsequence has been processed.
FIG. 10 discloses the preferred embodiment of block 164, a search for modifier zones 100 that the modifier chord fingertips lie within. The array Z[ ] of configured zones is assumed to contain only zones for the given modifier chord's hand, and these zones are assumed to be ordered within the array from farthest below home row to farthest above. In this embodiment, the loop of steps 250-276 will attempt to accumulate modifier types from each zone that any of the modifier chord fingertips lies within. If, however, decision diamond 262 finds that the fingertips are not clearly spaced across zones, i.e. if two or more fingertips are vertically bunched together yet straddling the border between two zones, it will direct the loop of steps 280-288 to find the single zone that the average of the fingertip offsets falls within. Step 250 clears selected_mods, the variable whose bits will accumulate the selected modifier types. Step 252 clears the modifier zone index i. Step 254 initializes the max vertical offset for fingertips found in the i th zone, Z[i].found_ymax, to the zone's minimum vertical boundary Z[i].range_ymin. Step 256 initialized the modifier fingertip index f to 2, representing the index finger. Decision diamond 260 checks whether touch MT[f]'s vertical offset from default is within zone Z[i]'s boundaries. If not, decision diamond 270 checks whether all fingertip touches including the pinky have been examined, and if not step 272 advances the fingertip index. If touch MT[f] is within Z[i]'s vertical range, decision diamond 262 checks whether it is vertically separated by at least zone_safety_sep from any touches already found in the zone below. zone_safety_sep should be set to about 1 cm or ⅜″. If the touch is clearly separated form any in the zone below, step 264 will bump up Z[i].found_ymax as necessary. Step 266 accumulates the modifier types 91 assigned to Z[i] into the selected_mods bits with a bitwise OR operation. Once all fingertips have been checked against the i th zone, decision diamond 274 will check whether all zones have been tested, and if not advance to the next zone through step 276. If the loop 254-276 gets through all zones without decision diamond 262 finding a fingertip separation violation, step 278 will return the accumulated modifier types selected_mods to decision diamond 170 of FIG. 8.
If decision diamond 262 detects an interzone fingertip separation violation, the zone index i and selected_mods are reset at steps 280 and 282. Step 284 computes from MT[ ]the average avg_yoffset of all four fingertips' vertical offsets. Decision diamond 286 checks whether this avg_yoffset is within the range of zone Z[i]. If not, step 290 advances the zone index to the next zone until either decision diamond 292 finds all zones have been exhausted or decision diamond 286 finds a matching zone. Assuming avg_yoffset falls within the vertical range of one of the zones, step 288 assigns selected_mods the modifier type(s) of that zone, and step 278 returns these. Note that step 278 will return zero if the fingertips are not within range of any zone.
Before describing in detail the liftoff/cancelation detection process 60 and the search for applicable modifiers 70, it will be helpful to define their preferred behavior with the diagramed typing examples of FIGS. 11-13. These timing diagrams display the touchdown (falling edge) and liftoff (rising edge) timings for particular fingers 84 from both hands, where LF5 denotes the left pinky, LF1 the left thumb, RF2 the right index finger, and so on. Fingers not shown in an example can be assumed to be lifted throughout. All synchronized chord touches 300 are assumed to fall within a Shift modifier zone. The vertical dotted lines 93 and 94 identify the chord touchdown time and liftoff time, respectively. Keys transmitted to the host by the typing recognizer 12 in response to asynchronous touches are denoted by the key's symbol encircled. The key symbol is only capitalized if the Shift modifier chord applies to it. The slight space drawn between each touchdown and the corresponding key character circle simply indicates that the typing recognizer must wait about 60 ms after each touchdown before sending a keypress to ensure that the touch will not be part of a chord. The left and right ends of SHIFT ellipses 318, 319, 338, 339, 348, 378, and 388 in the MODIFIER 66 row demarcate the modifier press and release signals that should be sent to the host computer. Thus the Shift modifier is being applied through the duration of each ellipse. The bottom two rows of each timing diagram indicate setting of the modifier state canceled flag 96 or committed flag 97 in response to certain touches.
FIG 11A shows typing before, during, and after performance of a left hand Shift chord. The initial ‘a’ key touch 310 clearly precedes modifier chord touchdown 93 and so should not be capitalized. Notice that the Shift modifier is not sent to the host, committing the chord 469, until the ‘J’ touch 311 needs to be sent to the host computer, well after modifier chord touchdown. Committing of modifier chords should be thus delayed until transmission of subsequent modifiable activity (the ‘J’ key touch in this case) to allow for the possibility that the 4 fingertips will begin sliding, suggesting that the typist is actually trying to scroll, or be supplemented with a thumb touch, suggesting the typist is just resting the hand sloppily. This subsequent-touch-activity-dependence of modifier chord committing is unique and novel, as the chord and key taps of the related art commit either on liftoff or sufficient touching time of the tapping fingers themselves, while chord slides for pointing and command gestures commit upon significant lateral motion of the involved fingers. The ‘I’ touch 312 is also capitalized, but the Shift modifier is temporarily released while the BackSpace key touch 302 is sent to the host, and then Shift is pressed again at 319 in time for the ‘U’ key touch 313. Decision channel 463 of FIG. 15 will handle such temporarily release of thumb editing keys such as BackSpace and Space under the assumption that typists usually intend to make quick, unshifted edits but are too lazy to lift and retouch the Shift chord before and after the thumb key activations. The ‘n’ key touch 314 clearly follows modifier chord liftoff, and therefore should not be capitalized.
FIG. 11B demonstrates the desired system behavior when the modifying hand's thumb touches down before typing or clicking activity amongst the other fingers has a chance to commit the modifier chord. As before, the touches 320 and 323 before and after modifier chord performance do not get capitalized. However, the ‘i’ and ‘n’ key touches 321 and 322 concurrent with the modifier chord do not get capitalized either because the thumb touch 302 preceding them causes the modifier chord to be canceled 404. Whether the thumb touch lands properly on and activates the BackSpace key or whether it lifts back off before the ‘i’ and ‘n’ touches is not important. The thumb touch 302 is quite likely the result of sloppy hand resting, and should cancel the modifier chord lest subsequent typing or clicking be shifted unintentionally and have to be undone. Decision diamond 402 of FIG. 14 will implement cancellation due to resting thumb touches.
FIGS. 12A and 12B demonstrate how a modifier chord can be sustained through typing by its fingers on its half of the key layout so that whole words can be capitalized from a single modifier chord touchdown. In FIG. 12A, the ‘K’key touch 330 on the opposite hand quickly commits 469 the modifier chord and the ‘I’ touch 331 follows. To obtain a capitalized ‘D’ and ‘S’, the typist need only lift the middle and ring fingertips of the modifier chord and set them down 332 and 333 on those keys one at a time. The typing recognizer must be configured to generate keys from asynchronous touches without waiting for touch liftoff for this to work well, as at least one of the resumed touches 332 and 333 will need to sustain one of the resumed touches 332 and 333 will need to sustain the modifier chord while the pinky and index of the modifier chord temporarily lift to touch the ‘A’ 335 and ‘T’ 336 keys. Chord liftoff 94 does not register and cause release of the modifier 349 until the first moment when none of the modifier fingertips are touching. The right thumb Space key touch 334 invokes temporary modifier release as in FIG. 11A. FIG. 12B is similar to FIG. 12A except that the modifier chord is performed by the right hand and committed 469 by a temporary liftoff and ‘P’ touchback 340 of one of its own pinky fingertip. Touches 341-343 walk through the other modifier chord fingertips ending with a ‘T’ touch 344 by the opposite hand to spell ‘POINT’. This demonstrates that a modifier chord need not always be committed by the opposite hand, which can be useful in situations where only one hand is available for interaction with the touch surface.
Conventional mechanical keyboards never apply modifier keypresses to keys barely preceding the modifier keypresses or just following the modifier key release. The comparative lack of keyswitch action or stroke for touch surfaces lessens the typist's control over the timing between modifier touch and the touches to be modified. FIGS. 13A and 13B demonstrate cases where it is helpful for the system to tolerate some inaccuracies in the timing between modifier chord touches and modifiable key touches. The same techniques also apply to individual modifier key touches on a surface. In FIG. 13A, the touch 370 actually precedes the modifier chord touchdowns 300 but is roughly synchronous with them. Assuming the modifier chord is detected before the key from touch 370 gets sent to the host, the modifier chord should apply to and capitalize the ‘T’. Such handling of key touches that very slightly precede modifier chord touches is important for typists who get in the habit of simultaneously touching both the modifier chord and the opposite hand's key to be capitalized. The system will implement it through the sync_slack term of decision diamond 456 in FIG. 15. The ‘T’ is capitalized again for touch 371 but not for ‘t’ touch 372, which follows chord liftoff. FIG. 13B shows a brief tap of a modifier chord without any coincident key touches. However, ‘T’ touch 382 does follow shortly after chord liftoff. This suggests the typist was intending to capitalize a single key but, in performing the modifier chord so quickly, synchronized it poorly with the key touch 382 on the opposite hand. It is desirable for the system to compensate by applying the modifier chord in this case even though this means committing 469 and sending the modifier press signal 388 after the chord has actually lifted 94. This is implemented with a non-zero lift_slack in step 464 and decision diamond 466 of FIG. 15.
FIG. 14 details process 60, thumb cancellation and chord liftoff detection. Step 400 denotes that whenever any finger touchdown or liftoff is detected on the surface, this process must be repeated for each modifier chord state 90 in the state chain 58. The combination of decision diamonds 401 and 402 causes the chord to be canceled in step 404 if a thumb touch by the modifier chord's hand is detected before the chord's committed flag gets set. Decision diamond 402 may also decide to cancel a chord if excessive sliding is detected amongst the chord's fingertips. If no such canceling thumb touch is detected, decision diamond 406 checks whether chord liftoff has previously been detected. If not, decision diamond 408 checks the latest sensor image frame 52 for any touches by the modifier chord fingertips. If none are found, step 410 sets the chord liftoff time 94 to the timestamp of the latest sensor image. If liftoff was just detected and decision diamond 412 finds that the modifier types 91 for this modifier state MS are currently pressed for the host, step 414 instructs the host communications interface 16 to send the corresponding modifier release signals. Steps 404 and 410, if executed, thus constitute the feedback path 68 from this process 60 to the modifier state chain 58.
FIG. 15 details the search for applicable modifiers 62 that should always execute just before the typing recognizer 12, synchronization detector 13, or chord motion recognizer 14 transmit any typing, pointing, or clicking signals to the host computer 18. These modules will pass to the search process the modifiable tstamp parameter, set to the current system time for pointing activity but otherwise to the original touchdown time of the key or click chord touches. Step 450 clears the applicable_mods variable, which will accumulate the bitmasks of the types of modifiers found applicable. Step 452 initializes the modifier state reference MS to the most recent modifier state 90 in the chain 58. Decision diamond 456 causes canceled 404 chords to be excluded from the applicable modifier search. At this point the modifier state for a canceled chord can, if memory resources are scarce, be entirely removed from the state chain 58.
Assuming the chord has not been canceled, decision diamond 456 ensures that modifiable_tstamp does not precede the chord touchdown time 93 by more than a sync_slack of about 60 ms. If the chord more or less precedes the modifiable activity, step 460 establishes a 0 ms default lift_slack. If the modifier chord has only been touching briefly and is not yet committed, decision diamond 462 will cause step 464 to establish a lift_slack of about 100-150 ms, allowing for late modifiable touches as in FIG. 13B. If the chord has been touching a few seconds or has already been committed by other key touches, decision diamond 463 checks whether the modifiable activity is a thumb keypress, and if so, should prevent the modifier chord from applying at least if the chord is a Shift modifier. This will also cause temporary Shift modifier release during thumb key transmission as in FIG. 11A and FIG. 12A. Note that this-check is not done for briefly touching chords that have not committed yet so as to allow the typist to perform, for example, a single Shift BackSpace macro command using a modifier chord from the hand opposite the thumb key.
The remaining steps ensure that the modifier chord did not lift substantially before modifiable_tstamp. This is certainly true if the modifier chord is still touching and has not yet been assigned a non-zero liftoff time, as detected by decision diamond 465. If the chord has already lifted, decision diamond 466 will need to compare the chord liftoff time 94 with the modifiable_tstamp less the lift_slack to determine whether the modifiable activity occurred before or within lift_slack milliseconds after chord liftoff. Assuming the modifiable activity is found to be sufficiently coincident with the modifier chord, step 469 will set the modifier state's committed flag 97, and step 468 will accumulate the modifier types selected by the applicable modifier state. Thus step 469 constitutes the main feedback path 70 from the applicable modifier search process 62 to the modifier state chain 58. Decision diamond 471 checks whether any older modifier states that might be applicable are left in the state chain 58, and if so step 470 continues the search on the previous modifier state. Once the entire state chain has been examined, the process returns at step 472 the modifier types from the modifier chords found applicable. The host computer interface 16 will compare these newly applicable modifier types to a bitmask of the modifier types already pressed and send additional modifier press or release signals to the host computer as necessary.
Those skilled in the chord keying art will realize that this chord modifier recognition method could easily be adapted to keyboards with conventional mechanical keyswitches, assuming the keyswitch matrix was wired such that the keyswitch scanner circuitry could reliably distinguish coincident presses of four home row keys forming the chord as well as one other key to be modified. However, such an adaptation would not be terribly advantageous because the effort needed to press four mechanical homerow keyswitches simultaneously is just about as straining as the awkward pinky reach for a conventional Shift key. However, when modifier chords are utilized with a proximity sensing multi-touch surface as in the preferred embodiment, holding the chord requires no effort giving modifier chords significant advantage over the pinky reach for a conventional Shift key.
While modifier zones beneficially allow simultaneous selection of any combination of up to four different modifier types by a single hand, the typist must be careful of hand alignment with respect to the zones to operate them accurately. On surfaces without a home row of keys to encourage hand alignment, or when the user is primarily pointing rather than typing, an alternative, hand-position-independent method to select different types of modifiers is desirable. FIG. 16 shows six sets of chord touches that are distinguishable not by their finger combination or alignment with respect to the key layout, but by the relative arrangement of the four fingertips index 502 through pinky 505 within the chord. The arrangements shown would most likely arise from right hand chord performances; left hand chord performances would produce the mirror image of these arrangements. Each arrangement can be assigned a different modifier type to allow a wide range of modifier selections anywhere on the touch surface. The arrangement of FIG. 16A is most relaxed and is thus preferred for the Shift modifier. The arrangement of FIG. 16B is preferably mapped to Ctrl, FIG. 16E to Alt, FIG. 16C to Ctrl Shift, and the remaining arrangements to any other modifiers useful on a particular host computer. The crosshairs 500 indicate the centroid of each fingertip arrangement and will be used as the coordinate origin when storing the arrangements as templates.
FIG. 17 shows a configuration data structure 520 suitable for storing fingertip arrangement templates and associating them with particular modifier types. Thus configuration data structure 520 is an alternative to the modifier zone embodiment 100 of configuration data 54. It has a modifier type bitmask 102 and modifier hand source 104 equivalent to the modifier zone data structure 100. Each data structure 520 encodes one of the fingertip arrangements from FIG. 16 or the like as a template of four x offsets 526 and four y offsets 528 from the chord center 500. Encoding them as offsets from the centroid 500 of the 4 fingertip locations will help template matching in FIG. 18 be neutral to absolute hand position on the surface.
The pattern matching process of FIG. 18 selects the template arrangement that best matches the measured arrangement of fingertips performing a modifier chord. This arrangement matching process is thus an alternative to FIG. 10's modifier zone selection embodiment of block 164. An array AR[ ] of the template data structure 520 constitutes the data 54. Step 550 clears selected mods, the variable whose bits will hold the selected modifier types. Step 552 initializes min_sumdist2, the variable that will hold the lowest match error of evaluated templates, to a very large number. Step 555 compares the average or centroid, avg_fpos, of the four fingertip locations from the MT[ ] modifier touch array. Step 554 clears the arrangement template index i. Step 556 clears the squared error sum_dist2 for the current template AR[i]. Step 558 initializes the finger identity index to 2 indicating the index finger. Steps 560-564 then loop through to the pinky accumulating the squared error between each fingertip's offset from chord center and the corresponding template point's offset from chord center. Decision diamond 566 checks whether the squared error for template AR[i] make it the best match so far, and if so step 568 updates min_sumdist2 and step 570 stores template AR[i]'s modifier type 522 as the modifier selected so far. Decision diamond 572 determines whether all template arrangements have been tested, and if not step 574 advances the index i to the next template. Once all arrangements have been tested, step 576 will return the modifier type 522 of the template 520 which most closely matches the measured arrangement of modifier touches MT[ ].
Those with ordinary skill in the art will be able to adapt the present invention to use different finger combinations such as three fingertips, or a thumb and three fingertips for modifier chords. However, the four fingertip combination is both easiest to perform and by far most compatible with the typing, hand resting, and pointing activities that can also take place on a multi-touch surface. Using the thumb and three fingertips is a bit more awkward and would necessitate cancellation when the pinky, or in general, a fifth digit touched the surface before commit by other modifiable input activity. Using three fingertips for modifier chords would necessitate cancellation when either the thumb or a fourth fingertip from the modifier chord hand touched the surface before commit by modifiable activity on the opposite hand. Three fingertips modifier chords would also be incompatible with drag operations that are preferably assigned to three-fingertips on a multi-touch surface.
Though embodiments and applications of this invention have been shown and described, it will be apparent to those skilled in the art that numerous further embodiments and modifications than mentioned above are possible without departing from the inventive concepts disclosed herein. The invention, therefore, is not to be restricted except in the true spirit and scope of the appended claims.

Claims (43)

1. A multi-touch surface apparatus that interprets 4-finger chords performed concurrently with other input activity as modifiers of that input activity, thereby avoiding awkward thumb or pinky reaches for conventional modifier keys, reducing hand strain and reducing the tendency of hands to drift off of home row, the apparatus comprising:
multiple-touch sensing means that reports the locations and times of finger contacts with a surface,
modifier configuration means establishes modifier zones as horizontal bands across the surface and associates each modifier zone with one or more modifier types;
synchronization detection means that scans for modifier chord touchdowns consisting of substantially simultaneous touches by four and no more than four digits of a hand;
modifier zone selection means that selects modifier types for the chord according to which modifier zones said simultaneous touches fall within;
chord cancellation means that cancels a modifier chord if the remaining unsynchronized digit from its hand touches before other modifiable input activity commits the chord;
modifier applicability search means that commits the modifier chord and sends modifier press signals corresponding to the chord's selected modifier types to a host computer upon detection of modifiable input activity that occurs substantially between touchdown and liftoff of the modifier chord; and,
chord liftoff detection means that reports when none of the digits originally constituting the modifier chord are touching the surface and sends modifier release signals to the host computer accordingly.
2. The apparatus of claim 1 wherein the apparatus additionally comprises a typing recognition means and surface key layout with a home row of keys upon which the hands normally rest, wherein the four digits that can constitute a modifier chord do not include the thumb, and wherein Shift modifier zones are configured along the home row of keys and other modifier types are configured for zones substantially above and below the home row of keys.
3. The apparatus of claim 2 wherein any Shift signals arising from a committed modifier chord are temporarily released during transmission to the host of keypress signals from a thumb key touch and then re-pressed if the modifier chord has not yet lifted.
4. The apparatus of claim 1 wherein the modifier zone selection means compensates for the natural arch in a row of relaxed fingers by computing for each touch of the modifier chord a vertical offset from a default location of the finger whose identity a contact identification means has assigned to the touch.
5. The apparatus of claim 4 wherein the modifier zone selection means simultaneously selects the modifier types from each zone within which any of the simultaneous touches fall except if some of the simultaneous touches are bunched straddling the border between two zones, in which case the modifier type associated with the single zone that the average of the vertical offsets lies within is selected.
6. The apparatus of claim 1 wherein the locations of the modifier zones on the surface are indicated to the user via printing on the surface background colors or textures unique to each zone.
7. The apparatus of claim 1 wherein the locations of the modifier zones on the surface are indicated to the user via active surface display of background colors or textures unique to each zone.
8. A multi-touch surface apparatus that interprets certain finger chords performed concurrently with other input activity as modifiers of that input activity, thereby avoiding awkward thumb or pinky reaches for conventional modifier keys, reducing hand strain and reducing the need to reposition the hands, the apparatus comprising:
multiple-touch sensing means that reports the locations and times of finger contacts with a surface;
modifier configuration means that establishes finger arrangement templates and associates each template with one or more modifier types;
synchronization detection means that scans for modifier chord touchdowns consisting of substantially simultaneous touches by a predetermined combination of digits of a hand;
finger arrangement matching means that selects for the chord the modifier types associated with the template that most closely matches the arrangement of the modifier chord touches;
chord cancellation means that cancels a modifier chord if the remaining digits from its hand not included in the predetermined combination touch before other modifiable input activity commits the chord;
modifier applicability search means that commits the modifier chord and sends modifier press signals corresponding to the chord's selected modifier types to a host computer upon detection of modifiable input activity that occurs substantially between touchdown and liftoff of the modifier chord; and
modifier chord liftoff detection means that reports when none of the digits originally constituting the modifier chord are touching the surface and sends modifier release signals to the host computer accordingly.
9. The apparatus of claim 8 wherein any Shift signals arising from a committed modifier chord are temporarily released during transmission to the host of keypress signals from thumb key touches and then re-pressed if the modifier chord has not yet lifted.
10. The apparatus of claim 8 wherein four fingertips excluding the thumb is used as the predetermined combination of digits, the neutral four fingertip arrangement of FIG. 16A is associated with the shift modifier type, and the horizontally spread arrangement template of FIG. 16B is associated with the Ctrl modifier type.
11. The apparatus of claim 8 wherein the template matching means computes the matching error as the sum of squared distances between each template point's offset from template center and the offset to the corresponding modifier chord touch from chord center, thus making the matching process independent of where the chord is performed on the surface.
12. A method of controlling a modifier on a multi-touch surface having a key layout thereon, the method comprising:
activating the modifier by simultaneously dropping a plurality of fingertips of a hand into a modifier zone on or near a home row of the key layout; and
maintaining activation of the modifier by leaving at least one of the plurality of fingertips in contact with the modifier zone.
13. The method of claim 12 further comprising:
deactivating the modifier by removing each of the plurality of fingertips from contact with the modifier zone.
14. The method of claim 12 further comprising:
deactivating the modifier by placing a thumb of the hand in contact with the key layout.
15. The method of claim 12 wherein the modifier zone on or near a home row of the key layout may be reached through straight flexion or extension of the fingers from their home row position.
16. The method of claim 12 further comprising:
concurrently activating a second modifier key by simultaneously dropping a plurality of fingertips of the hand into two modifier zones on or near a home row of the key layout, wherein the fingertips are clearly distributed into the two modifier zones.
17. A multi-touch surface having embodied thereon a key layout, the key layout comprising at least one modifier zone on or near a home row of the key layout wherein a modifier will be activated by the simultaneous touchdown of a plurality of fingertips within the at least one modifier zone and maintained by leaving at least one of the plurality of fingertips in contact with the modifier zone.
18. The multi-touch surface of claim 17 comprising a plurality of modifier zones on or near a home row of the key layout, wherein each modifier zone corresponds to a different modifier.
19. The multi-touch surface of claim 18 wherein each of the plurality of modifier zones is indicated by a different color.
20. A method of processing incoming signals from a multi-touch surface, the method comprising:
detecting modifier chords by monitoring the incoming touch sequence for simultaneous touchdowns of a plurality of fingertips of a hand within one or more predetermined modifier zones; and, if a new modifier chord is detected:
allocating a corresponding modifier state data structure; and
appending the modifier state data structure to a modifier chord state chain; and
checking for cancellation of a modifier chord; and, if cancellation of a modifier chord is detected:
recording either a modifier chord liftoff time or modifier chord cancellation.
21. The method of claim 20 further comprising:
searching a modifier state chain for un-cancelled modifier chords roughly coincident with modifiable input activity; and, if any applicable modifier chords whose signals have not already been sent to a host computer are identified;
sending the modifier signals to the host computer.
22. A multi-touch surface apparatus comprising:
a multi-touch surface that reports the locations and times of finger contacts with a surface;
a synchronization detector that scans for modifier chord touchdowns consisting of substantially simultaneous touches by a plurality of digits of a hand in a pre-defined modifier zone on the multi-touch surface and sends modifier signals accordingly; and
a chord liftoff detector that reports when none of the digits originally constituting the modifier chord are touching the surface and sends modifier release signals accordingly.
23. The multi-touch surface apparatus of claim 22 wherein there are a plurality of pre-defined modifier zones on the multi-touch surface, each pre-defined modifier zone corresponding to a different modifier, the multi-touch surface apparatus further comprising:
a modifier zone selector that selects modifier types for the chord according to which modifier zones the simultaneous touches fall within.
24. The multi-touch surface apparatus of claim 23 further comprising:
a chord canceller that cancels a modifier chord if a remaining unsynchronized digit from the hand touches the multi-touch surface before other modifiable input activity commits the chord.
25. The multi-touch surface apparatus of claim 22 further comprising:
a chord canceller that cancels a modifier chord if a remaining unsynchronized digit from the hand touches the multi-touch surface before other modifiable input activity commits the chord.
26. The apparatus of any of claims 22- 25 further comprising a key layout on the multi-touch surface, the key layout having a home row wherein a first modifier zone is configured along the home row.
27. The apparatus of claim 26 wherein at least one additional modifier zone is configured adjacent the home row.
28. The apparatus of claim 27 wherein the at least one additional modifier zone comprises a second modifier zone above the home row and a third modifier zone below the home row.
29. The apparatus of claim 28 wherein the modifiers corresponding to the first, second, and third modifier zones are selected from the group consisting of Shift, Ctrl, Alt, Windows, Open Apple, or Meta.
30. The apparatus of claim 26 wherein the location of the modifier zone is indicated by printing on the multi-touch surface.
31. The apparatus of claim 30 wherein the printing on the multi-touch surface comprises background color.
32. The apparatus of claim 26 wherein the location of the modifier zone is indicated by texture on the multi-touch surface.
33. The apparatus of any of claims 27- 29, wherein the locations of the modifier zones are indicated by printing on the multi-touch surface.
34. The apparatus of claim 33 wherein the printing on the multi-touch surface comprises a unique background color for each zone.
35. The apparatus of claim 27- 29 wherein the locations of the modifier zones are indicated by texture on the multi-touch surface.
36. The apparatus of claim 26 wherein the location of the modifier zone is indicated by active surface display of background color.
37. The apparatus of claim 26 wherein the location of the modifier zone is indicated by active surface display of texture.
38. The apparatus of any of claims 27- 29 wherein the locations of the modifier zones are indicated by active surface display of background colors unique to each zone.
39. The apparatus of claim 27- 29 wherein the locations of the modifier zones are indicated active surface display of textures unique to each zone.
40. A multi-touch surface apparatus comprising:
a multi-touch surface that reports the locations and times of finger contacts with a surface;
a synchronization detector that scans for modifier chord touchdowns consisting of substantially simultaneous touches by a predetermined combination of digits of a hand in a predetermined arrangement;
finger arrangement matcher that selects a modifier corresponding to the modifier chord and sends modifier signals accordingly; and
a chord liftoff detector that reports when none of the digits originally constituting the modifier chord are touching the surface and sends modifier release signals accordingly.
41. The multi-touch surface apparatus of claim 40 further comprising:
a chord canceller that cancels a modifier chord if the remaining digits from its hand not include in the predetermined combination touch before other modifiable input activity commits the chord.
42. The multi-touch surface apparatus of claim 40 or 41 wherein the predetermined arrangement is selected from the group consisting of a neutral fingertip arrangement and a spread fingertip arrangement.
43. The multi-touch surface apparatus of claim 42 wherein the modifiers associated with the predetermined arrangement are selected from the group consisting of: Shift, Ctrl, Alt, Windows, Open Apple, or Meta.
US11/140,529 2001-02-10 2005-05-27 Multi-touch system and method for emulating modifier keys via fingertip chords Expired - Lifetime USRE40153E1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/140,529 USRE40153E1 (en) 2001-02-10 2005-05-27 Multi-touch system and method for emulating modifier keys via fingertip chords

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US09/681,178 US6570557B1 (en) 2001-02-10 2001-02-10 Multi-touch system and method for emulating modifier keys via fingertip chords
US10/065,252 US7030861B1 (en) 2001-02-10 2002-09-28 System and method for packing multi-touch gestures onto a hand
US11/140,529 USRE40153E1 (en) 2001-02-10 2005-05-27 Multi-touch system and method for emulating modifier keys via fingertip chords

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09/681,178 Reissue US6570557B1 (en) 2001-02-10 2001-02-10 Multi-touch system and method for emulating modifier keys via fingertip chords

Publications (1)

Publication Number Publication Date
USRE40153E1 true USRE40153E1 (en) 2008-03-18

Family

ID=47844572

Family Applications (3)

Application Number Title Priority Date Filing Date
US10/065,252 Expired - Lifetime US7030861B1 (en) 2001-02-10 2002-09-28 System and method for packing multi-touch gestures onto a hand
US11/140,529 Expired - Lifetime USRE40153E1 (en) 2001-02-10 2005-05-27 Multi-touch system and method for emulating modifier keys via fingertip chords
US11/351,928 Expired - Fee Related US7705830B2 (en) 2001-02-10 2006-02-10 System and method for packing multitouch gestures onto a hand

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/065,252 Expired - Lifetime US7030861B1 (en) 2001-02-10 2002-09-28 System and method for packing multi-touch gestures onto a hand

Family Applications After (1)

Application Number Title Priority Date Filing Date
US11/351,928 Expired - Fee Related US7705830B2 (en) 2001-02-10 2006-02-10 System and method for packing multitouch gestures onto a hand

Country Status (1)

Country Link
US (3) US7030861B1 (en)

Cited By (86)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060267805A1 (en) * 2005-05-30 2006-11-30 Samsung Electronics Co., Ltd. Method and system for data input
US20070070052A1 (en) * 1998-01-26 2007-03-29 Fingerworks, Inc. Multi-touch contact motion extraction
US20070177804A1 (en) * 2006-01-30 2007-08-02 Apple Computer, Inc. Multi-touch gesture dictionary
US20080024463A1 (en) * 2001-02-22 2008-01-31 Timothy Pryor Reconfigurable tactile control display applications
US20080088587A1 (en) * 2001-02-22 2008-04-17 Timothy Pryor Compact rtd instrument panels and computer interfaces
US20080158175A1 (en) * 2007-01-03 2008-07-03 Apple Inc. Minimizing mismatch during compensation
US20080162967A1 (en) * 2007-01-03 2008-07-03 Apple Computer, Inc. Gated power management over a system bus
US20080158178A1 (en) * 2007-01-03 2008-07-03 Apple Inc. Front-end signal compensation
US20080163130A1 (en) * 2007-01-03 2008-07-03 Apple Inc Gesture learning
US20080158177A1 (en) * 2007-01-03 2008-07-03 Apple Inc. Master/slave mode for sensor processing devices
US20080165134A1 (en) * 2007-01-08 2008-07-10 Apple Computer, Inc. Digital Controller for a True Multi-point Touch Surface Useable in a Computer System
US20080211766A1 (en) * 2007-01-07 2008-09-04 Apple Inc. Multitouch data fusion
US20090178011A1 (en) * 2008-01-04 2009-07-09 Bas Ording Gesture movies
US20090189878A1 (en) * 2004-04-29 2009-07-30 Neonode Inc. Light-based touch screen
US20090195518A1 (en) * 2007-10-01 2009-08-06 Igt Method and apparatus for detecting lift off on a touchscreen
US7593000B1 (en) 2008-05-17 2009-09-22 David H. Chin Touch-based authentication of a mobile device through user generated pattern creation
US20090273575A1 (en) * 1995-06-29 2009-11-05 Pryor Timothy R Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US20090273563A1 (en) * 1999-11-08 2009-11-05 Pryor Timothy R Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US20090300531A1 (en) * 1995-06-29 2009-12-03 Pryor Timothy R Method for providing human input to a computer
US20090315850A1 (en) * 2006-05-02 2009-12-24 Steven Porter Hotelling Multipoint Touch Surface Controller
US20090325691A1 (en) * 2008-06-26 2009-12-31 Loose Timothy C Gaming machine having multi-touch sensing device
US20100017872A1 (en) * 2002-12-10 2010-01-21 Neonode Technologies User interface for mobile computer unit
US7653883B2 (en) 2004-07-30 2010-01-26 Apple Inc. Proximity detector in handheld device
US7656393B2 (en) 2005-03-04 2010-02-02 Apple Inc. Electronic device having display and surrounding touch sensitive bezel for user interface and control
US20100130280A1 (en) * 2006-10-10 2010-05-27 Wms Gaming, Inc. Multi-player, multi-touch table for use in wagering game systems
US20100241984A1 (en) * 2009-03-21 2010-09-23 Nokia Corporation Method and apparatus for displaying the non alphanumeric character based on a user input
US20110038114A1 (en) * 2009-08-17 2011-02-17 Apple Inc. Housing as an i/o device
US20110043485A1 (en) * 2007-07-06 2011-02-24 Neonode Inc. Scanning of a touch screen
US7920131B2 (en) 2006-04-25 2011-04-05 Apple Inc. Keystroke tactility arrangement on a smooth touch surface
US20110190061A1 (en) * 2010-02-03 2011-08-04 Nintendo Co., Ltd. Display device, game system, and game method
US20110225524A1 (en) * 2010-03-10 2011-09-15 Cifra Christopher G Multi-Touch Editing in a Graphical Programming Language
US8062115B2 (en) 2006-04-27 2011-11-22 Wms Gaming Inc. Wagering game with multi-point gesture sensing device
US20120229308A1 (en) * 2011-03-10 2012-09-13 Chun-Liang Tsai Low power wireless keyboard
US8339364B2 (en) 2010-02-03 2012-12-25 Nintendo Co., Ltd. Spatially-correlated multi-display human-machine interface
US8381135B2 (en) 2004-07-30 2013-02-19 Apple Inc. Proximity detector in handheld device
US8416217B1 (en) 2002-11-04 2013-04-09 Neonode Inc. Light-based finger gesture user interface
US8416209B2 (en) 2004-05-06 2013-04-09 Apple Inc. Multipoint touchscreen
US8427447B2 (en) 2007-10-01 2013-04-23 Igt Multi-user input systems and processing techniques for serving multiple users
US8432371B2 (en) 2006-06-09 2013-04-30 Apple Inc. Touch screen liquid crystal display
US20130120293A1 (en) * 2011-11-14 2013-05-16 Samsung Electronics Co., Ltd. Touchscreen-enabled terminal and application control method thereof
US8471814B2 (en) 2010-02-26 2013-06-25 Microsoft Corporation User interface control using a keyboard
US8493330B2 (en) 2007-01-03 2013-07-23 Apple Inc. Individual channel phase delay scheme
US8552989B2 (en) 2006-06-09 2013-10-08 Apple Inc. Integrated display and touch screen
US8576199B1 (en) 2000-02-22 2013-11-05 Apple Inc. Computer control systems
US8605872B2 (en) 2011-06-28 2013-12-10 Lifesize Communications, Inc. Muting a videoconference using touch-based gestures
US8605873B2 (en) 2011-06-28 2013-12-10 Lifesize Communications, Inc. Accessing settings of a videoconference using touch-based gestures
US8654941B2 (en) 2009-01-27 2014-02-18 Lifesize Communications, Inc. Using a touch interface to control a videoconference
US8654083B2 (en) 2006-06-09 2014-02-18 Apple Inc. Touch screen liquid crystal display
US8674966B2 (en) 2001-11-02 2014-03-18 Neonode Inc. ASIC controller for light-based touch screen
US8702514B2 (en) 2010-11-01 2014-04-22 Nintendo Co., Ltd. Controller device and controller system
US8713482B2 (en) 2011-07-28 2014-04-29 National Instruments Corporation Gestures for presentation of different views of a system diagram
US8743300B2 (en) 2010-12-22 2014-06-03 Apple Inc. Integrated touch screens
US8775023B2 (en) 2009-02-15 2014-07-08 Neanode Inc. Light-based touch controls on a steering wheel and dashboard
US8782525B2 (en) 2011-07-28 2014-07-15 National Insturments Corporation Displaying physical signal routing in a diagram of a system
US8814686B2 (en) 2010-02-03 2014-08-26 Nintendo Co., Ltd. Display device, game system, and game method
US8845426B2 (en) 2011-04-07 2014-09-30 Nintendo Co., Ltd. Input system, information processing device, storage medium storing information processing program, and three-dimensional position calculation method
US8913009B2 (en) 2010-02-03 2014-12-16 Nintendo Co., Ltd. Spatially-correlated multi-display human-machine interface
US8959459B2 (en) 2011-06-15 2015-02-17 Wms Gaming Inc. Gesture sensing enhancement system for a wagering game
US8956209B2 (en) 2010-08-30 2015-02-17 Nintendo Co., Ltd. Game system, game apparatus, storage medium having game program stored therein, and game process method
US8971572B1 (en) 2011-08-12 2015-03-03 The Research Foundation For The State University Of New York Hand pointing estimation for human computer interaction
US9047007B2 (en) 2011-07-28 2015-06-02 National Instruments Corporation Semantic zoom within a diagram of a system
US9052777B2 (en) 2001-11-02 2015-06-09 Neonode Inc. Optical elements with alternating reflective lens facets
USRE45559E1 (en) 1997-10-28 2015-06-09 Apple Inc. Portable computers
US9086732B2 (en) 2012-05-03 2015-07-21 Wms Gaming Inc. Gesture fusion
US9092093B2 (en) 2012-11-27 2015-07-28 Neonode Inc. Steering wheel user interface
US9132347B2 (en) 2010-08-30 2015-09-15 Nintendo Co., Ltd. Game system, game apparatus, storage medium having game program stored therein, and game process method
US9204094B2 (en) 2011-06-28 2015-12-01 Lifesize Communications, Inc. Adjusting volume of a videoconference using touch-based gestures
US9199168B2 (en) 2010-08-06 2015-12-01 Nintendo Co., Ltd. Game system, game apparatus, storage medium having game program stored therein, and game process method
US9211105B2 (en) 2009-01-28 2015-12-15 Samsung Medison Co., Ltd. Image indicator provision in an ultrasound system
US9213443B2 (en) 2009-02-15 2015-12-15 Neonode Inc. Optical touch screen systems using reflected light
US9239673B2 (en) 1998-01-26 2016-01-19 Apple Inc. Gesturing with a multipoint sensing device
US9239677B2 (en) 2004-05-06 2016-01-19 Apple Inc. Operation of a computer with touch screen interface
US9448712B2 (en) 2007-01-07 2016-09-20 Apple Inc. Application programming interfaces for scrolling operations
US9513744B2 (en) 1994-08-15 2016-12-06 Apple Inc. Control systems employing novel physical controls and touch screens
US9524097B2 (en) * 2011-12-22 2016-12-20 International Business Machines Corporation Touchscreen gestures for selecting a graphical object
US9576422B2 (en) 2013-04-18 2017-02-21 Bally Gaming, Inc. Systems, methods, and devices for operating wagering game machines with enhanced user interfaces
US9606668B2 (en) 2002-02-07 2017-03-28 Apple Inc. Mode-based graphical user interfaces for touch sensitive input devices
US20170111297A1 (en) * 2015-10-20 2017-04-20 Line Corporation Display control method, terminal, and information processing apparatus
US9710095B2 (en) 2007-01-05 2017-07-18 Apple Inc. Touch screen stack-ups
US9778794B2 (en) 2001-11-02 2017-10-03 Neonode Inc. Light-based touch screen
US9778841B2 (en) 2012-02-10 2017-10-03 Hand Held Products, Inc. Apparatus having random ordered keypad
US9785258B2 (en) 2003-09-02 2017-10-10 Apple Inc. Ambidextrous mouse
US9870093B2 (en) 2010-11-23 2018-01-16 Ge Aviation Systems Llc System and method for improving touch screen display use under vibration and turbulence
US10150033B2 (en) 2010-08-20 2018-12-11 Nintendo Co., Ltd. Position calculation system, position calculation device, storage medium storing position calculation program, and position calculation method
US10530717B2 (en) 2015-10-20 2020-01-07 Line Corporation Display control method, information processing apparatus, and terminal
US11275405B2 (en) 2005-03-04 2022-03-15 Apple Inc. Multi-functional hand-held device

Families Citing this family (606)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7760187B2 (en) * 2004-07-30 2010-07-20 Apple Inc. Visual expander
US9292111B2 (en) 1998-01-26 2016-03-22 Apple Inc. Gesturing with a multipoint sensing device
US7840912B2 (en) * 2006-01-30 2010-11-23 Apple Inc. Multi-touch gesture dictionary
JP4052498B2 (en) 1999-10-29 2008-02-27 株式会社リコー Coordinate input apparatus and method
JP2001184161A (en) 1999-12-27 2001-07-06 Ricoh Co Ltd Method and device for inputting information, writing input device, method for managing written data, method for controlling display, portable electronic writing device, and recording medium
US8645137B2 (en) 2000-03-16 2014-02-04 Apple Inc. Fast, language-independent method for user authentication by voice
DE60140909D1 (en) * 2000-07-05 2010-02-04 Smart Technologies Ulc Method for a camera-based touch system
US6803906B1 (en) 2000-07-05 2004-10-12 Smart Technologies, Inc. Passive touch system and method of detecting user input
US7030861B1 (en) * 2001-02-10 2006-04-18 Wayne Carl Westerman System and method for packing multi-touch gestures onto a hand
SE0103835L (en) * 2001-11-02 2003-05-03 Neonode Ab Touch screen realized by display unit with light transmitting and light receiving units
US9052771B2 (en) 2002-11-04 2015-06-09 Neonode Inc. Touch screen calibration and update methods
US9471170B2 (en) 2002-11-04 2016-10-18 Neonode Inc. Light-based touch screen with shift-aligned emitter and receiver lenses
US8095879B2 (en) 2002-12-10 2012-01-10 Neonode Inc. User interface for mobile handheld computer unit
WO2003098421A1 (en) 2002-05-16 2003-11-27 Sony Corporation Inputting method and inputting apparatus
US20040001144A1 (en) 2002-06-27 2004-01-01 Mccharles Randy Synchronization of camera images in camera-based touch system to enhance position determination of fast moving objects
KR100486711B1 (en) * 2002-08-12 2005-05-03 삼성전기주식회사 Apparatus and method for turning pages personal information terminal
US7358963B2 (en) 2002-09-09 2008-04-15 Apple Inc. Mouse having an optically-based scrolling feature
US8587562B2 (en) 2002-11-04 2013-11-19 Neonode Inc. Light-based touch screen using elliptical and parabolic reflectors
US8896575B2 (en) * 2002-11-04 2014-11-25 Neonode Inc. Pressure-sensitive touch screen
US6954197B2 (en) 2002-11-15 2005-10-11 Smart Technologies Inc. Size/scale and orientation determination of a pointer in a camera-based touch system
US9389730B2 (en) * 2002-12-10 2016-07-12 Neonode Inc. Light-based touch screen using elongated light guides
US8902196B2 (en) 2002-12-10 2014-12-02 Neonode Inc. Methods for determining a touch location on a touch screen
US8403203B2 (en) * 2002-12-10 2013-03-26 Neonoda Inc. Component bonding using a capillary effect
US9195344B2 (en) 2002-12-10 2015-11-24 Neonode Inc. Optical surface using a reflected image for determining three-dimensional position information
US8508508B2 (en) 2003-02-14 2013-08-13 Next Holdings Limited Touch screen signal processing with single-point calibration
US8456447B2 (en) 2003-02-14 2013-06-04 Next Holdings Limited Touch screen signal processing
US7629967B2 (en) 2003-02-14 2009-12-08 Next Holdings Limited Touch screen signal processing
US7532206B2 (en) 2003-03-11 2009-05-12 Smart Technologies Ulc System and method for differentiating between pointers used to contact touch surface
US7256772B2 (en) 2003-04-08 2007-08-14 Smart Technologies, Inc. Auto-aligning touch system and method
GB0311177D0 (en) * 2003-05-15 2003-06-18 Qinetiq Ltd Non contact human-computer interface
US8046701B2 (en) * 2003-08-07 2011-10-25 Fuji Xerox Co., Ltd. Peer to peer gesture based modular presentation system
EP2267589A3 (en) * 2003-08-29 2011-03-16 Nokia Corp. Method and device for recognizing a dual point user input on a touch based user input device
US7411575B2 (en) * 2003-09-16 2008-08-12 Smart Technologies Ulc Gesture recognition method and touch system incorporating the same
US7274356B2 (en) 2003-10-09 2007-09-25 Smart Technologies Inc. Apparatus for determining the location of a pointer within a region of interest
US7355593B2 (en) 2004-01-02 2008-04-08 Smart Technologies, Inc. Pointer tracking across multiple overlapping coordinate input sub-regions defining a generally contiguous input region
US7620915B2 (en) * 2004-02-13 2009-11-17 Ludwig Lester F Electronic document editing employing multiple cursors
US7232986B2 (en) * 2004-02-17 2007-06-19 Smart Technologies Inc. Apparatus for detecting a pointer within a region of interest
FR2866726B1 (en) * 2004-02-23 2006-05-26 Jazzmutant CONTROLLER BY HANDLING VIRTUAL OBJECTS ON A MULTI-CONTACT TOUCH SCREEN
US7460110B2 (en) 2004-04-29 2008-12-02 Smart Technologies Ulc Dual mode touch system
US7492357B2 (en) 2004-05-05 2009-02-17 Smart Technologies Ulc Apparatus and method for detecting a pointer relative to a touch surface
US7538759B2 (en) 2004-05-07 2009-05-26 Next Holdings Limited Touch panel display system with illumination and detection provided from a single edge
US8120596B2 (en) 2004-05-21 2012-02-21 Smart Technologies Ulc Tiled touch system
JP2005339444A (en) * 2004-05-31 2005-12-08 Toshiba Matsushita Display Technology Co Ltd Display device
US7864161B2 (en) * 2004-06-17 2011-01-04 Adrea, LLC Use of a two finger input on touch screens
US8466893B2 (en) * 2004-06-17 2013-06-18 Adrea, LLC Use of a two finger input on touch screens
TWI248576B (en) * 2004-07-05 2006-02-01 Elan Microelectronics Corp Method for controlling rolling of scroll bar on a touch panel
US7719523B2 (en) 2004-08-06 2010-05-18 Touchtable, Inc. Bounding box gesture recognition on a touch detecting interactive display
US7434173B2 (en) * 2004-08-30 2008-10-07 Microsoft Corporation Scrolling web pages using direct interaction
US7728823B2 (en) * 2004-09-24 2010-06-01 Apple Inc. System and method for processing raw data of track pad device
US7719522B2 (en) * 2004-09-24 2010-05-18 Apple Inc. Raw data track pad device and system
KR100687737B1 (en) * 2005-03-19 2007-02-27 한국전자통신연구원 Apparatus and method for a virtual mouse based on two-hands gesture
US8677377B2 (en) 2005-09-08 2014-03-18 Apple Inc. Method and apparatus for building an intelligent automated assistant
US20090275406A1 (en) * 2005-09-09 2009-11-05 Wms Gaming Inc Dynamic user interface in a gaming system
US20070063992A1 (en) * 2005-09-17 2007-03-22 Lundquist Paul B Finger-keyed human-machine interface device
TWI291116B (en) * 2005-12-05 2007-12-11 Elan Microelectronics Corp Method for gesture detection on a touch control bar with button and scroll bar functions
US7958456B2 (en) 2005-12-23 2011-06-07 Apple Inc. Scrolling list with floating adjacent index symbols
US7509588B2 (en) 2005-12-30 2009-03-24 Apple Inc. Portable electronic device with interface reconfiguration mode
US8077147B2 (en) * 2005-12-30 2011-12-13 Apple Inc. Mouse with optical sensing surface
US20070165007A1 (en) * 2006-01-13 2007-07-19 Gerald Morrison Interactive input system
EP1986548B1 (en) 2006-02-15 2013-01-02 Hologic, Inc. Breast biopsy and needle localization using tomosynthesis systems
US20070205994A1 (en) * 2006-03-02 2007-09-06 Taco Van Ieperen Touch system and method for interacting with the same
US8111243B2 (en) * 2006-03-30 2012-02-07 Cypress Semiconductor Corporation Apparatus and method for recognizing a tap gesture on a touch sensing device
US9274807B2 (en) 2006-04-20 2016-03-01 Qualcomm Incorporated Selective hibernation of activities in an electronic device
US8296684B2 (en) 2008-05-23 2012-10-23 Hewlett-Packard Development Company, L.P. Navigating among activities in a computing device
US8683362B2 (en) * 2008-05-23 2014-03-25 Qualcomm Incorporated Card metaphor for activities in a computing device
US20090278806A1 (en) * 2008-05-06 2009-11-12 Matias Gonzalo Duarte Extended touch-sensitive control area for electronic device
US20070257891A1 (en) * 2006-05-03 2007-11-08 Esenther Alan W Method and system for emulating a mouse on a multi-touch sensitive surface
US8717302B1 (en) * 2006-06-30 2014-05-06 Cypress Semiconductor Corporation Apparatus and method for recognizing a gesture on a sensing device
US8972902B2 (en) * 2008-08-22 2015-03-03 Northrop Grumman Systems Corporation Compound gesture recognition
WO2008010024A1 (en) * 2006-07-16 2008-01-24 Cherradi I Free fingers typing technology
US8144271B2 (en) 2006-08-03 2012-03-27 Perceptive Pixel Inc. Multi-touch sensing through frustrated total internal reflection
US8441467B2 (en) * 2006-08-03 2013-05-14 Perceptive Pixel Inc. Multi-touch sensing display through frustrated total internal reflection
US8253695B2 (en) * 2006-09-06 2012-08-28 Apple Inc. Email client for a portable multifunction device
US10313505B2 (en) 2006-09-06 2019-06-04 Apple Inc. Portable multifunction device, method, and graphical user interface for configuring and displaying widgets
US7956849B2 (en) 2006-09-06 2011-06-07 Apple Inc. Video manager for portable multifunction device
US8564544B2 (en) * 2006-09-06 2013-10-22 Apple Inc. Touch screen device, method, and graphical user interface for customizing display of content category icons
US7864163B2 (en) 2006-09-06 2011-01-04 Apple Inc. Portable electronic device, method, and graphical user interface for displaying structured electronic documents
US9304675B2 (en) 2006-09-06 2016-04-05 Apple Inc. Portable electronic device for instant messaging
US8842074B2 (en) 2006-09-06 2014-09-23 Apple Inc. Portable electronic device performing similar operations for different gestures
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
US8564543B2 (en) 2006-09-11 2013-10-22 Apple Inc. Media player with imaged based browsing
KR20110007237A (en) 2006-09-28 2011-01-21 교세라 가부시키가이샤 Portable terminal and control method therefor
US7856605B2 (en) * 2006-10-26 2010-12-21 Apple Inc. Method, system, and graphical user interface for positioning an insertion marker in a touch screen display
US8570278B2 (en) 2006-10-26 2013-10-29 Apple Inc. Portable multifunction device, method, and graphical user interface for adjusting an insertion point marker
KR100886337B1 (en) * 2006-11-23 2009-03-02 삼성전자주식회사 Apparatus for simultaneously saving the areas selected on image and apparatus for making documents by automatically recording image informations
JP2008140211A (en) * 2006-12-04 2008-06-19 Matsushita Electric Ind Co Ltd Control method for input part and input device using the same and electronic equipment
US9442607B2 (en) 2006-12-04 2016-09-13 Smart Technologies Inc. Interactive input system and method
US7956847B2 (en) * 2007-01-05 2011-06-07 Apple Inc. Gestures for controlling, manipulating, and editing of media files using touch sensitive devices
US7877707B2 (en) * 2007-01-06 2011-01-25 Apple Inc. Detecting and interpreting real-world and security gestures on touch and hover sensitive devices
US20080168478A1 (en) 2007-01-07 2008-07-10 Andrew Platzer Application Programming Interfaces for Scrolling
US8519964B2 (en) 2007-01-07 2013-08-27 Apple Inc. Portable multifunction device, method, and graphical user interface supporting user navigations of graphical objects on a touch screen display
US20080165148A1 (en) * 2007-01-07 2008-07-10 Richard Williamson Portable Electronic Device, Method, and Graphical User Interface for Displaying Inline Multimedia Content
US9001047B2 (en) 2007-01-07 2015-04-07 Apple Inc. Modal change based on orientation of a portable multifunction device
US8689132B2 (en) 2007-01-07 2014-04-01 Apple Inc. Portable electronic device, method, and graphical user interface for displaying electronic documents and lists
US20080168402A1 (en) 2007-01-07 2008-07-10 Christopher Blumenberg Application Programming Interfaces for Gesture Operations
US20080165151A1 (en) * 2007-01-07 2008-07-10 Lemay Stephen O System and Method for Viewing and Managing Calendar Entries
WO2008095132A2 (en) * 2007-01-31 2008-08-07 Perceptive Pixel, Inc. Methods of interfacing with multi-point input devices and multi-point input systems employing interfacing techniques
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
WO2008128096A2 (en) 2007-04-11 2008-10-23 Next Holdings, Inc. Touch screen system with hover and click input methods
US8681104B2 (en) 2007-06-13 2014-03-25 Apple Inc. Pinch-throw and translation gestures
JP4812124B2 (en) * 2007-06-19 2011-11-09 キヤノン株式会社 Coordinate input device, coordinate detection method, and computer program
US9933937B2 (en) 2007-06-20 2018-04-03 Apple Inc. Portable multifunction device, method, and graphical user interface for playing online videos
US8302033B2 (en) * 2007-06-22 2012-10-30 Apple Inc. Touch screen device, method, and graphical user interface for providing maps, directions, and location-based information
US8059101B2 (en) * 2007-06-22 2011-11-15 Apple Inc. Swipe gestures for touch screen keyboards
US9772751B2 (en) 2007-06-29 2017-09-26 Apple Inc. Using gestures to slide between user interfaces
EP2017756A1 (en) * 2007-07-20 2009-01-21 BrainLAB AG Method for displaying and/or processing or manipulating image data for medical purposes with gesture recognition
US8094137B2 (en) 2007-07-23 2012-01-10 Smart Technologies Ulc System and method of detecting contact on a display
US20090048707A1 (en) * 2007-08-15 2009-02-19 Deline Jonathan E Fuel dispenser
US8947364B2 (en) * 2007-08-20 2015-02-03 Synaptics Incorporated Proximity sensor device and method with activation confirmation
US8432365B2 (en) * 2007-08-30 2013-04-30 Lg Electronics Inc. Apparatus and method for providing feedback for three-dimensional touchscreen
US8432377B2 (en) 2007-08-30 2013-04-30 Next Holdings Limited Optical touchscreen with improved illumination
US8384693B2 (en) 2007-08-30 2013-02-26 Next Holdings Limited Low profile touch panel systems
US8219936B2 (en) * 2007-08-30 2012-07-10 Lg Electronics Inc. User interface for a mobile device using a user's gesture in the proximity of an electronic device
US8619038B2 (en) 2007-09-04 2013-12-31 Apple Inc. Editing interface
US11126321B2 (en) 2007-09-04 2021-09-21 Apple Inc. Application menu user interface
US9619143B2 (en) * 2008-01-06 2017-04-11 Apple Inc. Device, method, and graphical user interface for viewing application launch icons
US20090058820A1 (en) * 2007-09-04 2009-03-05 Microsoft Corporation Flick-based in situ search from ink, text, or an empty selection region
EP2203865A2 (en) 2007-09-24 2010-07-07 Apple Inc. Embedded authentication systems in an electronic device
US8077143B2 (en) * 2007-09-27 2011-12-13 Microsoft Corporation Motion based display management
KR101398134B1 (en) * 2007-10-04 2014-05-20 엘지전자 주식회사 Apparatus and method for playing moving-picture in mobile terminal
SG152092A1 (en) * 2007-10-26 2009-05-29 Creative Tech Ltd Wireless handheld device able to accept text input and methods for inputting text on a wireless handheld device
US10146320B2 (en) 2007-10-29 2018-12-04 The Boeing Company Aircraft having gesture-based control for an onboard passenger service unit
US20090109036A1 (en) * 2007-10-29 2009-04-30 The Boeing Company System and Method for Alternative Communication
US7996045B1 (en) 2007-11-09 2011-08-09 Google Inc. Providing interactive alert information
US7976372B2 (en) 2007-11-09 2011-07-12 Igt Gaming system having multiple player simultaneous display/input device
US8439756B2 (en) 2007-11-09 2013-05-14 Igt Gaming system having a display/input device configured to interactively operate with external device
US8545321B2 (en) 2007-11-09 2013-10-01 Igt Gaming system having user interface with uploading and downloading capability
US9171454B2 (en) 2007-11-14 2015-10-27 Microsoft Technology Licensing, Llc Magic wand
US20090125848A1 (en) * 2007-11-14 2009-05-14 Susann Marie Keohane Touch surface-sensitive edit system
KR20090052673A (en) * 2007-11-21 2009-05-26 엘지전자 주식회사 Terminal, method for controlling the terminal, and recorable medium for the method
US8610671B2 (en) 2007-12-27 2013-12-17 Apple Inc. Insertion marker placement on touch sensitive display
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US8327272B2 (en) 2008-01-06 2012-12-04 Apple Inc. Portable multifunction device, method, and graphical user interface for viewing and managing electronic calendars
US8405636B2 (en) 2008-01-07 2013-03-26 Next Holdings Limited Optical position sensing system and optical position sensor assembly
US20090183098A1 (en) * 2008-01-14 2009-07-16 Dell Products, Lp Configurable Keyboard
US8677285B2 (en) * 2008-02-01 2014-03-18 Wimm Labs, Inc. User interface of a small touch sensitive display for an electronic data and communication device
US8446373B2 (en) * 2008-02-08 2013-05-21 Synaptics Incorporated Method and apparatus for extended adjustment based on relative positioning of multiple objects contemporaneously in a sensing region
US8645827B2 (en) 2008-03-04 2014-02-04 Apple Inc. Touch event model
US8174502B2 (en) 2008-03-04 2012-05-08 Apple Inc. Touch event processing for web pages
US9483755B2 (en) * 2008-03-04 2016-11-01 Apple Inc. Portable multifunction device, method, and graphical user interface for an email client
US8205157B2 (en) 2008-03-04 2012-06-19 Apple Inc. Methods and graphical user interfaces for conducting searches on a portable multifunction device
US8650507B2 (en) * 2008-03-04 2014-02-11 Apple Inc. Selecting of text using gestures
US8416196B2 (en) 2008-03-04 2013-04-09 Apple Inc. Touch event model programming interface
US8201109B2 (en) * 2008-03-04 2012-06-12 Apple Inc. Methods and graphical user interfaces for editing on a portable multifunction device
US8717305B2 (en) 2008-03-04 2014-05-06 Apple Inc. Touch event model for web pages
JP4650699B2 (en) 2008-03-06 2011-03-16 Necインフロンティア株式会社 Input device, input method, and program
US8996376B2 (en) 2008-04-05 2015-03-31 Apple Inc. Intelligent text-to-speech conversion
US8159469B2 (en) 2008-05-06 2012-04-17 Hewlett-Packard Development Company, L.P. User interface for initiating activities in an electronic device
US8902193B2 (en) 2008-05-09 2014-12-02 Smart Technologies Ulc Interactive input system and bezel therefor
US20090282332A1 (en) * 2008-05-12 2009-11-12 Nokia Corporation Apparatus, method and computer program product for selecting multiple items using multi-touch
US8952894B2 (en) 2008-05-12 2015-02-10 Microsoft Technology Licensing, Llc Computer vision-based multi-touch sensing using infrared lasers
US10496753B2 (en) 2010-01-18 2019-12-03 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US20090284478A1 (en) * 2008-05-15 2009-11-19 Microsoft Corporation Multi-Contact and Single-Contact Input
US9268483B2 (en) * 2008-05-16 2016-02-23 Microsoft Technology Licensing, Llc Multi-touch input platform
US8830181B1 (en) * 2008-06-01 2014-09-09 Cypress Semiconductor Corporation Gesture recognition system for a touch-sensing surface
TWI421731B (en) * 2008-06-02 2014-01-01 Asustek Comp Inc Method for executing mouse function of electronic device and electronic device thereof
JP5164675B2 (en) * 2008-06-04 2013-03-21 キヤノン株式会社 User interface control method, information processing apparatus, and program
US9549585B2 (en) 2008-06-13 2017-01-24 Nike, Inc. Footwear having sensor system
WO2009152456A2 (en) 2008-06-13 2009-12-17 Nike, Inc. Footwear having sensor system
US10070680B2 (en) 2008-06-13 2018-09-11 Nike, Inc. Footwear having sensor system
US9037530B2 (en) 2008-06-26 2015-05-19 Microsoft Technology Licensing, Llc Wearable electromyography-based human-computer interface
US8447704B2 (en) 2008-06-26 2013-05-21 Microsoft Corporation Recognizing gestures from forearm EMG signals
WO2010006087A1 (en) * 2008-07-08 2010-01-14 David Seaberg Process for providing and editing instructions, data, data structures, and algorithms in a computer system
US8169414B2 (en) * 2008-07-12 2012-05-01 Lim Seung E Control of electronic games via finger angle using a high dimensional touchpad (HDTP) touch user interface
TW201005599A (en) * 2008-07-18 2010-02-01 Asustek Comp Inc Touch-type mobile computing device and control method of the same
US8390577B2 (en) * 2008-07-25 2013-03-05 Intuilab Continuous recognition of multi-touch gestures
US20100030549A1 (en) 2008-07-31 2010-02-04 Lee Michael M Mobile device having human language translation capability with positional feedback
JP4957750B2 (en) * 2008-07-31 2012-06-20 ソニー株式会社 Information processing apparatus and method, and program
US8847739B2 (en) * 2008-08-04 2014-09-30 Microsoft Corporation Fusing RFID and vision for surface object tracking
US20100031202A1 (en) * 2008-08-04 2010-02-04 Microsoft Corporation User-defined gesture set for surface computing
US20100045703A1 (en) * 2008-08-22 2010-02-25 Google Inc. User Interface Gestures For Moving a Virtual Camera On A Mobile Device
KR101611601B1 (en) * 2008-09-12 2016-04-12 코닌클리케 필립스 엔.브이. Navigating in graphical user interface on handheld devices
FR2936326B1 (en) * 2008-09-22 2011-04-29 Stantum DEVICE FOR THE CONTROL OF ELECTRONIC APPARATUS BY HANDLING GRAPHIC OBJECTS ON A MULTICONTACT TOUCH SCREEN
US20100079385A1 (en) * 2008-09-29 2010-04-01 Smart Technologies Ulc Method for calibrating an interactive input system and interactive input system executing the calibration method
US8347230B2 (en) * 2008-09-30 2013-01-01 Apple Inc. Visual presentation of multiple internet pages
US8529345B2 (en) 2008-10-02 2013-09-10 Igt Gaming system including a gaming table with mobile user input devices
US20100095234A1 (en) * 2008-10-07 2010-04-15 Research In Motion Limited Multi-touch motion simulation using a non-touch screen computer input device
KR101503835B1 (en) * 2008-10-13 2015-03-18 삼성전자주식회사 Apparatus and method for object management using multi-touch
US20100097322A1 (en) * 2008-10-16 2010-04-22 Motorola, Inc. Apparatus and method for switching touch screen operation
US8174504B2 (en) * 2008-10-21 2012-05-08 Synaptics Incorporated Input device and method for adjusting a parameter of an electronic system
KR101500503B1 (en) * 2008-10-29 2015-03-09 엘지전자 주식회사 Mobile terminal and operating method thereof
US8433138B2 (en) * 2008-10-29 2013-04-30 Nokia Corporation Interaction using touch and non-touch gestures
US8339378B2 (en) * 2008-11-05 2012-12-25 Smart Technologies Ulc Interactive input system with multi-angle reflector
US8502785B2 (en) * 2008-11-12 2013-08-06 Apple Inc. Generating gestures tailored to a hand resting on a surface
US9482755B2 (en) 2008-11-17 2016-11-01 Faro Technologies, Inc. Measurement system having air temperature compensation between a target and a laser tracker
US8610673B2 (en) 2008-12-03 2013-12-17 Microsoft Corporation Manipulation of list on a multi-touch display
US8749497B2 (en) * 2008-12-12 2014-06-10 Apple Inc. Multi-touch shape drawing
DE102008061988A1 (en) 2008-12-12 2010-06-17 Volkswagen Ag Method and device for displaying information
US8707170B2 (en) 2008-12-15 2014-04-22 Hewlett-Packard Development Company, L.P. Gesture based edit mode
US20100149114A1 (en) * 2008-12-16 2010-06-17 Motorola, Inc. Simulating a multi-touch screen on a single-touch screen
US20100153996A1 (en) * 2008-12-17 2010-06-17 Migos Charles J Gesture based electronic program management system
US20100162181A1 (en) * 2008-12-22 2010-06-24 Palm, Inc. Interpreting Gesture Input Including Introduction Or Removal Of A Point Of Contact While A Gesture Is In Progress
EP2199949A1 (en) * 2008-12-22 2010-06-23 BRITISH TELECOMMUNICATIONS public limited company Viewpoint determination
US8451236B2 (en) * 2008-12-22 2013-05-28 Hewlett-Packard Development Company L.P. Touch-sensitive display screen with absolute and relative input modes
US8547244B2 (en) * 2008-12-22 2013-10-01 Palm, Inc. Enhanced visual feedback for touch-sensitive input device
EP2370878B8 (en) 2008-12-29 2019-06-19 Hewlett-Packard Development Company, L.P. Gesture detection zones
US8957865B2 (en) * 2009-01-05 2015-02-17 Apple Inc. Device, method, and graphical user interface for manipulating a user interface object
US8365091B2 (en) * 2009-01-06 2013-01-29 Microsoft Corporation Non-uniform scrolling
DE102009008041A1 (en) 2009-02-09 2010-08-12 Volkswagen Ag Method for operating a motor vehicle with a touchscreen
US9063614B2 (en) 2009-02-15 2015-06-23 Neonode Inc. Optical touch screens
US9424578B2 (en) * 2009-02-24 2016-08-23 Ebay Inc. System and method to provide gesture functions at a device
US8589374B2 (en) 2009-03-16 2013-11-19 Apple Inc. Multifunction device with integrated search and application selection
US9684521B2 (en) 2010-01-26 2017-06-20 Apple Inc. Systems having discrete and continuous gesture recognizers
US10705701B2 (en) 2009-03-16 2020-07-07 Apple Inc. Device, method, and graphical user interface for moving a current position in content at a variable scrubbing rate
US8566044B2 (en) * 2009-03-16 2013-10-22 Apple Inc. Event recognition
US8255830B2 (en) * 2009-03-16 2012-08-28 Apple Inc. Methods and graphical user interfaces for editing on a multifunction device with a touch screen display
US9311112B2 (en) 2009-03-16 2016-04-12 Apple Inc. Event recognition
US8566045B2 (en) 2009-03-16 2013-10-22 Apple Inc. Event recognition
US8285499B2 (en) 2009-03-16 2012-10-09 Apple Inc. Event recognition
JP5091180B2 (en) * 2009-03-27 2012-12-05 ソニーモバイルコミュニケーションズ, エービー Mobile terminal device
US8493384B1 (en) 2009-04-01 2013-07-23 Perceptive Pixel Inc. 3D manipulation using applied pressure
US20100259482A1 (en) * 2009-04-10 2010-10-14 Microsoft Corporation Keyboard gesturing
TWI497357B (en) * 2009-04-23 2015-08-21 Waltop Int Corp Multi-touch pad control method
US8633904B2 (en) 2009-04-24 2014-01-21 Cypress Semiconductor Corporation Touch identification for multi-touch technology
TW201039184A (en) * 2009-04-24 2010-11-01 Asustek Comp Inc Method of determining mouse command according to triggered points
CN102405029A (en) * 2009-05-08 2012-04-04 吉列公司 Ersonal care systems, products, and methods
US8375295B2 (en) 2009-05-21 2013-02-12 Sony Computer Entertainment Inc. Customization of GUI layout based on history of use
US8581856B2 (en) * 2009-05-27 2013-11-12 Microsoft Corporation Touch sensitive display apparatus using sensor input
US9141284B2 (en) * 2009-05-28 2015-09-22 Microsoft Technology Licensing, Llc Virtual input devices created by touch input
US9207806B2 (en) * 2009-05-28 2015-12-08 Microsoft Technology Licensing, Llc Creating a virtual mouse input device
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US20100309140A1 (en) * 2009-06-05 2010-12-09 Microsoft Corporation Controlling touch input modes
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US10255566B2 (en) 2011-06-03 2019-04-09 Apple Inc. Generating and processing task items that represent tasks to perform
US8681106B2 (en) * 2009-06-07 2014-03-25 Apple Inc. Devices, methods, and graphical user interfaces for accessibility using a touch-sensitive surface
US20100315335A1 (en) * 2009-06-16 2010-12-16 Microsoft Corporation Pointing Device with Independently Movable Portions
US9703398B2 (en) * 2009-06-16 2017-07-11 Microsoft Technology Licensing, Llc Pointing device using proximity sensing
EP2443537A1 (en) * 2009-06-19 2012-04-25 Alcatel Lucent Gesture on touch sensitive input devices for closing a window or an application
US8462134B2 (en) * 2009-06-29 2013-06-11 Autodesk, Inc. Multi-finger mouse emulation
JP4843696B2 (en) * 2009-06-30 2011-12-21 株式会社東芝 Information processing apparatus and touch operation support program
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
US8692768B2 (en) 2009-07-10 2014-04-08 Smart Technologies Ulc Interactive input system
US20110029904A1 (en) * 2009-07-30 2011-02-03 Adam Miles Smith Behavior and Appearance of Touch-Optimized User Interface Elements for Controlling Computer Function
US9535599B2 (en) 2009-08-18 2017-01-03 Adobe Systems Incorporated Methods and apparatus for image editing using multitouch gestures
US20120151415A1 (en) * 2009-08-24 2012-06-14 Park Yong-Gook Method for providing a user interface using motion and device adopting the method
US20110043462A1 (en) * 2009-08-24 2011-02-24 Ayman Shabra Touchscreen apparatus, integrated circuit device, electronic device and method therefor
CN102023791A (en) * 2009-09-18 2011-04-20 比亚迪股份有限公司 Scrolling control method for touch control device
EP2480957B1 (en) 2009-09-22 2017-08-09 Apple Inc. Device, method, and graphical user interface for manipulating user interface objects
US8624933B2 (en) 2009-09-25 2014-01-07 Apple Inc. Device, method, and graphical user interface for scrolling a multi-section document
US8832585B2 (en) * 2009-09-25 2014-09-09 Apple Inc. Device, method, and graphical user interface for manipulating workspace views
US8766928B2 (en) * 2009-09-25 2014-07-01 Apple Inc. Device, method, and graphical user interface for manipulating user interface objects
US8780069B2 (en) 2009-09-25 2014-07-15 Apple Inc. Device, method, and graphical user interface for manipulating user interface objects
US8799826B2 (en) * 2009-09-25 2014-08-05 Apple Inc. Device, method, and graphical user interface for moving a calendar entry in a calendar application
US8601402B1 (en) * 2009-09-29 2013-12-03 Rockwell Collins, Inc. System for and method of interfacing with a three dimensional display
US11704473B2 (en) * 2009-09-30 2023-07-18 Georgia Tech Research Corporation Systems and methods to facilitate active reading
US9513798B2 (en) * 2009-10-01 2016-12-06 Microsoft Technology Licensing, Llc Indirect multi-touch interaction
CN106420066B (en) 2009-10-08 2020-08-25 霍罗吉克公司 Method for guiding needle biopsy assembly to target position and X-ray imaging system
US8390584B1 (en) * 2009-10-13 2013-03-05 Intuit Inc. Digit aware touchscreen
US8261212B2 (en) * 2009-10-20 2012-09-04 Microsoft Corporation Displaying GUI elements on natural user interfaces
KR20110044496A (en) * 2009-10-23 2011-04-29 삼성전자주식회사 Input signal processing device for portable device and method including the same
US20110095977A1 (en) * 2009-10-23 2011-04-28 Smart Technologies Ulc Interactive input system incorporating multi-angle reflecting structure
US8957918B2 (en) * 2009-11-03 2015-02-17 Qualcomm Incorporated Methods for implementing multi-touch gestures on a single-touch touch surface
US8843857B2 (en) * 2009-11-19 2014-09-23 Microsoft Corporation Distance scalable no touch computing
US8432367B2 (en) * 2009-11-19 2013-04-30 Google Inc. Translating user interaction with a touch screen into input commands
US20110126094A1 (en) * 2009-11-24 2011-05-26 Horodezky Samuel J Method of modifying commands on a touch screen user interface
US20100085323A1 (en) * 2009-12-04 2010-04-08 Adam Bogue Segmenting a Multi-Touch Input Region by User
EP2333651B1 (en) * 2009-12-11 2016-07-20 Dassault Systèmes Method and system for duplicating an object using a touch-sensitive display
US9465532B2 (en) * 2009-12-18 2016-10-11 Synaptics Incorporated Method and apparatus for operating in pointing and enhanced gesturing modes
US20110148786A1 (en) * 2009-12-18 2011-06-23 Synaptics Incorporated Method and apparatus for changing operating modes
US8514188B2 (en) * 2009-12-30 2013-08-20 Microsoft Corporation Hand posture mode constraints on touch input
US8438504B2 (en) * 2010-01-06 2013-05-07 Apple Inc. Device, method, and graphical user interface for navigating through multiple viewing areas
US8736561B2 (en) 2010-01-06 2014-05-27 Apple Inc. Device, method, and graphical user interface with content display modes and display rotation heuristics
US20110167350A1 (en) * 2010-01-06 2011-07-07 Apple Inc. Assist Features For Content Display Device
US8786559B2 (en) * 2010-01-06 2014-07-22 Apple Inc. Device, method, and graphical user interface for manipulating tables using multi-contact gestures
US8786639B2 (en) * 2010-01-06 2014-07-22 Apple Inc. Device, method, and graphical user interface for manipulating a collection of objects
US8502789B2 (en) * 2010-01-11 2013-08-06 Smart Technologies Ulc Method for handling user input in an interactive input system, and interactive input system executing the method
EP2524280A1 (en) 2010-01-14 2012-11-21 BrainLAB AG Controlling a surgical navigation system
US10553209B2 (en) 2010-01-18 2020-02-04 Apple Inc. Systems and methods for hands-free notification summaries
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US10679605B2 (en) 2010-01-18 2020-06-09 Apple Inc. Hands-free list-reading by intelligent automated assistant
US10705794B2 (en) 2010-01-18 2020-07-07 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US20110179381A1 (en) * 2010-01-21 2011-07-21 Research In Motion Limited Portable electronic device and method of controlling same
WO2011089450A2 (en) 2010-01-25 2011-07-28 Andrew Peter Nelson Jerram Apparatuses, methods and systems for a digital conversation management platform
US8612884B2 (en) 2010-01-26 2013-12-17 Apple Inc. Device, method, and graphical user interface for resizing objects
US8539386B2 (en) * 2010-01-26 2013-09-17 Apple Inc. Device, method, and graphical user interface for selecting and moving objects
US8781520B2 (en) * 2010-01-26 2014-07-15 Hand Held Products, Inc. Mobile device having hybrid keypad
US8539385B2 (en) * 2010-01-26 2013-09-17 Apple Inc. Device, method, and graphical user interface for precise positioning of objects
US8570286B2 (en) * 2010-02-12 2013-10-29 Honeywell International Inc. Gestures on a touch-sensitive display
US20110199516A1 (en) * 2010-02-12 2011-08-18 Honeywell International Inc. Method of showing video on a touch-sensitive display
US8638371B2 (en) * 2010-02-12 2014-01-28 Honeywell International Inc. Method of manipulating assets shown on a touch-sensitive display
US20110199386A1 (en) * 2010-02-12 2011-08-18 Honeywell International Inc. Overlay feature to provide user assistance in a multi-touch interactive display environment
US20110199517A1 (en) * 2010-02-12 2011-08-18 Honeywell International Inc. Method of showing video on a touch-sensitive display
US8682667B2 (en) 2010-02-25 2014-03-25 Apple Inc. User profiling for selecting user specific voice input processing information
US20110221701A1 (en) * 2010-03-10 2011-09-15 Focaltech Systems Ltd. Multi-touch detection method for capacitive touch screens
US9292161B2 (en) * 2010-03-24 2016-03-22 Microsoft Technology Licensing, Llc Pointer tool with touch-enabled precise placement
US20110239114A1 (en) * 2010-03-24 2011-09-29 David Robbins Falkenburg Apparatus and Method for Unified Experience Across Different Devices
US8704783B2 (en) 2010-03-24 2014-04-22 Microsoft Corporation Easy word selection and selection ahead of finger
JP5179537B2 (en) * 2010-04-09 2013-04-10 株式会社ソニー・コンピュータエンタテインメント Information processing device
US8724119B2 (en) 2010-04-21 2014-05-13 Faro Technologies, Inc. Method for using a handheld appliance to select, lock onto, and track a retroreflector with a laser tracker
US8422034B2 (en) 2010-04-21 2013-04-16 Faro Technologies, Inc. Method and apparatus for using gestures to control a laser tracker
US9377885B2 (en) 2010-04-21 2016-06-28 Faro Technologies, Inc. Method and apparatus for locking onto a retroreflector with a laser tracker
US9772394B2 (en) 2010-04-21 2017-09-26 Faro Technologies, Inc. Method and apparatus for following an operator and locking onto a retroreflector with a laser tracker
US8537371B2 (en) 2010-04-21 2013-09-17 Faro Technologies, Inc. Method and apparatus for using gestures to control a laser tracker
US8619265B2 (en) 2011-03-14 2013-12-31 Faro Technologies, Inc. Automatic measurement of dimensional data with a laser tracker
US9400170B2 (en) 2010-04-21 2016-07-26 Faro Technologies, Inc. Automatic measurement of dimensional data within an acceptance region by a laser tracker
WO2011130919A1 (en) 2010-04-23 2011-10-27 Motorola Mobility, Inc. Electronic device and method using touch-detecting surface
TWI464622B (en) * 2010-05-10 2014-12-11 Egalax Empia Technology Inc Method and device for gesture determination
JP5533254B2 (en) * 2010-05-24 2014-06-25 アイシン・エィ・ダブリュ株式会社 Information display device, information display method, and program
US8286102B1 (en) 2010-05-27 2012-10-09 Adobe Systems Incorporated System and method for image processing using multi-touch gestures
AP2012006600A0 (en) * 2010-06-01 2012-12-31 Nokia Corp A method, a device and a system for receiving userinput
US8702512B2 (en) * 2010-06-02 2014-04-22 Jean Etienne Mineur Multi player material figure/electronic games board interactive assembly with automatic figure authentification
US9542091B2 (en) 2010-06-04 2017-01-10 Apple Inc. Device, method, and graphical user interface for navigating through a user interface using a dynamic object selection indicator
EP2393000B1 (en) * 2010-06-04 2019-08-07 Lg Electronics Inc. Mobile terminal capable of providing multiplayer game and method of controlling operation of the mobile terminal
US10216408B2 (en) 2010-06-14 2019-02-26 Apple Inc. Devices and methods for identifying user interface objects based on view hierarchy
US8542204B2 (en) 2010-06-19 2013-09-24 International Business Machines Corporation Method, system, and program product for no-look digit entry in a multi-touch device
US20110317871A1 (en) * 2010-06-29 2011-12-29 Microsoft Corporation Skeletal joint recognition and tracking system
KR101667586B1 (en) * 2010-07-12 2016-10-19 엘지전자 주식회사 Mobile terminal and method for controlling the same
US8773370B2 (en) 2010-07-13 2014-07-08 Apple Inc. Table editing systems with gesture-based insertion and deletion of columns and rows
US9098182B2 (en) 2010-07-30 2015-08-04 Apple Inc. Device, method, and graphical user interface for copying user interface objects between content regions
US20120026100A1 (en) * 2010-07-30 2012-02-02 Migos Charles J Device, Method, and Graphical User Interface for Aligning and Distributing Objects
US8972879B2 (en) 2010-07-30 2015-03-03 Apple Inc. Device, method, and graphical user interface for reordering the front-to-back positions of objects
US9081494B2 (en) 2010-07-30 2015-07-14 Apple Inc. Device, method, and graphical user interface for copying formatting attributes
US20120042193A1 (en) * 2010-08-13 2012-02-16 Apple Inc. Clock frequency selection scheme
US9164542B2 (en) * 2010-08-31 2015-10-20 Symbol Technologies, Llc Automated controls for sensor enabled user interface
JP2012058857A (en) * 2010-09-06 2012-03-22 Sony Corp Information processor, operation method and information processing program
EP2437151B1 (en) * 2010-10-01 2020-07-08 Samsung Electronics Co., Ltd. Apparatus and method for turning e-book pages in portable terminal
US9678572B2 (en) 2010-10-01 2017-06-13 Samsung Electronics Co., Ltd. Apparatus and method for turning e-book pages in portable terminal
KR101743632B1 (en) 2010-10-01 2017-06-07 삼성전자주식회사 Apparatus and method for turning e-book pages in portable terminal
KR101727040B1 (en) * 2010-10-14 2017-04-14 엘지전자 주식회사 An electronic device, a method for providing menu using the same
US20120102401A1 (en) * 2010-10-25 2012-04-26 Nokia Corporation Method and apparatus for providing text selection
JP5815932B2 (en) * 2010-10-27 2015-11-17 京セラ株式会社 Electronics
KR101199970B1 (en) * 2010-10-29 2012-11-12 전남대학교산학협력단 Acquisition method of multi-touch feature and multi-touch gesture recognition using the multi-touch feature
CA2817573C (en) 2010-11-10 2018-07-10 Nike International Ltd. Systems and methods for time-based athletic activity measurement and display
US9870141B2 (en) 2010-11-19 2018-01-16 Microsoft Technology Licensing, Llc Gesture recognition
WO2012071429A1 (en) * 2010-11-26 2012-05-31 Hologic, Inc. User interface for medical image review workstation
DE102010054859A1 (en) * 2010-12-17 2012-06-21 Rohde & Schwarz Gmbh & Co. Kg System with gesture recognition unit
WO2012087939A1 (en) * 2010-12-20 2012-06-28 Apple Inc. Event recognition
FR2969781B1 (en) * 2010-12-22 2013-07-05 Peugeot Citroen Automobiles Sa MAN MACHINE INTERFACE COMPRISING A TOUCH-SENSITIVE TOUCH-SENSITIVE CONTROL SURFACE FOR SCROLLING THE ELEMENTS OF A LIST
KR20120071670A (en) * 2010-12-23 2012-07-03 엘지전자 주식회사 Mobile terminal and method for controlling the same
US9245259B2 (en) 2011-01-14 2016-01-26 Apple Inc. Presenting E-mail on a touch device
US8761658B2 (en) 2011-01-31 2014-06-24 FastTrack Technologies Inc. System and method for a computerized learning system
KR101873787B1 (en) * 2011-02-10 2018-07-03 삼성전자주식회사 Method for processing multi-touch input in touch screen terminal and device thereof
CA2827684C (en) 2011-02-17 2016-09-27 Nike International Ltd. Footwear having sensor system
KR101741238B1 (en) 2011-02-17 2017-05-29 나이키 이노베이트 씨.브이. Footwear having sensor system
US9381420B2 (en) * 2011-02-17 2016-07-05 Nike, Inc. Workout user experience
JP6061869B2 (en) 2011-02-17 2017-01-18 ナイキ イノベイト シーブイ Location mapping
US8782566B2 (en) 2011-02-22 2014-07-15 Cisco Technology, Inc. Using gestures to schedule and manage meetings
JP5797282B2 (en) 2011-03-03 2015-10-21 ファロ テクノロジーズ インコーポレーテッド Target apparatus and method
CN110353709A (en) 2011-03-08 2019-10-22 霍洛吉克公司 The system and method for dual intensity and/or radiography enhancing breast imaging
US8933888B2 (en) 2011-03-17 2015-01-13 Intellitact Llc Relative touch user interface enhancements
US8479110B2 (en) 2011-03-20 2013-07-02 William J. Johnson System and method for summoning user interface objects
US8836802B2 (en) 2011-03-21 2014-09-16 Honeywell International Inc. Method of defining camera scan movements using gestures
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US8724146B2 (en) 2011-03-28 2014-05-13 Apple Inc. Systems and methods for defining print settings using device movements
JP5784944B2 (en) * 2011-03-29 2015-09-24 京セラ株式会社 Electronics
WO2012129670A1 (en) 2011-03-31 2012-10-04 Smart Technologies Ulc Manipulating graphical objects γν a multi-touch interactive system
US9298363B2 (en) 2011-04-11 2016-03-29 Apple Inc. Region activation for touch sensitive surface
US8854324B2 (en) 2011-04-13 2014-10-07 Sony Corporation Information processing control device
GB2504890A (en) 2011-04-15 2014-02-12 Faro Tech Inc Enhanced position detector in laser tracker
US9482529B2 (en) 2011-04-15 2016-11-01 Faro Technologies, Inc. Three-dimensional coordinate scanner and method of operation
US9164173B2 (en) 2011-04-15 2015-10-20 Faro Technologies, Inc. Laser tracker that uses a fiber-optic coupler and an achromatic launch to align and collimate two wavelengths of light
US9686532B2 (en) 2011-04-15 2017-06-20 Faro Technologies, Inc. System and method of acquiring three-dimensional coordinates using multiple coordinate measurement devices
CA2833544A1 (en) * 2011-04-18 2012-10-26 Eyesee360, Inc. Apparatus and method for panoramic video imaging with mobile computing devices
BR112013026624A2 (en) * 2011-04-20 2016-12-27 Koninkl Philips Nv apparatus for controlling an element or an item, device, method for controlling an element or an item, computer program product and storage medium
US20120272180A1 (en) * 2011-04-20 2012-10-25 Nokia Corporation Method and apparatus for providing content flipping based on a scrolling operation
US8873841B2 (en) * 2011-04-21 2014-10-28 Nokia Corporation Methods and apparatuses for facilitating gesture recognition
US9261972B2 (en) 2011-04-21 2016-02-16 Inpris Innovative Products Ltd Ergonomic motion detection for receiving character input to electronic devices
US10146428B2 (en) * 2011-04-21 2018-12-04 Inpris Innovative Products From Israel Ltd Device, system, and methods for entering commands or characters using a touch screen
JP5962654B2 (en) * 2011-05-12 2016-08-03 日本電気株式会社 Electronic device, control method thereof, and program
US8896555B2 (en) * 2011-05-20 2014-11-25 Robert H Duffield Touch alphabet and communication system
US9032338B2 (en) 2011-05-30 2015-05-12 Apple Inc. Devices, methods, and graphical user interfaces for navigating and editing text
US8677232B2 (en) 2011-05-31 2014-03-18 Apple Inc. Devices, methods, and graphical user interfaces for document manipulation
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
KR101838260B1 (en) 2011-06-03 2018-03-13 구글 엘엘씨 Gestures for selecting text
US9727227B2 (en) * 2011-07-28 2017-08-08 Microsoft Technology Licensing, Llc Multi-touch remoting
US9317196B2 (en) 2011-08-10 2016-04-19 Microsoft Technology Licensing, Llc Automatic zooming for text selection/cursor placement
US9754085B2 (en) 2011-08-17 2017-09-05 Integrated Chemistry Design, Inc. Systems and methods of editing a chemical structure on a touch-screen
US8994660B2 (en) 2011-08-29 2015-03-31 Apple Inc. Text correction processing
RU2014112207A (en) 2011-08-31 2015-10-10 Корос Отомоутив Ко., Лтд. INTERACTIVE VEHICLE SYSTEM
JP5790380B2 (en) * 2011-09-28 2015-10-07 株式会社Jvcケンウッド Electronic device, control method of electronic device, and program
TWI528252B (en) * 2011-10-28 2016-04-01 緯創資通股份有限公司 Touch panel
US10082950B2 (en) * 2011-11-09 2018-09-25 Joseph T. LAPP Finger-mapped character entry systems
TWI571790B (en) * 2011-11-10 2017-02-21 財團法人資訊工業策進會 Method and electronic device for changing coordinate values of icons according to a sensing signal
JP2013105461A (en) * 2011-11-16 2013-05-30 Canon Inc Information processing apparatus and method of controlling the same
KR102109588B1 (en) 2011-11-27 2020-05-12 홀로직, 인크. Methods for processing, displaying and navigating breast tissue images
US8581870B2 (en) * 2011-12-06 2013-11-12 Apple Inc. Touch-sensitive button with two levels
JP5846887B2 (en) * 2011-12-13 2016-01-20 京セラ株式会社 Mobile terminal, edit control program, and edit control method
JP6080355B2 (en) 2011-12-21 2017-02-15 京セラ株式会社 Apparatus, method, and program
US9176666B2 (en) * 2011-12-23 2015-11-03 Symbol Technologies, Llc Method and device for a multi-touch based correction of a handwriting sentence system
WO2013095677A1 (en) * 2011-12-23 2013-06-27 Intel Corporation Computing system utilizing three-dimensional manipulation command gestures
EP2795430A4 (en) 2011-12-23 2015-08-19 Intel Ip Corp Transition mechanism for computing system utilizing user sensing
WO2013095678A1 (en) 2011-12-23 2013-06-27 Intel Corporation Mechanism to provide feedback regarding computing system command gestures
US9684379B2 (en) 2011-12-23 2017-06-20 Intel Corporation Computing system utilizing coordinated two-hand command gestures
TWI493438B (en) * 2012-01-09 2015-07-21 Amtran Technology Co Ltd Touch control method
CN103207756A (en) * 2012-01-13 2013-07-17 瑞轩科技股份有限公司 Touch control method
US9638507B2 (en) 2012-01-27 2017-05-02 Faro Technologies, Inc. Measurement machine utilizing a barcode to identify an inspection plan for an object
US9323322B2 (en) 2012-02-02 2016-04-26 Smart Technologies Ulc Interactive input system and method of detecting objects
KR20130090585A (en) * 2012-02-06 2013-08-14 삼성전자주식회사 Wearable robot and teaching method of motion using the same
WO2013123091A1 (en) 2012-02-13 2013-08-22 Hologic, Inc. System and method for navigating a tomosynthesis stack using synthesized image data
US11071344B2 (en) 2012-02-22 2021-07-27 Nike, Inc. Motorized shoe with gesture control
US11684111B2 (en) 2012-02-22 2023-06-27 Nike, Inc. Motorized shoe with gesture control
US20130213147A1 (en) 2012-02-22 2013-08-22 Nike, Inc. Footwear Having Sensor System
US9977876B2 (en) * 2012-02-24 2018-05-22 Perkinelmer Informatics, Inc. Systems, methods, and apparatus for drawing chemical structures using touch and gestures
US9600169B2 (en) 2012-02-27 2017-03-21 Yahoo! Inc. Customizable gestures for mobile devices
US8796566B2 (en) 2012-02-28 2014-08-05 Grayhill, Inc. Rotary pushbutton and touchpad device and system and method for detecting rotary movement, axial displacement and touchpad gestures
KR101318244B1 (en) 2012-02-29 2013-10-15 한국과학기술연구원 System and Method for Implemeting 3-Dimensional User Interface
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US9292197B2 (en) * 2012-03-30 2016-03-22 Mckesson Financial Holdings Method, apparatus and computer program product for facilitating the manipulation of medical images
US9772700B2 (en) * 2012-04-30 2017-09-26 Blackberry Limited Device and method for processing user input
US9280610B2 (en) 2012-05-14 2016-03-08 Apple Inc. Crowd sourcing information to fulfill user requests
US20130311947A1 (en) * 2012-05-16 2013-11-21 Ekata Systems, Inc. Network image sharing with synchronized image display and manipulation
CN106133748B (en) * 2012-05-18 2020-01-31 苹果公司 Device, method and graphical user interface for manipulating a user interface based on fingerprint sensor input
CN103229127A (en) * 2012-05-21 2013-07-31 华为技术有限公司 Method and device for contact-free control by hand gesture
US9721563B2 (en) 2012-06-08 2017-08-01 Apple Inc. Name recognition system
US20130339859A1 (en) 2012-06-15 2013-12-19 Muzik LLC Interactive networked headphones
US20130335340A1 (en) * 2012-06-19 2013-12-19 International Business Machines Corporation Controlling display of images received from secondary display devices
US20130346892A1 (en) * 2012-06-25 2013-12-26 Google Inc. Graphical user interface element expansion and contraction using a rotating gesture
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US9785338B2 (en) * 2012-07-02 2017-10-10 Mosaiqq, Inc. System and method for providing a user interaction interface using a multi-touch gesture recognition engine
US9268457B2 (en) 2012-07-13 2016-02-23 Google Inc. Touch-based fluid window management
CN103576948A (en) * 2012-07-23 2014-02-12 英华达(上海)科技有限公司 Touch electronic device and digital position signal selecting method thereof
CN102819350B (en) * 2012-08-02 2016-04-06 东莞宇龙通信科技有限公司 Terminal and terminal control method
US9547647B2 (en) 2012-09-19 2017-01-17 Apple Inc. Voice-based media searching
US9164625B2 (en) 2012-10-14 2015-10-20 Neonode Inc. Proximity sensor for determining two-dimensional coordinates of a proximal object
US9921661B2 (en) 2012-10-14 2018-03-20 Neonode Inc. Optical proximity sensor and associated user interface
US10282034B2 (en) 2012-10-14 2019-05-07 Neonode Inc. Touch sensitive curved and flexible displays
US9207800B1 (en) 2014-09-23 2015-12-08 Neonode Inc. Integrated light guide and touch screen frame and multi-touch determination method
KR20140051719A (en) * 2012-10-23 2014-05-02 엘지전자 주식회사 Mobile terminal and control method thereof
US8949735B2 (en) 2012-11-02 2015-02-03 Google Inc. Determining scroll direction intent
US9459760B2 (en) * 2012-11-16 2016-10-04 Xiaomi Inc. Method and device for managing a user interface
US9535583B2 (en) 2012-12-13 2017-01-03 Perkinelmer Informatics, Inc. Draw-ahead feature for chemical structure drawing applications
US9063575B2 (en) 2012-12-20 2015-06-23 Synaptics Incorporated Detecting a gesture
JP5958326B2 (en) * 2012-12-21 2016-07-27 カシオ計算機株式会社 Dictionary search device, dictionary search method, dictionary search program, dictionary search system, server device, terminal device
CN103902080A (en) * 2012-12-27 2014-07-02 华硕电脑股份有限公司 Touch device and touch processing method
US9743861B2 (en) 2013-02-01 2017-08-29 Nike, Inc. System and method for analyzing athletic activity
US11006690B2 (en) 2013-02-01 2021-05-18 Nike, Inc. System and method for analyzing athletic activity
US10926133B2 (en) 2013-02-01 2021-02-23 Nike, Inc. System and method for analyzing athletic activity
CN105190644B (en) * 2013-02-01 2020-12-18 英特尔公司 Techniques for image-based searching using touch control
KR20140100316A (en) * 2013-02-06 2014-08-14 엘지전자 주식회사 Mobile terminal and control method thereof
JP2014153936A (en) 2013-02-08 2014-08-25 Japan Display Inc Touch position detection device and touch position detection method
MY167493A (en) * 2013-02-20 2018-08-30 Panasonic Ip Corp America Program and Method for Controlling Portable Information Terminal
KR102144587B1 (en) * 2013-02-21 2020-08-13 삼성전자주식회사 Apparatus and method for editing display information in an electronic device
CN104010125B (en) * 2013-02-22 2017-11-28 联想(北京)有限公司 Electronic equipment and method
KR102117086B1 (en) * 2013-03-08 2020-06-01 삼성디스플레이 주식회사 Terminal and method for controlling thereof
US8854361B1 (en) 2013-03-13 2014-10-07 Cambridgesoft Corporation Visually augmenting a graphical rendering of a chemical structure representation or biological sequence representation with multi-dimensional information
US9279734B2 (en) 2013-03-15 2016-03-08 Nike, Inc. System and method for analyzing athletic activity
JP6388347B2 (en) 2013-03-15 2018-09-12 ホロジック, インコーポレイテッドHologic, Inc. Tomosynthesis guided biopsy in prone position
US9041914B2 (en) 2013-03-15 2015-05-26 Faro Technologies, Inc. Three-dimensional coordinate scanner and method of operation
EP2984550A1 (en) * 2013-04-08 2016-02-17 Rohde & Schwarz GmbH & Co. KG Multitouch gestures for a measurement system
US9430127B2 (en) 2013-05-08 2016-08-30 Cambridgesoft Corporation Systems and methods for providing feedback cues for touch screen interface interaction with chemical and biological structure drawing applications
US9751294B2 (en) 2013-05-09 2017-09-05 Perkinelmer Informatics, Inc. Systems and methods for translating three dimensional graphic molecular models to computer aided design format
DE102013009009B4 (en) 2013-05-17 2023-08-03 Elektrobit Automotive Gmbh System and method for data selection using a touch-sensitive surface
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
WO2014197334A2 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
WO2014197336A1 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
WO2014197335A1 (en) 2013-06-08 2014-12-11 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
KR101959188B1 (en) 2013-06-09 2019-07-02 애플 인크. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US9733716B2 (en) 2013-06-09 2017-08-15 Apple Inc. Proxy gesture recognizer
KR102034587B1 (en) 2013-08-29 2019-10-21 엘지전자 주식회사 Mobile terminal and controlling method thereof
US10114486B2 (en) * 2013-09-19 2018-10-30 Change Healthcare Holdings, Llc Method and apparatus for providing touch input via a touch sensitive surface utilizing a support object
US10152136B2 (en) * 2013-10-16 2018-12-11 Leap Motion, Inc. Velocity field interaction for free space gesture interface and control
US9531722B1 (en) 2013-10-31 2016-12-27 Google Inc. Methods for generating an activity stream
US9542457B1 (en) 2013-11-07 2017-01-10 Google Inc. Methods for displaying object history information
US9614880B1 (en) 2013-11-12 2017-04-04 Google Inc. Methods for real-time notifications in an activity stream
KR102206053B1 (en) * 2013-11-18 2021-01-21 삼성전자주식회사 Apparatas and method for changing a input mode according to input method in an electronic device
US9507520B2 (en) 2013-12-16 2016-11-29 Microsoft Technology Licensing, Llc Touch-based reorganization of page element
US10126822B2 (en) 2013-12-16 2018-11-13 Leap Motion, Inc. User-defined virtual interaction space and manipulation of virtual configuration
US9594893B2 (en) * 2014-01-15 2017-03-14 Lenovo (Singapore) Pte. Ltd. Multi-touch local device authentication
US9715286B2 (en) 2014-01-28 2017-07-25 Solid Art Labs, Inc. Hand-controllable signal-generating devices and systems
US10169957B2 (en) 2014-02-13 2019-01-01 Igt Multiple player gaming station interaction systems and methods
US9509772B1 (en) 2014-02-13 2016-11-29 Google Inc. Visualization and control of ongoing ingress actions
ES2943561T3 (en) 2014-02-28 2023-06-14 Hologic Inc System and method for generating and visualizing tomosynthesis image blocks
US9690478B2 (en) 2014-03-04 2017-06-27 Texas Instruments Incorporated Method and system for processing gestures to cause computation of measurement of an angle or a segment using a touch system
CN104932797A (en) * 2014-03-17 2015-09-23 深圳富泰宏精密工业有限公司 Gesture unlocking method and system
RU2652457C2 (en) * 2014-03-31 2018-04-26 Общество с ограниченной ответственностью "Аби Девелопмент" Multi-touch input processing for selection of the displayed option
KR102278507B1 (en) * 2014-04-15 2021-07-16 삼성디스플레이 주식회사 Wearable device
KR102265143B1 (en) * 2014-05-16 2021-06-15 삼성전자주식회사 Apparatus and method for processing input
WO2015181162A1 (en) * 2014-05-28 2015-12-03 Thomson Licensing Method and system for touch input
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US9978043B2 (en) 2014-05-30 2018-05-22 Apple Inc. Automatic event scheduling
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US9966065B2 (en) 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US9898162B2 (en) 2014-05-30 2018-02-20 Apple Inc. Swiping functions for messaging applications
US9971500B2 (en) 2014-06-01 2018-05-15 Apple Inc. Displaying options, assigning notification, ignoring messages, and simultaneous user interface displays in a messaging application
US9575560B2 (en) 2014-06-03 2017-02-21 Google Inc. Radar-based gesture-recognition through a wearable device
US9536199B1 (en) 2014-06-09 2017-01-03 Google Inc. Recommendations based on device usage
US9507791B2 (en) 2014-06-12 2016-11-29 Google Inc. Storage system user interface with floating file collection
US10078781B2 (en) 2014-06-13 2018-09-18 Google Llc Automatically organizing images
US9999280B2 (en) 2014-06-27 2018-06-19 David Gareth Zebley Interactive bracelet for practicing an activity between user devices
US9395174B2 (en) 2014-06-27 2016-07-19 Faro Technologies, Inc. Determining retroreflector orientation by optimizing spatial fit
AU2015279544B2 (en) 2014-06-27 2018-03-15 Apple Inc. Electronic device with rotatable input mechanism for navigating calendar application
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US10659851B2 (en) 2014-06-30 2020-05-19 Apple Inc. Real-time digital assistant knowledge updates
US9811164B2 (en) 2014-08-07 2017-11-07 Google Inc. Radar-based gesture sensing and data transmission
US10268321B2 (en) 2014-08-15 2019-04-23 Google Llc Interactive textiles within hard objects
US11169988B2 (en) 2014-08-22 2021-11-09 Google Llc Radar recognition-aided search
US9778749B2 (en) 2014-08-22 2017-10-03 Google Inc. Occluded gesture recognition
US10446141B2 (en) 2014-08-28 2019-10-15 Apple Inc. Automatic speech recognition based on user feedback
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US10789041B2 (en) 2014-09-12 2020-09-29 Apple Inc. Dynamic thresholds for always listening speech trigger
US10296206B2 (en) 2014-09-23 2019-05-21 Microsoft Technology Licensing, Llc Multi-finger touchpad gestures
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
KR102238535B1 (en) * 2014-10-01 2021-04-09 엘지전자 주식회사 Mobile terminal and method for controlling the same
US9600080B2 (en) 2014-10-02 2017-03-21 Google Inc. Non-line-of-sight radar-based gesture recognition
CN107210950A (en) 2014-10-10 2017-09-26 沐择歌有限责任公司 Equipment for sharing user mutual
KR102213897B1 (en) * 2014-10-31 2021-02-08 삼성전자주식회사 A method for selecting one or more items according to an user input and an electronic device therefor
US9619043B2 (en) 2014-11-26 2017-04-11 At&T Intellectual Property I, L.P. Gesture multi-function on a physical keyboard
US10552013B2 (en) 2014-12-02 2020-02-04 Apple Inc. Data detection
DE102014224632A1 (en) * 2014-12-02 2016-06-02 Robert Bosch Gmbh Method for operating an input device, input device
US9870420B2 (en) 2015-01-19 2018-01-16 Google Llc Classification and storage of documents
US10031638B2 (en) * 2015-02-10 2018-07-24 Etter Studio Ltd. Multi-touch GUI featuring directional compression and expansion of graphical content
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US10120567B2 (en) 2015-04-02 2018-11-06 Inpris Innovative Products From Israel Ltd System, apparatus and method for vehicle command and control
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
CN107430443B (en) 2015-04-30 2020-07-10 谷歌有限责任公司 Gesture recognition based on wide field radar
EP3289433A1 (en) 2015-04-30 2018-03-07 Google LLC Type-agnostic rf signal representations
KR102236958B1 (en) 2015-04-30 2021-04-05 구글 엘엘씨 Rf-based micro-motion tracking for gesture tracking and recognition
US10088908B1 (en) * 2015-05-27 2018-10-02 Google Llc Gesture detection and interactions
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US9693592B2 (en) 2015-05-27 2017-07-04 Google Inc. Attaching electronic components to interactive textiles
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US9578173B2 (en) 2015-06-05 2017-02-21 Apple Inc. Virtual assistant aided communication with 3rd party service in a communication session
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
JP6041939B2 (en) * 2015-07-23 2016-12-14 京セラ株式会社 Electronics
WO2017029555A2 (en) * 2015-08-20 2017-02-23 Inpris Innovative Products Ltd Device, system, and methods for entering commands or characters using a touch screen
WO2017034425A1 (en) * 2015-08-21 2017-03-02 Motorola Solutions, Inc. System and method for disambiguating touch interactions
CN106502553A (en) * 2015-09-08 2017-03-15 中强光电股份有限公司 Gesture interaction operational approach
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US9516662B1 (en) * 2015-09-11 2016-12-06 WashSense, Inc. Beacon disambiguation system
US9633546B2 (en) 2015-09-11 2017-04-25 WashSense, Inc. Touchless compliance system
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US10503361B2 (en) 2015-09-30 2019-12-10 Samsung Electronics Company, Ltd. Interactive graphical object
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US10817065B1 (en) 2015-10-06 2020-10-27 Google Llc Gesture recognition using multiple antenna
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
US10492302B2 (en) 2016-05-03 2019-11-26 Google Llc Connecting an electronic component to an interactive textile
US10285456B2 (en) 2016-05-16 2019-05-14 Google Llc Interactive fabric
WO2017200570A1 (en) 2016-05-16 2017-11-23 Google Llc Interactive object with multiple electronics modules
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
DK179309B1 (en) 2016-06-09 2018-04-23 Apple Inc Intelligent automated assistant in a home environment
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
US10620812B2 (en) 2016-06-10 2020-04-14 Apple Inc. Device, method, and graphical user interface for managing electronic communications
US10586535B2 (en) 2016-06-10 2020-03-10 Apple Inc. Intelligent digital assistant in a multi-tasking environment
DK179415B1 (en) 2016-06-11 2018-06-14 Apple Inc Intelligent device arbitration and control
DK201670540A1 (en) 2016-06-11 2018-01-08 Apple Inc Application integration with a digital assistant
DK179049B1 (en) 2016-06-11 2017-09-18 Apple Inc Data driven natural language event detection and classification
DK179343B1 (en) 2016-06-11 2018-05-14 Apple Inc Intelligent task discovery
WO2018006129A1 (en) * 2016-07-08 2018-01-11 Blanchard Mehdi Gesture-based input command interface, method and system
US10444983B2 (en) * 2016-09-20 2019-10-15 Rohde & Schwarz Gmbh & Co. Kg Signal analyzing instrument with touch gesture control and method of operating thereof
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US11073980B2 (en) 2016-09-29 2021-07-27 Microsoft Technology Licensing, Llc User interfaces for bi-manual control
US10283082B1 (en) 2016-10-29 2019-05-07 Dvir Gassner Differential opacity position indicator
US10579150B2 (en) 2016-12-05 2020-03-03 Google Llc Concurrent detection of absolute distance and relative movement for sensing action gestures
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
JP7175605B2 (en) 2017-01-25 2022-11-21 キヤノンメディカルシステムズ株式会社 ultrasound diagnostic equipment
JP2017084427A (en) * 2017-02-17 2017-05-18 日本電信電話株式会社 Input device, input support method and input support program
US10572545B2 (en) 2017-03-03 2020-02-25 Perkinelmer Informatics, Inc Systems and methods for searching and indexing documents comprising chemical information
JP7169986B2 (en) 2017-03-30 2022-11-11 ホロジック, インコーポレイテッド Systems and methods for synthesizing low-dimensional image data from high-dimensional image data using object grid augmentation
WO2018183550A1 (en) 2017-03-30 2018-10-04 Hologic, Inc. System and method for targeted object enhancement to generate synthetic breast tissue images
DK201770439A1 (en) 2017-05-11 2018-12-13 Apple Inc. Offline personal assistant
DK179745B1 (en) 2017-05-12 2019-05-01 Apple Inc. SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT
DK179496B1 (en) 2017-05-12 2019-01-15 Apple Inc. USER-SPECIFIC Acoustic Models
DK201770431A1 (en) 2017-05-15 2018-12-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
DK201770432A1 (en) 2017-05-15 2018-12-21 Apple Inc. Hierarchical belief states for digital assistants
DK179560B1 (en) 2017-05-16 2019-02-18 Apple Inc. Far-field extension for digital assistant services
US10698561B2 (en) * 2017-06-12 2020-06-30 Google Llc Intelligent command batching in an augmented and/or virtual reality environment
WO2018236565A1 (en) 2017-06-20 2018-12-27 Hologic, Inc. Dynamic self-learning medical image method and system
US11449167B2 (en) 2017-06-26 2022-09-20 Inpris Innovative Products Fromisrael, Ltd Systems using dual touch and sound control, and methods thereof
US11797150B2 (en) * 2018-05-07 2023-10-24 Apple Inc. Devices, methods, and graphical user interfaces for navigating between user interfaces, displaying a dock, and displaying system user interface elements
DK180116B1 (en) 2018-05-07 2020-05-13 Apple Inc. Devices, methods, and graphical user interfaces for navigating between user interfaces and displaying a dock
AU2019100488B4 (en) 2018-05-07 2019-08-22 Apple Inc. Devices, methods, and graphical user interfaces for navigating between user interfaces, displaying a dock, and displaying system user interface elements
US11875012B2 (en) 2018-05-25 2024-01-16 Ultrahaptics IP Two Limited Throwable interface for augmented reality and virtual reality environments
US11669243B2 (en) 2018-06-03 2023-06-06 Apple Inc. Systems and methods for activating and using a trackpad at an electronic device with a touch-sensitive display and no force sensors
US10776006B2 (en) 2018-06-03 2020-09-15 Apple Inc. Systems and methods for activating and using a trackpad at an electronic device with a touch-sensitive display and no force sensors
WO2020112585A1 (en) 2018-11-28 2020-06-04 Neonode Inc. Motorist user interface sensor
CN109766054B (en) * 2019-01-31 2021-02-02 恒生电子股份有限公司 Touch screen device and control method and medium thereof
DK180359B1 (en) 2019-04-15 2021-02-03 Apple Inc Accelerated scrolling and selection
US10996761B2 (en) 2019-06-01 2021-05-04 Apple Inc. User interfaces for non-visual output of time
US11194467B2 (en) 2019-06-01 2021-12-07 Apple Inc. Keyboard management user interfaces
US11379113B2 (en) 2019-06-01 2022-07-05 Apple Inc. Techniques for selecting text
KR20210004567A (en) * 2019-07-05 2021-01-13 엘지이노텍 주식회사 Electronic device
US11842014B2 (en) 2019-12-31 2023-12-12 Neonode Inc. Contactless touch input system
US11409410B2 (en) 2020-09-14 2022-08-09 Apple Inc. User input interfaces
KR20230074269A (en) 2020-09-30 2023-05-26 네오노드, 인크. optical touch sensor
CN113157185B (en) * 2021-05-17 2022-07-15 上海临冠数据科技有限公司 Gesture operation method and device for undo redo
CN113110793B (en) * 2021-05-17 2022-07-15 上海临冠数据科技有限公司 Gesture operation method and device for realizing undo and redo
US20230107097A1 (en) * 2021-10-06 2023-04-06 Fotonation Limited Method for identifying a gesture

Citations (93)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3333160A (en) 1964-02-24 1967-07-25 Water Economy And Res Company Proximity responsive system
US3541541A (en) 1967-06-21 1970-11-17 Stanford Research Inst X-y position indicator for a display system
US3662105A (en) 1970-05-21 1972-05-09 Univ Kentucky Res Found Electrical sensor of plane coordinates
US3798370A (en) 1972-04-17 1974-03-19 Elographics Inc Electrographic sensor for determining planar coordinates
US4246452A (en) 1979-01-05 1981-01-20 Mattel, Inc. Switch apparatus
US4550221A (en) 1983-10-07 1985-10-29 Scott Mabusth Touch sensitive control device
US4672558A (en) 1984-09-25 1987-06-09 Aquila Technologies Group, Inc. Touch-sensitive data input device
US4672364A (en) 1984-06-18 1987-06-09 Carroll Touch Inc Touch input device having power profiling
US4692809A (en) 1984-11-20 1987-09-08 Hughes Aircraft Company Integrated touch paint system for displays
US4695827A (en) 1984-11-20 1987-09-22 Hughes Aircraft Company Electromagnetic energy interference seal for light beam touch panels
US4733222A (en) 1983-12-27 1988-03-22 Integrated Touch Arrays, Inc. Capacitance-variation-sensitive touch sensing array system
US4734685A (en) 1983-07-28 1988-03-29 Canon Kabushiki Kaisha Position control apparatus
US4746770A (en) 1987-02-17 1988-05-24 Sensor Frame Incorporated Method and apparatus for isolating and manipulating graphic objects on computer video monitor
US4771276A (en) 1985-04-15 1988-09-13 International Business Machines Corporation Electromagnetic touch sensor input system in a cathode ray tube display device
US4788384A (en) 1986-12-18 1988-11-29 Centre National De La Recherche Scientifique Device for two-dimensional localization of events that generate current on a resistive surface
US4806846A (en) 1987-07-06 1989-02-21 Kerber George L High accuracy direct reading capacitance-to-voltage converter
US4898555A (en) 1989-03-23 1990-02-06 Bell & Howell Publication Systems Company Display screen bezel and assembly method
US4968877A (en) 1988-09-14 1990-11-06 Sensor Frame Corporation VideoHarp
US5003519A (en) 1988-05-26 1991-03-26 Eta S.A. Fabriques D'ebauches Alarm arrangement for a timepiece
US5017030A (en) 1986-07-07 1991-05-21 Crews Jay A Ergonomically designed keyboard
US5178477A (en) 1991-06-06 1993-01-12 Gambaro Thomas L Ergonomic keyboard input device
US5189403A (en) 1989-09-26 1993-02-23 Home Row, Inc. Integrated keyboard and pointing device system with automatic mode change
US5194862A (en) 1990-06-29 1993-03-16 U.S. Philips Corporation Touch sensor array systems and display systems incorporating such
US5224861A (en) 1990-09-17 1993-07-06 Hughes Aircraft Company Training device onboard instruction station
US5241308A (en) 1990-02-22 1993-08-31 Paragon Systems, Inc. Force sensitive touch panel
US5252951A (en) 1989-04-28 1993-10-12 International Business Machines Corporation Graphical user interface with gesture recognition in a multiapplication environment
US5281966A (en) 1992-01-31 1994-01-25 Walsh A Peter Method of encoding alphabetic characters for a chord keyboard
US5305017A (en) 1989-08-16 1994-04-19 Gerpheide George E Methods and apparatus for data input
US5345543A (en) 1992-11-16 1994-09-06 Apple Computer, Inc. Method for manipulating objects on a computer display
US5376948A (en) 1992-03-25 1994-12-27 Visage, Inc. Method of and apparatus for touch-input computer and related display employing touch force location external to the display
US5398310A (en) 1992-04-13 1995-03-14 Apple Computer, Incorporated Pointing gesture based computer note pad paging and scrolling interface
US5442742A (en) 1990-12-21 1995-08-15 Apple Computer, Inc. Method and apparatus for the manipulation of text on a computer display screen
US5463696A (en) 1992-05-27 1995-10-31 Apple Computer, Inc. Recognition system and method for user inputs to a computer system
US5463388A (en) 1993-01-29 1995-10-31 At&T Ipm Corp. Computer mouse or keyboard input device utilizing capacitive sensors
US5483261A (en) 1992-02-14 1996-01-09 Itu Research, Inc. Graphical input controller and method with rear screen image detection
US5488204A (en) 1992-06-08 1996-01-30 Synaptics, Incorporated Paintbrush stylus for capacitive touch sensor pad
US5495077A (en) 1992-06-08 1996-02-27 Synaptics, Inc. Object position and proximity detector
US5513309A (en) 1993-01-05 1996-04-30 Apple Computer, Inc. Graphic editor user interface for a pointer-based computer system
US5523775A (en) 1992-05-26 1996-06-04 Apple Computer, Inc. Method for selecting objects on a computer display
US5530455A (en) 1994-08-10 1996-06-25 Mouse Systems Corporation Roller mouse for implementing scrolling in windows applications
US5543591A (en) 1992-06-08 1996-08-06 Synaptics, Incorporated Object position detector with edge motion feature and gesture recognition
US5543590A (en) 1992-06-08 1996-08-06 Synaptics, Incorporated Object position detector with edge motion feature
US5563632A (en) 1993-04-30 1996-10-08 Microtouch Systems, Inc. Method of and apparatus for the elimination of the effects of internal interference in force measurement systems, including touch - input computer and related displays employing touch force location measurement techniques
US5563996A (en) 1992-04-13 1996-10-08 Apple Computer, Inc. Computer note pad including gesture based note division tools and method
US5565658A (en) 1992-07-13 1996-10-15 Cirque Corporation Capacitance-based proximity with interference rejection apparatus and methods
US5579036A (en) 1994-04-28 1996-11-26 At&T Global Information Solutions Company Touch screen device and shielding bracket therefor
US5583946A (en) 1993-09-30 1996-12-10 Apple Computer, Inc. Method and apparatus for recognizing gestures on a computer system
US5592566A (en) 1992-05-27 1997-01-07 Apple Computer, Incorporated Method and apparatus for computerized recognition
US5596694A (en) 1992-05-27 1997-01-21 Apple Computer, Inc. Method and apparatus for indicating a change in status of an object and its disposition using animation
US5612719A (en) 1992-12-03 1997-03-18 Apple Computer, Inc. Gesture sensitive buttons for graphical user interfaces
US5631805A (en) 1995-09-27 1997-05-20 Dynapro Systems, Inc. Touch screen enclosure having an insertable graphic sheet
US5634102A (en) 1995-08-07 1997-05-27 Apple Computer, Inc. Methods and apparatus for a selectable backdrop
US5633955A (en) 1992-05-27 1997-05-27 Apple Computer, Inc. Method of connecting shapes on a display of a computer system
US5636101A (en) 1995-09-27 1997-06-03 Dynapro Systems, Inc. Touch screen enclosure system having touch screen pan and hinged rear enclosure section for ease of serviceability
US5642108A (en) 1991-06-28 1997-06-24 Infogrip, Inc. Chordic keyboard system for generating a signal in response to a chord that is assigned using a correlation based on a composite chord-difficulty index
US5644657A (en) 1992-05-27 1997-07-01 Apple Computer, Inc. Method for locating and displaying information in a pointer-based computer system
US5666113A (en) 1991-07-31 1997-09-09 Microtouch Systems, Inc. System for using a touchpad input device for cursor control and keyboard emulation
US5666502A (en) 1995-08-07 1997-09-09 Apple Computer, Inc. Graphical user interface using historical lists with field classes
US5675361A (en) 1995-08-23 1997-10-07 Santilli; Donald S. Computer keyboard pointing device
US5677710A (en) 1993-05-10 1997-10-14 Apple Computer, Inc. Recognition keypad
US5689253A (en) 1991-04-10 1997-11-18 Kinesis Corporation Ergonomic keyboard apparatus
US5729250A (en) 1995-05-08 1998-03-17 International Business Machines Corporation Front cover assembly for a touch sensitive device
US5730165A (en) 1995-12-26 1998-03-24 Philipp; Harald Time domain capacitive field detector
US5736976A (en) 1995-02-13 1998-04-07 Cheung; Nina T. Computer data entry apparatus with hand motion sensing and monitoring
US5741990A (en) 1989-02-17 1998-04-21 Notepool, Ltd. Method of and means for producing musical note relationships
US5745716A (en) 1995-08-07 1998-04-28 Apple Computer, Inc. Method and apparatus for tab access and tab cycling in a pen-based computer system
US5745116A (en) 1996-09-09 1998-04-28 Motorola, Inc. Intuitive gesture-based graphical user interface
US5748269A (en) 1996-11-21 1998-05-05 Westinghouse Air Brake Company Environmentally-sealed, convectively-cooled active matrix liquid crystal display (LCD)
US5746818A (en) 1995-08-31 1998-05-05 Seiko Epson Corporation Pigment ink composition capable of forming image having no significant bleeding or feathering
US5764222A (en) 1996-05-28 1998-06-09 International Business Machines Corporation Virtual pointing device for touchscreens
US5767842A (en) 1992-02-07 1998-06-16 International Business Machines Corporation Method and device for optical input of commands or data
US5767457A (en) 1995-11-13 1998-06-16 Cirque Corporation Apparatus and method for audible feedback from input device
US5790104A (en) 1996-06-25 1998-08-04 International Business Machines Corporation Multiple, moveable, customizable virtual pointing devices
US5790107A (en) 1995-06-07 1998-08-04 Logitech, Inc. Touch sensing method and apparatus
US5802516A (en) 1993-11-03 1998-09-01 Apple Computer, Inc. Method of controlling an electronic book for a computer system
US5809267A (en) 1993-12-30 1998-09-15 Xerox Corporation Apparatus and method for executing multiple-concatenated command gestures in a gesture based input system
US5808567A (en) 1993-05-17 1998-09-15 Dsi Datotech Systems, Inc. Apparatus and method of communicating using three digits of a hand
US5821930A (en) 1992-08-23 1998-10-13 U S West, Inc. Method and system for generating a working window in a computer system
US5821690A (en) 1993-08-26 1998-10-13 Cambridge Display Technology Limited Electroluminescent devices having a light-emitting layer
US5825351A (en) 1994-05-12 1998-10-20 Apple Computer, Inc. Method and apparatus for noise filtering for an input device
US5823782A (en) 1995-12-29 1998-10-20 Tinkers & Chance Character recognition educational system
US5825352A (en) 1996-01-04 1998-10-20 Logitech, Inc. Multiple fingers contact sensing method for emulating mouse buttons and mouse operations on a touch sensor pad
US5854625A (en) 1996-11-06 1998-12-29 Synaptics, Incorporated Force sensing touchpad
US5880411A (en) 1992-06-08 1999-03-09 Synaptics, Incorporated Object position detector with edge motion feature and gesture recognition
US5898434A (en) 1991-05-15 1999-04-27 Apple Computer, Inc. User interface system having programmable user interface elements
US5920309A (en) 1996-01-04 1999-07-06 Logitech, Inc. Touch sensing method and apparatus
US5933134A (en) 1996-06-25 1999-08-03 International Business Machines Corporation Touch screen virtual pointing device which goes into a translucent hibernation state when not in use
US5943044A (en) 1996-08-05 1999-08-24 Interlink Electronics Force sensing semiconductive touchpad
US6002808A (en) 1996-07-26 1999-12-14 Mitsubishi Electric Information Technology Center America, Inc. Hand gesture control system
US6002389A (en) 1996-04-24 1999-12-14 Logitech, Inc. Touch and pressure sensing method and apparatus
US6020881A (en) 1993-05-24 2000-02-01 Sun Microsystems Graphical user interface with method and apparatus for interfacing to remote devices
US6031524A (en) 1995-06-07 2000-02-29 Intermec Ip Corp. Hand-held portable data terminal having removably interchangeable, washable, user-replaceable components with liquid-impervious seal
US6037882A (en) 1997-09-30 2000-03-14 Levy; David H. Method and apparatus for inputting data to an electronic system

Family Cites Families (108)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4549279A (en) 1983-01-21 1985-10-22 The Laitram Corporation Single hand, single finger stroke alphameric data processing keyboard system
GB2178570A (en) 1985-06-07 1987-02-11 Remanco Systems Inc Computer overlay keyboard
GB2204131B (en) 1987-04-28 1991-04-17 Ibm Graphics input tablet
CA1336418C (en) 1989-08-29 1995-07-25 Edgar Matias One-handed keyboard
US5367298A (en) 1991-10-25 1994-11-22 Axthelm John K Data input terminal
JP3244798B2 (en) * 1992-09-08 2002-01-07 株式会社東芝 Moving image processing device
US5336002A (en) 1992-09-18 1994-08-09 Russo Malcolm G One-handed alphanumeric keyboard and method
US6072489A (en) 1993-05-10 2000-06-06 Apple Computer, Inc. Method and apparatus for providing translucent images on a computer display
US5638501A (en) 1993-05-10 1997-06-10 Apple Computer, Inc. Method and apparatus for displaying an overlay image
DE4344050C1 (en) * 1993-12-23 1995-03-09 Henry Rudolph Entry method and entry apparatus for computer terminals for the purpose of protecting against covert observation and for ensuring the privacy of a user
EP0667567B1 (en) 1993-12-30 2001-10-17 Xerox Corporation Apparatus and method for supporting the implicit structure of freeform lists, outlines, text, tables, and diagrams in a gesture-based input system and editing system
US5572573A (en) 1994-01-25 1996-11-05 U S West Advanced Technologies, Inc. Removable user interface for use with interactive electronic devices
JPH0863326A (en) * 1994-08-22 1996-03-08 Hitachi Ltd Image processing device/method
US5591945A (en) 1995-04-19 1997-01-07 Elo Touchsystems, Inc. Acoustic touch position sensor using higher order horizontally polarized shear wave propagation
JPH09146708A (en) * 1995-11-09 1997-06-06 Internatl Business Mach Corp <Ibm> Driving method for touch panel and touch input method
US6107997A (en) * 1996-06-27 2000-08-22 Ure; Michael J. Touch-sensitive keyboard/mouse and computing device using the same
US5856824A (en) * 1996-06-25 1999-01-05 International Business Machines Corp. Reshapable pointing device for touchscreens
US6288707B1 (en) 1996-07-29 2001-09-11 Harald Philipp Capacitive position sensor
US6208329B1 (en) * 1996-08-13 2001-03-27 Lsi Logic Corporation Supplemental mouse button emulation system, method and apparatus for a coordinate based data input device
US5917476A (en) * 1996-09-24 1999-06-29 Czerniecki; George V. Cursor feedback text input method
US6650319B1 (en) 1996-10-29 2003-11-18 Elo Touchsystems, Inc. Touch screen based topological mapping with resistance framing design
DE69626208T2 (en) 1996-12-20 2003-11-13 Hitachi Europ Ltd Method and system for recognizing hand gestures
US6289326B1 (en) 1997-06-04 2001-09-11 Lafleur Bernard B. Portable interactive kiosk
CH691711A5 (en) 1997-06-11 2001-09-14 Asulab Sa Watch with tactile reading and adjustment of horological functions.
US6084576A (en) * 1997-09-27 2000-07-04 Leu; Neng-Chyang User friendly keyboard
US6072494A (en) 1997-10-15 2000-06-06 Electric Planet, Inc. Method and apparatus for real-time gesture recognition
GB2330670B (en) 1997-10-24 2002-09-11 Sony Uk Ltd Data processing
US6057845A (en) 1997-11-14 2000-05-02 Sensiva, Inc. System, method, and apparatus for generation and recognizing universal commands
US7614008B2 (en) 2004-07-30 2009-11-03 Apple Inc. Operation of a computer with touch screen interface
US7663607B2 (en) 2004-05-06 2010-02-16 Apple Inc. Multipoint touchscreen
US8479122B2 (en) 2004-07-30 2013-07-02 Apple Inc. Gestures for touch sensitive input devices
US7844914B2 (en) 2004-07-30 2010-11-30 Apple Inc. Activating virtual keys of a touch-screen virtual keyboard
US20060033724A1 (en) 2004-07-30 2006-02-16 Apple Computer, Inc. Virtual input device placement on a touch screen user interface
US7760187B2 (en) 2004-07-30 2010-07-20 Apple Inc. Visual expander
EP2256605B1 (en) 1998-01-26 2017-12-06 Apple Inc. Method and apparatus for integrating manual input
US6088022A (en) * 1998-02-06 2000-07-11 Rakoski; Robert Charles Spherical keyboard with built-in mouse
US6249606B1 (en) 1998-02-19 2001-06-19 Mindmaker, Inc. Method and system for gesture category recognition and training using a feature vector
US6639577B2 (en) 1998-03-04 2003-10-28 Gemstar-Tv Guide International, Inc. Portable information display device with ergonomic bezel
US6198515B1 (en) 1998-03-16 2001-03-06 Joseph W. Cole Apparatus and method for controlled sealing between bezel and CRT
US6050825A (en) 1998-05-08 2000-04-18 Nichol; Claudia D. Opaque, one-size-fits-all computer keyboard cover which covers only the three or four alpha-numeric rows
JP3792920B2 (en) 1998-12-25 2006-07-05 株式会社東海理化電機製作所 Touch operation input device
US6347290B1 (en) 1998-06-24 2002-02-12 Compaq Information Technologies Group, L.P. Apparatus and method for detecting and executing positional and gesture commands corresponding to movement of handheld computing device
US6131299A (en) 1998-07-01 2000-10-17 Faro Technologies, Inc. Display device for a coordinate measurement machine
US6188391B1 (en) 1998-07-09 2001-02-13 Synaptics, Inc. Two-layer capacitive touchpad and method of making same
US6135958A (en) 1998-08-06 2000-10-24 Acuson Corporation Ultrasound imaging system with touch-pad pointing device
US6392637B2 (en) 1998-08-13 2002-05-21 Dell Usa, L.P. Computer system having a configurable touchpad-mouse button combination
KR100437652B1 (en) 1998-09-04 2004-06-30 이노베이티브 솔루션즈 앤드 서포트 인코포레이티드 Flat panel display using dual cpu's for an aircraft cockpit
US6292178B1 (en) 1998-10-19 2001-09-18 Allergan Sales, Inc. Screen navigation control apparatus for ophthalmic surgical instruments
JP2000132305A (en) * 1998-10-23 2000-05-12 Olympus Optical Co Ltd Operation input device
US6466036B1 (en) 1998-11-25 2002-10-15 Harald Philipp Charge transfer capacitance measurement circuit
US6222465B1 (en) 1998-12-09 2001-04-24 Lucent Technologies Inc. Gesture-based computer interface
US6323849B1 (en) 1999-01-22 2001-11-27 Motorola, Inc. Display module with reduced power consumption
US6535200B2 (en) 1999-01-25 2003-03-18 Harald Philipp Capacitive position sensor
JP4275865B2 (en) 1999-01-26 2009-06-10 キューアールジー リミテッド Capacitive sensors and arrays
US6246862B1 (en) 1999-02-03 2001-06-12 Motorola, Inc. Sensor controlled user interface for portable communication device
US6142687A (en) 1999-03-16 2000-11-07 Lisak; Stephen J. One handed sequential alpha numerical keyboard
US6677934B1 (en) 1999-07-30 2004-01-13 L-3 Communications Infrared touch panel with improved sunlight rejection
GB9920301D0 (en) 1999-08-27 1999-11-03 Philipp Harald Level sensing
US6377009B1 (en) 1999-09-08 2002-04-23 Harald Philipp Capacitive closure obstruction sensor
US6411287B1 (en) 1999-09-08 2002-06-25 Elo Touchsystems, Inc. Stress seal for acoustic wave touchscreens
KR101192279B1 (en) * 1999-10-27 2012-10-17 피루쯔 가사비안 Integrated Keypad System
US20030132950A1 (en) 2001-11-27 2003-07-17 Fahri Surucu Detecting, classifying, and interpreting input events based on stimuli in multiple sensory domains
US6757002B1 (en) * 1999-11-04 2004-06-29 Hewlett-Packard Development Company, L.P. Track pad pointing device with areas of specialized function
US6543684B1 (en) 2000-03-28 2003-04-08 Ncr Corporation Transaction terminal with privacy shield for touch-screen pin entry
US6624833B1 (en) * 2000-04-17 2003-09-23 Lucent Technologies Inc. Gesture-based input interface system with shadow detection
US6803906B1 (en) 2000-07-05 2004-10-12 Smart Technologies, Inc. Passive touch system and method of detecting user input
US6421234B1 (en) 2000-10-10 2002-07-16 Juniper Systems, Inc. Handheld electronics device having ergonomic features
US6942571B1 (en) 2000-10-16 2005-09-13 Bally Gaming, Inc. Gaming device with directional and speed control of mechanical reels using touch screen
US6593916B1 (en) 2000-11-03 2003-07-15 James L. Aroyan Touchscreen having multiple parallel connections to each electrode in a series resistor chain on the periphery of the touch area
US6677932B1 (en) * 2001-01-28 2004-01-13 Finger Works, Inc. System and method for recognizing touch typing under limited tactile feedback conditions
US6670894B2 (en) * 2001-02-05 2003-12-30 Carsten Mehring System and method for keyboard independent touch typing
US7030861B1 (en) * 2001-02-10 2006-04-18 Wayne Carl Westerman System and method for packing multi-touch gestures onto a hand
US6570557B1 (en) * 2001-02-10 2003-05-27 Finger Works, Inc. Multi-touch system and method for emulating modifier keys via fingertip chords
US7133531B2 (en) 2001-02-27 2006-11-07 Nissim Karpenstein Device using analog controls to mix compressed digital audio data
US6543947B2 (en) 2001-03-14 2003-04-08 Lee Jose H. Keyboard having keys arranged in a pan configuration
US6603462B2 (en) * 2001-03-21 2003-08-05 Multidigit, Inc. System and method for selecting functions based on a finger feature such as a fingerprint
US6724366B2 (en) 2001-04-03 2004-04-20 Peter James Crawford Thumb actuated x-y input device
US6965375B1 (en) 2001-04-27 2005-11-15 Palm, Inc. Compact integrated touch panel display for a handheld device
US6992659B2 (en) 2001-05-22 2006-01-31 Palmone, Inc. High transparency integrated enclosure touch screen assembly for a portable hand held device
US6670951B2 (en) 2001-07-03 2003-12-30 Hewlett-Packard Development Company, L.P. Methods and systems for increasing the input efficiency of personal digital assistants and other handheld stylus-engagable computing devices
US7312785B2 (en) 2001-10-22 2007-12-25 Apple Inc. Method and apparatus for accelerated scrolling
US7084856B2 (en) 2001-10-22 2006-08-01 Apple Computer, Inc. Mouse having a rotary dial
US7046230B2 (en) 2001-10-22 2006-05-16 Apple Computer, Inc. Touch pad handheld device
US7345671B2 (en) 2001-10-22 2008-03-18 Apple Inc. Method and apparatus for use of rotational user inputs
US7009599B2 (en) 2001-11-20 2006-03-07 Nokia Corporation Form factor for portable device
US6762752B2 (en) 2001-11-29 2004-07-13 N-Trig Ltd. Dual function input device and method
DE10251296A1 (en) 2002-11-03 2004-05-19 Trachte, Ralf, Dr. Device for computer input has a touch-sensitive input surface with multiple input surface areas and a control unit coupled to the input surface.
US6900795B1 (en) 2002-02-27 2005-05-31 Knight, Iii Jack Unitary molded lens filter for touch screen interface
US6985801B1 (en) 2002-02-28 2006-01-10 Garmin International, Inc. Cockpit instrument panel systems and methods with redundant flight data display
US6832138B1 (en) 2002-02-28 2004-12-14 Garmin International, Inc. Cockpit instrument panel systems and methods with redundant flight data display
US6927761B2 (en) 2002-03-29 2005-08-09 3M Innovative Properties Company Moisture deflector for capacitive NFI touch screens for use with bezels of conductive material
US7038659B2 (en) * 2002-04-06 2006-05-02 Janusz Wiktor Rajkowski Symbol encoding apparatus and method
US6658994B1 (en) 2002-04-10 2003-12-09 Chromalox, Inc. Modular assembly for a holding cabinet controller
JP2003323259A (en) 2002-05-02 2003-11-14 Nec Corp Information processing apparatus
JP3852368B2 (en) 2002-05-16 2006-11-29 ソニー株式会社 Input method and data processing apparatus
US11275405B2 (en) 2005-03-04 2022-03-15 Apple Inc. Multi-functional hand-held device
EP1394640B1 (en) 2002-08-30 2010-03-24 Asulab S.A. Timepiece with tactile reading and operation of time information
GB0220290D0 (en) 2002-08-31 2002-10-09 Mcgrath Peter Handheld device control system
US6972401B2 (en) 2003-01-30 2005-12-06 Smart Technologies Inc. Illuminated bezel and touch system incorporating the same
US20040263484A1 (en) 2003-06-25 2004-12-30 Tapio Mantysalo Multifunctional UI input device for moblie terminals
US8373660B2 (en) 2003-07-14 2013-02-12 Matt Pallakoff System and method for a portable multimedia client
US7499040B2 (en) 2003-08-18 2009-03-03 Apple Inc. Movable touch pad with added functionality
US9024884B2 (en) 2003-09-02 2015-05-05 Apple Inc. Touch-sensitive electronic apparatus for media applications, and methods therefor
US7495659B2 (en) 2003-11-25 2009-02-24 Apple Inc. Touch pad for handheld device
US6856259B1 (en) 2004-02-06 2005-02-15 Elo Touchsystems, Inc. Touch sensor system to detect multiple touch events
WO2006023569A1 (en) 2004-08-16 2006-03-02 Fingerworks, Inc. A method of increasing the spatial resolution of touch sensitive devices
US7719522B2 (en) 2004-09-24 2010-05-18 Apple Inc. Raw data track pad device and system
US20060256090A1 (en) 2005-05-12 2006-11-16 Apple Computer, Inc. Mechanical overlay

Patent Citations (99)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3333160A (en) 1964-02-24 1967-07-25 Water Economy And Res Company Proximity responsive system
US3541541A (en) 1967-06-21 1970-11-17 Stanford Research Inst X-y position indicator for a display system
US3662105A (en) 1970-05-21 1972-05-09 Univ Kentucky Res Found Electrical sensor of plane coordinates
US3798370A (en) 1972-04-17 1974-03-19 Elographics Inc Electrographic sensor for determining planar coordinates
US4246452A (en) 1979-01-05 1981-01-20 Mattel, Inc. Switch apparatus
US4734685A (en) 1983-07-28 1988-03-29 Canon Kabushiki Kaisha Position control apparatus
US4550221A (en) 1983-10-07 1985-10-29 Scott Mabusth Touch sensitive control device
US4733222A (en) 1983-12-27 1988-03-22 Integrated Touch Arrays, Inc. Capacitance-variation-sensitive touch sensing array system
US4672364A (en) 1984-06-18 1987-06-09 Carroll Touch Inc Touch input device having power profiling
US4672558A (en) 1984-09-25 1987-06-09 Aquila Technologies Group, Inc. Touch-sensitive data input device
US4695827A (en) 1984-11-20 1987-09-22 Hughes Aircraft Company Electromagnetic energy interference seal for light beam touch panels
US4692809A (en) 1984-11-20 1987-09-08 Hughes Aircraft Company Integrated touch paint system for displays
US4771276A (en) 1985-04-15 1988-09-13 International Business Machines Corporation Electromagnetic touch sensor input system in a cathode ray tube display device
US5017030A (en) 1986-07-07 1991-05-21 Crews Jay A Ergonomically designed keyboard
US4788384A (en) 1986-12-18 1988-11-29 Centre National De La Recherche Scientifique Device for two-dimensional localization of events that generate current on a resistive surface
US4746770A (en) 1987-02-17 1988-05-24 Sensor Frame Incorporated Method and apparatus for isolating and manipulating graphic objects on computer video monitor
US4806846A (en) 1987-07-06 1989-02-21 Kerber George L High accuracy direct reading capacitance-to-voltage converter
US5003519A (en) 1988-05-26 1991-03-26 Eta S.A. Fabriques D'ebauches Alarm arrangement for a timepiece
US4968877A (en) 1988-09-14 1990-11-06 Sensor Frame Corporation VideoHarp
US5741990A (en) 1989-02-17 1998-04-21 Notepool, Ltd. Method of and means for producing musical note relationships
US4898555A (en) 1989-03-23 1990-02-06 Bell & Howell Publication Systems Company Display screen bezel and assembly method
US5252951A (en) 1989-04-28 1993-10-12 International Business Machines Corporation Graphical user interface with gesture recognition in a multiapplication environment
US5305017A (en) 1989-08-16 1994-04-19 Gerpheide George E Methods and apparatus for data input
US5189403A (en) 1989-09-26 1993-02-23 Home Row, Inc. Integrated keyboard and pointing device system with automatic mode change
US5241308A (en) 1990-02-22 1993-08-31 Paragon Systems, Inc. Force sensitive touch panel
US5194862A (en) 1990-06-29 1993-03-16 U.S. Philips Corporation Touch sensor array systems and display systems incorporating such
US5224861A (en) 1990-09-17 1993-07-06 Hughes Aircraft Company Training device onboard instruction station
US5666552A (en) 1990-12-21 1997-09-09 Apple Computer, Inc. Method and apparatus for the manipulation of text on a computer display screen
US5442742A (en) 1990-12-21 1995-08-15 Apple Computer, Inc. Method and apparatus for the manipulation of text on a computer display screen
US5689253A (en) 1991-04-10 1997-11-18 Kinesis Corporation Ergonomic keyboard apparatus
US5898434A (en) 1991-05-15 1999-04-27 Apple Computer, Inc. User interface system having programmable user interface elements
US5178477A (en) 1991-06-06 1993-01-12 Gambaro Thomas L Ergonomic keyboard input device
US5642108A (en) 1991-06-28 1997-06-24 Infogrip, Inc. Chordic keyboard system for generating a signal in response to a chord that is assigned using a correlation based on a composite chord-difficulty index
US5666113A (en) 1991-07-31 1997-09-09 Microtouch Systems, Inc. System for using a touchpad input device for cursor control and keyboard emulation
US5281966A (en) 1992-01-31 1994-01-25 Walsh A Peter Method of encoding alphabetic characters for a chord keyboard
US5767842A (en) 1992-02-07 1998-06-16 International Business Machines Corporation Method and device for optical input of commands or data
US5483261A (en) 1992-02-14 1996-01-09 Itu Research, Inc. Graphical input controller and method with rear screen image detection
US5376948A (en) 1992-03-25 1994-12-27 Visage, Inc. Method of and apparatus for touch-input computer and related display employing touch force location external to the display
US5398310A (en) 1992-04-13 1995-03-14 Apple Computer, Incorporated Pointing gesture based computer note pad paging and scrolling interface
US5581681A (en) 1992-04-13 1996-12-03 Apple Computer, Inc. Pointing gesture based computer note pad paging and scrolling interface
US5563996A (en) 1992-04-13 1996-10-08 Apple Computer, Inc. Computer note pad including gesture based note division tools and method
US5523775A (en) 1992-05-26 1996-06-04 Apple Computer, Inc. Method for selecting objects on a computer display
US5463696A (en) 1992-05-27 1995-10-31 Apple Computer, Inc. Recognition system and method for user inputs to a computer system
US5592566A (en) 1992-05-27 1997-01-07 Apple Computer, Incorporated Method and apparatus for computerized recognition
US5710844A (en) 1992-05-27 1998-01-20 Apple Computer Method for searching and displaying results in a pen-based computer system
US5644657A (en) 1992-05-27 1997-07-01 Apple Computer, Inc. Method for locating and displaying information in a pointer-based computer system
US5633955A (en) 1992-05-27 1997-05-27 Apple Computer, Inc. Method of connecting shapes on a display of a computer system
US5596694A (en) 1992-05-27 1997-01-21 Apple Computer, Inc. Method and apparatus for indicating a change in status of an object and its disposition using animation
US5880411A (en) 1992-06-08 1999-03-09 Synaptics, Incorporated Object position detector with edge motion feature and gesture recognition
US5543590A (en) 1992-06-08 1996-08-06 Synaptics, Incorporated Object position detector with edge motion feature
US5488204A (en) 1992-06-08 1996-01-30 Synaptics, Incorporated Paintbrush stylus for capacitive touch sensor pad
US5495077A (en) 1992-06-08 1996-02-27 Synaptics, Inc. Object position and proximity detector
US5543591A (en) 1992-06-08 1996-08-06 Synaptics, Incorporated Object position detector with edge motion feature and gesture recognition
US5565658A (en) 1992-07-13 1996-10-15 Cirque Corporation Capacitance-based proximity with interference rejection apparatus and methods
US5821930A (en) 1992-08-23 1998-10-13 U S West, Inc. Method and system for generating a working window in a computer system
US5345543A (en) 1992-11-16 1994-09-06 Apple Computer, Inc. Method for manipulating objects on a computer display
US5612719A (en) 1992-12-03 1997-03-18 Apple Computer, Inc. Gesture sensitive buttons for graphical user interfaces
US5513309A (en) 1993-01-05 1996-04-30 Apple Computer, Inc. Graphic editor user interface for a pointer-based computer system
US5463388A (en) 1993-01-29 1995-10-31 At&T Ipm Corp. Computer mouse or keyboard input device utilizing capacitive sensors
US5563632A (en) 1993-04-30 1996-10-08 Microtouch Systems, Inc. Method of and apparatus for the elimination of the effects of internal interference in force measurement systems, including touch - input computer and related displays employing touch force location measurement techniques
US5677710A (en) 1993-05-10 1997-10-14 Apple Computer, Inc. Recognition keypad
US5808567A (en) 1993-05-17 1998-09-15 Dsi Datotech Systems, Inc. Apparatus and method of communicating using three digits of a hand
US6020881A (en) 1993-05-24 2000-02-01 Sun Microsystems Graphical user interface with method and apparatus for interfacing to remote devices
US5821690A (en) 1993-08-26 1998-10-13 Cambridge Display Technology Limited Electroluminescent devices having a light-emitting layer
US5583946A (en) 1993-09-30 1996-12-10 Apple Computer, Inc. Method and apparatus for recognizing gestures on a computer system
US5590219A (en) 1993-09-30 1996-12-31 Apple Computer, Inc. Method and apparatus for recognizing gestures on a computer system
US5594810A (en) 1993-09-30 1997-01-14 Apple Computer, Inc. Method and apparatus for recognizing gestures on a computer system
US5802516A (en) 1993-11-03 1998-09-01 Apple Computer, Inc. Method of controlling an electronic book for a computer system
US5809267A (en) 1993-12-30 1998-09-15 Xerox Corporation Apparatus and method for executing multiple-concatenated command gestures in a gesture based input system
US5579036A (en) 1994-04-28 1996-11-26 At&T Global Information Solutions Company Touch screen device and shielding bracket therefor
US5825351A (en) 1994-05-12 1998-10-20 Apple Computer, Inc. Method and apparatus for noise filtering for an input device
US5530455A (en) 1994-08-10 1996-06-25 Mouse Systems Corporation Roller mouse for implementing scrolling in windows applications
US5736976A (en) 1995-02-13 1998-04-07 Cheung; Nina T. Computer data entry apparatus with hand motion sensing and monitoring
US5729250A (en) 1995-05-08 1998-03-17 International Business Machines Corporation Front cover assembly for a touch sensitive device
US5923319A (en) 1995-05-08 1999-07-13 International Business Machines Corporation Front cover assembly for touch sensitive device
US5790107A (en) 1995-06-07 1998-08-04 Logitech, Inc. Touch sensing method and apparatus
US6031524A (en) 1995-06-07 2000-02-29 Intermec Ip Corp. Hand-held portable data terminal having removably interchangeable, washable, user-replaceable components with liquid-impervious seal
US5634102A (en) 1995-08-07 1997-05-27 Apple Computer, Inc. Methods and apparatus for a selectable backdrop
US5745716A (en) 1995-08-07 1998-04-28 Apple Computer, Inc. Method and apparatus for tab access and tab cycling in a pen-based computer system
US5666502A (en) 1995-08-07 1997-09-09 Apple Computer, Inc. Graphical user interface using historical lists with field classes
US5675361A (en) 1995-08-23 1997-10-07 Santilli; Donald S. Computer keyboard pointing device
US5746818A (en) 1995-08-31 1998-05-05 Seiko Epson Corporation Pigment ink composition capable of forming image having no significant bleeding or feathering
US5636101A (en) 1995-09-27 1997-06-03 Dynapro Systems, Inc. Touch screen enclosure system having touch screen pan and hinged rear enclosure section for ease of serviceability
US5631805A (en) 1995-09-27 1997-05-20 Dynapro Systems, Inc. Touch screen enclosure having an insertable graphic sheet
US5767457A (en) 1995-11-13 1998-06-16 Cirque Corporation Apparatus and method for audible feedback from input device
US5730165A (en) 1995-12-26 1998-03-24 Philipp; Harald Time domain capacitive field detector
US5823782A (en) 1995-12-29 1998-10-20 Tinkers & Chance Character recognition educational system
US5825352A (en) 1996-01-04 1998-10-20 Logitech, Inc. Multiple fingers contact sensing method for emulating mouse buttons and mouse operations on a touch sensor pad
US5920309A (en) 1996-01-04 1999-07-06 Logitech, Inc. Touch sensing method and apparatus
US6002389A (en) 1996-04-24 1999-12-14 Logitech, Inc. Touch and pressure sensing method and apparatus
US5764222A (en) 1996-05-28 1998-06-09 International Business Machines Corporation Virtual pointing device for touchscreens
US5933134A (en) 1996-06-25 1999-08-03 International Business Machines Corporation Touch screen virtual pointing device which goes into a translucent hibernation state when not in use
US5790104A (en) 1996-06-25 1998-08-04 International Business Machines Corporation Multiple, moveable, customizable virtual pointing devices
US6002808A (en) 1996-07-26 1999-12-14 Mitsubishi Electric Information Technology Center America, Inc. Hand gesture control system
US5943044A (en) 1996-08-05 1999-08-24 Interlink Electronics Force sensing semiconductive touchpad
US5745116A (en) 1996-09-09 1998-04-28 Motorola, Inc. Intuitive gesture-based graphical user interface
US5854625A (en) 1996-11-06 1998-12-29 Synaptics, Incorporated Force sensing touchpad
US5748269A (en) 1996-11-21 1998-05-05 Westinghouse Air Brake Company Environmentally-sealed, convectively-cooled active matrix liquid crystal display (LCD)
US6037882A (en) 1997-09-30 2000-03-14 Levy; David H. Method and apparatus for inputting data to an electronic system

Non-Patent Citations (83)

* Cited by examiner, † Cited by third party
Title
"4-Wire Resistive Touchscreens" obtained from http://www.touchscreens.com/intro-touchtypes-4resistive.html generated Aug. 5, 2005.
"5-Wire Resistive Touchscreens" obtained from http://www.touchscreens.com/intro-touchtypes-resistive.html generated Aug. 5, 2005.
"A Brief Overview of Gesture Recognition" obtained from http://www.dai.ed.ac.uk/Cvonline/LOCA_COPIES/COHEN/gesture_overview.html, generated Apr. 20, 2004.
"Capacitive Position Sensing" obtained from http://www.synaptics.com/technology/cps.cfm generated Aug. 5, 2005.
"Capacitive Touchscreens" obtained from http://www.touchscreens.com/intro-touchtypes-capacitive.html generated Aug. 5, 2005.
"Comparing Touch Technologies" obtained from http://www.touchscreens.com/intro-touchtypes.html generated Oct. 10, 2004.
"FingerWorks-Gesture Guide-Application Switching," obtained from http://www.fingerworks.com/gesture_guide_apps.html, generated on Aug. 27, 2004, 1-pg.
"FingerWorks-Gesture Guide-Editing," obtained from http://www.fingerworks.com/gesure_guide_editing.html, generated on Aug. 27, 2004, 1-pg.
"FingerWorks-Gesture Guide-File Operations," obtained from http://www.fingerworks.com/gesture_guide_files.html, generated on Aug. 27, 2004, 1-pg.
"FingerWorks-Gesture Guide-Text Manipulation," obtained from http://www.fingerworks.com/gesture_guide_text_manip.html, generated on Aug. 27, 2004, 2-pg.
"FingerWorks-Gesture Guide-Tips and Tricks," obtained from http://www.fingerworks.com/gesture_guide_tips.html, generated Aug. 27, 2004, 2-pgs.
"FingerWorks-Gesture Guide-Web," obtained from http://www.fingerworks.com/gesture_guide_web.html, generated on Aug. 27, 2004, 1-pg.
"FingerWorks-Guide to Hand Gestures for USB Touchpads," obtained from http://www.fingerworks.com/igesture_userguide.html, generated Aug. 27, 2004, 1-pg.
"FingerWorks-iGesture-Technical Details," obtained from http://www.fingerworks.com/igesture_tech.html, generated Aug. 27, 2004, 1-pg.
"FingerWorks-The Only Touchpads with Ergonomic Full-Hand Resting and Relaxation!" obtained from http://www.fingerworks.com/resting.html, Copyright 2001, 1-pg.
"FingerWorks-Tips for Typing on the Mini," obtained from http://www.fingerworks.com/mini_typing.html, generated on Aug. 27, 2004, 2-pgs.
"Gesture Recognition" http://www.fingerworks.com/gesture_recognition.html.
"GlidePoint(R)" obtained from http://www.cirque.com/technology/technology_gp.html generated Aug. 5, 2005.
"How do touchscreen monitors know where you're touching?" obtained from http://www.electronics.howstuffworks.com/question716.html generated Aug. 5, 2005.
"How does a touchscreen work?" obtained from http://www.touchscreens.com/intro-anatomy.html generated Aug. 5, 2005.
"iGesture Pad-the MultiFinger USB TouchPad with Whole-Hand Gestures," obtained from http://www.fingerworks.com/igesture.html, generated Aug. 27, 2004, 2-pgs.
"iGesture Products for Everyone (learn in minutes) Product Overview" FingerWorks.com.
"Infrared Touchscreens" obtained from http://www.touchscreens.com/intro-touchtypes-infrared.html generated Aug. 5, 2005.
"Mouse Emulation" FingerWorks obtained from http://www.fingerworks.com/gesture_guide_mouse.html generated Aug. 30, 2005.
"Mouse Gestures in Opera" obtained from http://www.opera.com/products/desktop/mouse/index.dml generated Aug. 30, 2005.
"Mouse Gestures," Optim oz, May 21, 2004.
"MultiTouch Overview" FingerWorks obtained from http://www.fingerworks.com/multoverview.html generated Aug. 30, 2005.
"Near Field Imaging Touchscreens" obtained from http://www.touchscreens.com/intro-touchtypes.nfi.html generated Aug. 5, 2005.
"PenTouch Capacitive Touchscreens" obtained from http://www.touchscreens.com/intro-touchtypes-pentouch.html generated Aug. 5, 2005.
"Surface Acoustic Wave Touchscreens" obtained from http://www.touchscreens.com/intro-touchtypes.saw.html generated Aug. 5, 2005.
"Symbol Commander" obtained from http://www.sensiva.com/symbolcomander/, generated Aug. 30, 2005.
"Tips for Typing" FingerWorks http://www.fingerworks.com/mini_typing.html generated Aug. 30, 2005.
"Touch Technologies Overview" 2001, 3M Touch Systems, Massachusetts.
"Wacom Components-Technology" obtained from http://www.wacom-components.com/english/tech.asp generated on Oct. 10, 2004.
"Watershed Algorithm" http://rsb.info.nih.gov/ij/plugins/watershed.html generated Aug. 5, 2005.
Bier, et al., "Toolglass and Magic Lenses: The see-through interface" In James Kijiya, editor, Computer Graphics (SIGGRAPH '93 Proceedings), vol. 27, pp. 73-80, Aug. 1993.
Douglas et al., The Ergonomics of Computer Pointing Devices (1997).
European Search Report received in EP 1 621 989 (@ Beyer Weaver & Thomas, LLP) dated Mar. 27, 2006.
EVB Elektronik "TSOP6238 IR Receiver Modules for Infrared Remote Control Systems" dated Jan. 2004 1-pg.
Fisher et al., "Repetitive Motion Disorders: The Design of Optimal Rate-Rest Profiles," Human Factors, 35(2):283-304 (Jun. 1993).
Fukumoto and Yoshinobu Tonomura, "Body Coupled Fingering: Wireless Wearable Keyboard," CHI 97, pp. 147-154 (Mar. 1997).
Fukumoto, et al., "ActiveClick: Tactile Feedback for Touch Panels," In CHI 2001 Summary, pp. 121-122, 2001.
Hardy, "Fingerworks" Mar. 7, 2002; BBC World On Line.
Hillier and Gerald J. Lieberman, Introduction to Operations Research (1986).
International Search Report dated Mar. 3, 2006 (PCT/US 05/03325; 119-0052WO).
International Search Report received in corresponding PCT application No. PCT/US2006/008349 dated Oct. 6, 2006.
Jacob et al., "Integrality and Separability of Input Devices," ACM Transactions on Computer-Human Interaction, 1:3-26 (Mar. 1994).
Kinkley et al., "Touch-Sensing Input Devices," in CHI '99 Proceedings, pp. 223-230, 1999.
Kionx "KXP84 Series Summary Data Sheet" copyright 2005,dated Oct. 21, 2005, 4-pgs.
Lee et al., "A Multi-Touch Three Dimensional Touch-Sensitive Tablet," in CHI '85 Proceedings, pp. 121-128, 2000.
Lee, "A Fast Multiple-Touch-Sensitive Input Device," Master's Thesis, University of Toronto (1984).
Matsushita et al., "HoloWall: Designing a Finger, Hand, Body and Object Sensitive Wall," In Proceedings of UIST '97, Oct. 1997.
Quantum Research Group "QT510 / QWheel(TM) Touch Slider IC" copyright 2004-2005, 14-pgs.
Quek, "Unencumbered Gestural Interaction," IEEE Multimedia, 3:36-47 (Winter 1996).
Radwin, "Activation Force and Travel Effects on Overexertion in Repetitive Key Tapping," Human Factors, 39(1):130-140 (Mar. 1997).
Rekimoto "SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces" CHI 2002, Apr. 20-25, 2002.
Rekimoto et al., "ToolStone: Effective Use of the Physical Manipulation Vocabularies of Input Devices," In Proc. Of UIST 2000, 2000.
Rubine et al., "Programmable Finger-Tracking Instrument Controllers" Computer Music Journal, vol. 14, No. 1 (Spring 1990).
Rutledge et al., "Force-To-Motion Functions For Pointing," Human-Computer Interaction-INTERACT (1990).
Subatai Ahmad, "A Usable Real-Time 3D Hand Tracker," Proceedings of the 28<SUP>th </SUP>Asilomar Conference on Signals, Systems and Computers-Part 2 (of2), vol. 2 (Oct. 1994).
Texas Instruments "TSC2003 / I2C Touch Screen Controller" Data Sheet SBAS 162, dated Oct. 2001, 20-pgs.
U.S. Appl. No. 10/654,108, filed Sep. 2, 2003 entitled "Ambidextrous Mouse".
U.S. Appl. No. 10/774,053, filed on Feb. 5, 2004.
U.S. Appl. No. 10/789,676, filed Feb. 27, 2004 entitled "Shape Detecting Input Device".
U.S. Appl. No. 11/140,529, filed May 27, 2005 which is a Reissue of 6,570,557.
U.S. Appl. No. 11/278,080, filed Mar. 30, 2006 entitled "Force Imaging Input Device and System".
U.S. Appl. No. 11/332,861, filed Jan. 13, 2006 which is a Reiuuse of 6,677,932.
U.S. Appl. No. 11/380,109, filed Apr. 25, 2006 entitled "Keystroke Tactility Arrangement On Smooth Touch Surface."
U.S. Appl. No. 11/381,313, filed May 2, 2006 entitled "Multipoint Touch Surface Controller".
U.S. Appl. No. 11/382,402, filed May 9, 2006 entitled "Force and Location Sensitive Display" which is a Continuation of U.S. Appl. No. 11/278,080.
U.S. Appl. No. 11/426,078, filed Jun. 23, 2006 entitled "Electronic Device Having Display and Surrounding Touch Sensitive Bezel For User Interface and Control" which is a Continuation-In-Part of 2006/0197753.
U.S. Appl. No. 11/428,501, filed Jul. 3, 2006 entitled "Capacitive Sensing Arrangement," which is a Continuation of US 2005/0104867.
U.S. Appl. No. 11/428,503, filed Jul. 3, 2006 entitled "Touch Surface" which is a Continuation of US 2005/0104867.
U.S. Appl. No. 11/428,506, filed Jul. 3, 2006 entitled "User Interface Gestures" which is a Continuation of US 2005/0104867.
U.S. Appl. No. 11/428,515, filed Jul. 3, 2006 entitled "User Interface Gestures" which is a Continuation of US 2005/0104867.
U.S. Appl. No. 11/428,521, filed Jul. 3, 2006 entitled "Identifying Contacts on a Touch Surface" which is a Continuation of US 2005/0104867.
U.S. Appl. No. 11/428,522, filed Jul. 3, 2006 entitled "Identifying Contacts on a Touch Surface" which is a Continuation of US 2005/0104867.
Wellner, "The Digital Desk Calculators: Tangible Manipulation on a Desk Top Display" IN ACM UIST '91 Proceedings, pp. 27-34, Nov. 1991.
Williams, "Applications for a Switched-Capacitor Instrumentation Building Block" Linear Technology Application Note 3, Jul. 1985, pp. 1-16.
Yamada et al., "A Switched-Capacitor Interface for Capacitive Presssure Sensors" IEEE Transactions on Instrumentation and Measurement, vol. 41, No. 1, Feb. 1992, pp. 81-86.
Yeh et al., "Switched Capacitor Interface Circuit for Capacitive Transducers" 1985 IEEE.
Zhai et al., "Dual Stream Input for Pointing and Scrolling," Proceedings of CHI '97 Extended Abstracts (1997).
Zimmerman et al., "Applying Electric Field Sensing to Human-Computer Interfaces," In CHI '85 Proceedings, pp. 280-287, 1995.

Cited By (225)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9513744B2 (en) 1994-08-15 2016-12-06 Apple Inc. Control systems employing novel physical controls and touch screens
US20090273575A1 (en) * 1995-06-29 2009-11-05 Pryor Timothy R Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US8610674B2 (en) 1995-06-29 2013-12-17 Apple Inc. Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US8427449B2 (en) 1995-06-29 2013-04-23 Apple Inc. Method for providing human input to a computer
US8228305B2 (en) 1995-06-29 2012-07-24 Apple Inc. Method for providing human input to a computer
US9758042B2 (en) 1995-06-29 2017-09-12 Apple Inc. Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US20090322499A1 (en) * 1995-06-29 2009-12-31 Pryor Timothy R Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US20090300531A1 (en) * 1995-06-29 2009-12-03 Pryor Timothy R Method for providing human input to a computer
US20090273574A1 (en) * 1995-06-29 2009-11-05 Pryor Timothy R Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
USRE45559E1 (en) 1997-10-28 2015-06-09 Apple Inc. Portable computers
USRE46548E1 (en) 1997-10-28 2017-09-12 Apple Inc. Portable computers
US9348452B2 (en) 1998-01-26 2016-05-24 Apple Inc. Writing using a touch sensor
US9804701B2 (en) 1998-01-26 2017-10-31 Apple Inc. Contact tracking and identification module for touch sensing
US8730177B2 (en) 1998-01-26 2014-05-20 Apple Inc. Contact tracking and identification module for touch sensing
US8736555B2 (en) 1998-01-26 2014-05-27 Apple Inc. Touch sensing through hand dissection
US8698755B2 (en) 1998-01-26 2014-04-15 Apple Inc. Touch sensor contact information
US9329717B2 (en) 1998-01-26 2016-05-03 Apple Inc. Touch sensing with mobile sensors
US8674943B2 (en) 1998-01-26 2014-03-18 Apple Inc. Multi-touch hand position offset computation
US20090244032A1 (en) * 1998-01-26 2009-10-01 Wayne Westerman Contact Tracking and Identification Module for Touch Sensing
US8665240B2 (en) 1998-01-26 2014-03-04 Apple Inc. Degree of freedom extraction from multiple contacts
US8334846B2 (en) 1998-01-26 2012-12-18 Apple Inc. Multi-touch contact tracking using predicted paths
US8902175B2 (en) 1998-01-26 2014-12-02 Apple Inc. Contact tracking and identification module for touch sensing
US9298310B2 (en) 1998-01-26 2016-03-29 Apple Inc. Touch sensor contact information
US8633898B2 (en) 1998-01-26 2014-01-21 Apple Inc. Sensor arrangement for use with a touch sensor that identifies hand parts
US8629840B2 (en) 1998-01-26 2014-01-14 Apple Inc. Touch sensing architecture
US20070070052A1 (en) * 1998-01-26 2007-03-29 Fingerworks, Inc. Multi-touch contact motion extraction
US8593426B2 (en) 1998-01-26 2013-11-26 Apple Inc. Identifying contacts on a touch surface
US8576177B2 (en) 1998-01-26 2013-11-05 Apple Inc. Typing with a touch sensor
US9001068B2 (en) 1998-01-26 2015-04-07 Apple Inc. Touch sensor contact information
US8514183B2 (en) 1998-01-26 2013-08-20 Apple Inc. Degree of freedom extraction from multiple contacts
US9342180B2 (en) 1998-01-26 2016-05-17 Apple Inc. Contact tracking and identification module for touch sensing
US7782307B2 (en) 1998-01-26 2010-08-24 Apple Inc. Maintaining activity after contact liftoff or touchdown
US8482533B2 (en) 1998-01-26 2013-07-09 Apple Inc. Contact tracking and identification module for touch sensing
US7812828B2 (en) 1998-01-26 2010-10-12 Apple Inc. Ellipse fitting for multi-touch surfaces
US8330727B2 (en) 1998-01-26 2012-12-11 Apple Inc. Generating control signals from multiple contacts
US20070078919A1 (en) * 1998-01-26 2007-04-05 Fingerworks, Inc. Multi-touch hand position offset computation
US8466880B2 (en) 1998-01-26 2013-06-18 Apple Inc. Multi-touch contact motion extraction
US8466881B2 (en) 1998-01-26 2013-06-18 Apple Inc. Contact tracking and identification module for touch sensing
US8466883B2 (en) 1998-01-26 2013-06-18 Apple Inc. Identifying contacts on a touch surface
US9098142B2 (en) 1998-01-26 2015-08-04 Apple Inc. Sensor arrangement for use with a touch sensor that identifies hand parts
US9552100B2 (en) 1998-01-26 2017-01-24 Apple Inc. Touch sensing with mobile sensors
US8441453B2 (en) 1998-01-26 2013-05-14 Apple Inc. Contact tracking and identification module for touch sensing
US9239673B2 (en) 1998-01-26 2016-01-19 Apple Inc. Gesturing with a multipoint sensing device
US20070139395A1 (en) * 1998-01-26 2007-06-21 Fingerworks, Inc. Ellipse Fitting for Multi-Touch Surfaces
US9626032B2 (en) 1998-01-26 2017-04-18 Apple Inc. Sensor arrangement for use with a touch sensor
US8866752B2 (en) 1998-01-26 2014-10-21 Apple Inc. Contact tracking and identification module for touch sensing
US9448658B2 (en) 1998-01-26 2016-09-20 Apple Inc. Resting contacts
US9383855B2 (en) 1998-01-26 2016-07-05 Apple Inc. Identifying contacts on a touch surface
US8730192B2 (en) 1998-01-26 2014-05-20 Apple Inc. Contact tracking and identification module for touch sensing
US8482535B2 (en) 1999-11-08 2013-07-09 Apple Inc. Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US20090273563A1 (en) * 1999-11-08 2009-11-05 Pryor Timothy R Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics
US8576199B1 (en) 2000-02-22 2013-11-05 Apple Inc. Computer control systems
US20080088587A1 (en) * 2001-02-22 2008-04-17 Timothy Pryor Compact rtd instrument panels and computer interfaces
US20080024463A1 (en) * 2001-02-22 2008-01-31 Timothy Pryor Reconfigurable tactile control display applications
US8674966B2 (en) 2001-11-02 2014-03-18 Neonode Inc. ASIC controller for light-based touch screen
US9052777B2 (en) 2001-11-02 2015-06-09 Neonode Inc. Optical elements with alternating reflective lens facets
US9778794B2 (en) 2001-11-02 2017-10-03 Neonode Inc. Light-based touch screen
US9606668B2 (en) 2002-02-07 2017-03-28 Apple Inc. Mode-based graphical user interfaces for touch sensitive input devices
US9983742B2 (en) 2002-07-01 2018-05-29 Apple Inc. Electronic device having display and surrounding touch sensitive bezel for user interface and control
US8416217B1 (en) 2002-11-04 2013-04-09 Neonode Inc. Light-based finger gesture user interface
US9262074B2 (en) 2002-11-04 2016-02-16 Neonode, Inc. Finger gesture user interface
US8884926B1 (en) 2002-11-04 2014-11-11 Neonode Inc. Light-based finger gesture user interface
US8810551B2 (en) 2002-11-04 2014-08-19 Neonode Inc. Finger gesture user interface
US9164654B2 (en) 2002-12-10 2015-10-20 Neonode Inc. User interface for mobile computer unit
US20100017872A1 (en) * 2002-12-10 2010-01-21 Neonode Technologies User interface for mobile computer unit
US10156914B2 (en) 2003-09-02 2018-12-18 Apple Inc. Ambidextrous mouse
US9785258B2 (en) 2003-09-02 2017-10-10 Apple Inc. Ambidextrous mouse
US10474251B2 (en) 2003-09-02 2019-11-12 Apple Inc. Ambidextrous mouse
US20090189878A1 (en) * 2004-04-29 2009-07-30 Neonode Inc. Light-based touch screen
US8339379B2 (en) 2004-04-29 2012-12-25 Neonode Inc. Light-based touch screen
US9035907B2 (en) 2004-05-06 2015-05-19 Apple Inc. Multipoint touchscreen
US11604547B2 (en) 2004-05-06 2023-03-14 Apple Inc. Multipoint touchscreen
US9454277B2 (en) 2004-05-06 2016-09-27 Apple Inc. Multipoint touchscreen
US8416209B2 (en) 2004-05-06 2013-04-09 Apple Inc. Multipoint touchscreen
US10338789B2 (en) 2004-05-06 2019-07-02 Apple Inc. Operation of a computer with touch screen interface
US10331259B2 (en) 2004-05-06 2019-06-25 Apple Inc. Multipoint touchscreen
US10908729B2 (en) 2004-05-06 2021-02-02 Apple Inc. Multipoint touchscreen
US8872785B2 (en) 2004-05-06 2014-10-28 Apple Inc. Multipoint touchscreen
US9239677B2 (en) 2004-05-06 2016-01-19 Apple Inc. Operation of a computer with touch screen interface
US8982087B2 (en) 2004-05-06 2015-03-17 Apple Inc. Multipoint touchscreen
US8605051B2 (en) 2004-05-06 2013-12-10 Apple Inc. Multipoint touchscreen
US8928618B2 (en) 2004-05-06 2015-01-06 Apple Inc. Multipoint touchscreen
US8612856B2 (en) 2004-07-30 2013-12-17 Apple Inc. Proximity detector in handheld device
US8381135B2 (en) 2004-07-30 2013-02-19 Apple Inc. Proximity detector in handheld device
US10042418B2 (en) 2004-07-30 2018-08-07 Apple Inc. Proximity detector in handheld device
US7653883B2 (en) 2004-07-30 2010-01-26 Apple Inc. Proximity detector in handheld device
US11036282B2 (en) 2004-07-30 2021-06-15 Apple Inc. Proximity detector in handheld device
US11360509B2 (en) 2005-03-04 2022-06-14 Apple Inc. Electronic device having display and surrounding touch sensitive surfaces for user interface and control
US10921941B2 (en) 2005-03-04 2021-02-16 Apple Inc. Electronic device having display and surrounding touch sensitive surfaces for user interface and control
US11275405B2 (en) 2005-03-04 2022-03-15 Apple Inc. Multi-functional hand-held device
US7656393B2 (en) 2005-03-04 2010-02-02 Apple Inc. Electronic device having display and surrounding touch sensitive bezel for user interface and control
US9047009B2 (en) 2005-03-04 2015-06-02 Apple Inc. Electronic device having display and surrounding touch sensitive bezel for user interface and control
US10386980B2 (en) 2005-03-04 2019-08-20 Apple Inc. Electronic device having display and surrounding touch sensitive surfaces for user interface and control
US20060267805A1 (en) * 2005-05-30 2006-11-30 Samsung Electronics Co., Ltd. Method and system for data input
US20070177804A1 (en) * 2006-01-30 2007-08-02 Apple Computer, Inc. Multi-touch gesture dictionary
US7920131B2 (en) 2006-04-25 2011-04-05 Apple Inc. Keystroke tactility arrangement on a smooth touch surface
US8062115B2 (en) 2006-04-27 2011-11-22 Wms Gaming Inc. Wagering game with multi-point gesture sensing device
US9547394B2 (en) 2006-05-02 2017-01-17 Apple Inc. Multipoint touch surface controller
US20090315850A1 (en) * 2006-05-02 2009-12-24 Steven Porter Hotelling Multipoint Touch Surface Controller
US10915207B2 (en) 2006-05-02 2021-02-09 Apple Inc. Multipoint touch surface controller
US11853518B2 (en) 2006-05-02 2023-12-26 Apple Inc. Multipoint touch surface controller
US8816984B2 (en) 2006-05-02 2014-08-26 Apple Inc. Multipoint touch surface controller
US9262029B2 (en) 2006-05-02 2016-02-16 Apple Inc. Multipoint touch surface controller
US9268429B2 (en) 2006-06-09 2016-02-23 Apple Inc. Integrated display and touch screen
US11886651B2 (en) 2006-06-09 2024-01-30 Apple Inc. Touch screen liquid crystal display
US9575610B2 (en) 2006-06-09 2017-02-21 Apple Inc. Touch screen liquid crystal display
US8432371B2 (en) 2006-06-09 2013-04-30 Apple Inc. Touch screen liquid crystal display
US8451244B2 (en) 2006-06-09 2013-05-28 Apple Inc. Segmented Vcom
US11175762B2 (en) 2006-06-09 2021-11-16 Apple Inc. Touch screen liquid crystal display
US8552989B2 (en) 2006-06-09 2013-10-08 Apple Inc. Integrated display and touch screen
US9244561B2 (en) 2006-06-09 2016-01-26 Apple Inc. Touch screen liquid crystal display
US10191576B2 (en) 2006-06-09 2019-01-29 Apple Inc. Touch screen liquid crystal display
US8654083B2 (en) 2006-06-09 2014-02-18 Apple Inc. Touch screen liquid crystal display
US10976846B2 (en) 2006-06-09 2021-04-13 Apple Inc. Touch screen liquid crystal display
US8147316B2 (en) 2006-10-10 2012-04-03 Wms Gaming, Inc. Multi-player, multi-touch table for use in wagering game systems
US8348747B2 (en) 2006-10-10 2013-01-08 Wms Gaming Inc. Multi-player, multi-touch table for use in wagering game systems
US20100130280A1 (en) * 2006-10-10 2010-05-27 Wms Gaming, Inc. Multi-player, multi-touch table for use in wagering game systems
US8926421B2 (en) 2006-10-10 2015-01-06 Wms Gaming Inc. Multi-player, multi-touch table for use in wagering game systems
US9323405B2 (en) 2007-01-03 2016-04-26 Apple Inc. Front-end signal compensation
US9311528B2 (en) 2007-01-03 2016-04-12 Apple Inc. Gesture learning
US20080163130A1 (en) * 2007-01-03 2008-07-03 Apple Inc Gesture learning
US7848825B2 (en) 2007-01-03 2010-12-07 Apple Inc. Master/slave mode for sensor processing devices
US8405617B2 (en) 2007-01-03 2013-03-26 Apple Inc. Gated power management over a system bus
US20080158177A1 (en) * 2007-01-03 2008-07-03 Apple Inc. Master/slave mode for sensor processing devices
US20080158178A1 (en) * 2007-01-03 2008-07-03 Apple Inc. Front-end signal compensation
US8553004B2 (en) 2007-01-03 2013-10-08 Apple Inc. Front-end signal compensation
US10254890B2 (en) 2007-01-03 2019-04-09 Apple Inc. Front-end signal compensation
US11353989B2 (en) 2007-01-03 2022-06-07 Apple Inc. Front-end signal compensation
US8049732B2 (en) 2007-01-03 2011-11-01 Apple Inc. Front-end signal compensation
US8493330B2 (en) 2007-01-03 2013-07-23 Apple Inc. Individual channel phase delay scheme
US8711129B2 (en) 2007-01-03 2014-04-29 Apple Inc. Minimizing mismatch during compensation
US20080162967A1 (en) * 2007-01-03 2008-07-03 Apple Computer, Inc. Gated power management over a system bus
US20080158175A1 (en) * 2007-01-03 2008-07-03 Apple Inc. Minimizing mismatch during compensation
US10725587B2 (en) 2007-01-03 2020-07-28 Apple Inc. Front-end signal compensation
US9710095B2 (en) 2007-01-05 2017-07-18 Apple Inc. Touch screen stack-ups
US10521065B2 (en) 2007-01-05 2019-12-31 Apple Inc. Touch screen stack-ups
US9448712B2 (en) 2007-01-07 2016-09-20 Apple Inc. Application programming interfaces for scrolling operations
US10817162B2 (en) 2007-01-07 2020-10-27 Apple Inc. Application programming interfaces for scrolling operations
US10437459B2 (en) 2007-01-07 2019-10-08 Apple Inc. Multitouch data fusion
US11481109B2 (en) 2007-01-07 2022-10-25 Apple Inc. Multitouch data fusion
US20080211766A1 (en) * 2007-01-07 2008-09-04 Apple Inc. Multitouch data fusion
US11816329B2 (en) 2007-01-07 2023-11-14 Apple Inc. Multitouch data fusion
US10481785B2 (en) 2007-01-07 2019-11-19 Apple Inc. Application programming interfaces for scrolling operations
US9760272B2 (en) 2007-01-07 2017-09-12 Apple Inc. Application programming interfaces for scrolling operations
US8054299B2 (en) 2007-01-08 2011-11-08 Apple Inc. Digital controller for a true multi-point touch surface useable in a computer system
US20080165134A1 (en) * 2007-01-08 2008-07-10 Apple Computer, Inc. Digital Controller for a True Multi-point Touch Surface Useable in a Computer System
US20110043485A1 (en) * 2007-07-06 2011-02-24 Neonode Inc. Scanning of a touch screen
US8471830B2 (en) 2007-07-06 2013-06-25 Neonode Inc. Scanning of a touch screen
US9710103B2 (en) 2007-10-01 2017-07-18 Igt Method and apparatus for detecting lift off of a touchscreen
US8427447B2 (en) 2007-10-01 2013-04-23 Igt Multi-user input systems and processing techniques for serving multiple users
US20090195518A1 (en) * 2007-10-01 2009-08-06 Igt Method and apparatus for detecting lift off on a touchscreen
US9335869B2 (en) * 2007-10-01 2016-05-10 Igt Method and apparatus for detecting lift off on a touchscreen
US9652074B2 (en) 2007-10-01 2017-05-16 Igt Method and apparatus for detecting lift off of a touchscreen
US20090178011A1 (en) * 2008-01-04 2009-07-09 Bas Ording Gesture movies
US8413075B2 (en) * 2008-01-04 2013-04-02 Apple Inc. Gesture movies
US7593000B1 (en) 2008-05-17 2009-09-22 David H. Chin Touch-based authentication of a mobile device through user generated pattern creation
US20090284482A1 (en) * 2008-05-17 2009-11-19 Chin David H Touch-based authentication of a mobile device through user generated pattern creation
US8174503B2 (en) 2008-05-17 2012-05-08 David H. Cain Touch-based authentication of a mobile device through user generated pattern creation
US20090325691A1 (en) * 2008-06-26 2009-12-31 Loose Timothy C Gaming machine having multi-touch sensing device
US8241912B2 (en) 2008-06-26 2012-08-14 Wms Gaming Inc. Gaming machine having multi-touch sensing device
US8654941B2 (en) 2009-01-27 2014-02-18 Lifesize Communications, Inc. Using a touch interface to control a videoconference
US9211105B2 (en) 2009-01-28 2015-12-15 Samsung Medison Co., Ltd. Image indicator provision in an ultrasound system
US9389710B2 (en) 2009-02-15 2016-07-12 Neonode Inc. Light-based controls on a toroidal steering wheel
US9213443B2 (en) 2009-02-15 2015-12-15 Neonode Inc. Optical touch screen systems using reflected light
US8775023B2 (en) 2009-02-15 2014-07-08 Neanode Inc. Light-based touch controls on a steering wheel and dashboard
US8918252B2 (en) 2009-02-15 2014-12-23 Neonode Inc. Light-based touch controls on a steering wheel
US20100241984A1 (en) * 2009-03-21 2010-09-23 Nokia Corporation Method and apparatus for displaying the non alphanumeric character based on a user input
US10248221B2 (en) 2009-08-17 2019-04-02 Apple Inc. Housing as an I/O device
US8654524B2 (en) 2009-08-17 2014-02-18 Apple Inc. Housing as an I/O device
US10739868B2 (en) 2009-08-17 2020-08-11 Apple Inc. Housing as an I/O device
US11644865B2 (en) 2009-08-17 2023-05-09 Apple Inc. Housing as an I/O device
US9600037B2 (en) 2009-08-17 2017-03-21 Apple Inc. Housing as an I/O device
US20110038114A1 (en) * 2009-08-17 2011-02-17 Apple Inc. Housing as an i/o device
US20110190061A1 (en) * 2010-02-03 2011-08-04 Nintendo Co., Ltd. Display device, game system, and game method
US9358457B2 (en) 2010-02-03 2016-06-07 Nintendo Co., Ltd. Game system, controller device, and game method
US8317615B2 (en) 2010-02-03 2012-11-27 Nintendo Co., Ltd. Display device, game system, and game method
US8339364B2 (en) 2010-02-03 2012-12-25 Nintendo Co., Ltd. Spatially-correlated multi-display human-machine interface
US8814686B2 (en) 2010-02-03 2014-08-26 Nintendo Co., Ltd. Display device, game system, and game method
US8913009B2 (en) 2010-02-03 2014-12-16 Nintendo Co., Ltd. Spatially-correlated multi-display human-machine interface
US9776083B2 (en) 2010-02-03 2017-10-03 Nintendo Co., Ltd. Spatially-correlated multi-display human-machine interface
US8961305B2 (en) 2010-02-03 2015-02-24 Nintendo Co., Ltd. Game system, controller device and game method
US8896534B2 (en) 2010-02-03 2014-11-25 Nintendo Co., Ltd. Spatially-correlated multi-display human-machine interface
US8684842B2 (en) 2010-02-03 2014-04-01 Nintendo Co., Ltd. Display device, game system, and game process method
US8471814B2 (en) 2010-02-26 2013-06-25 Microsoft Corporation User interface control using a keyboard
US9292194B2 (en) 2010-02-26 2016-03-22 Microsoft Technology Licensing, LC User interface control using a keyboard
US9851809B2 (en) 2010-02-26 2017-12-26 Microsoft Technology Licensing, Llc User interface control using a keyboard
US20110225524A1 (en) * 2010-03-10 2011-09-15 Cifra Christopher G Multi-Touch Editing in a Graphical Programming Language
US9199168B2 (en) 2010-08-06 2015-12-01 Nintendo Co., Ltd. Game system, game apparatus, storage medium having game program stored therein, and game process method
US10150033B2 (en) 2010-08-20 2018-12-11 Nintendo Co., Ltd. Position calculation system, position calculation device, storage medium storing position calculation program, and position calculation method
US8956209B2 (en) 2010-08-30 2015-02-17 Nintendo Co., Ltd. Game system, game apparatus, storage medium having game program stored therein, and game process method
US9132347B2 (en) 2010-08-30 2015-09-15 Nintendo Co., Ltd. Game system, game apparatus, storage medium having game program stored therein, and game process method
US8804326B2 (en) 2010-11-01 2014-08-12 Nintendo Co., Ltd. Device support system and support device
US9889384B2 (en) 2010-11-01 2018-02-13 Nintendo Co., Ltd. Controller device and controller system
US8827818B2 (en) 2010-11-01 2014-09-09 Nintendo Co., Ltd. Controller device and information processing device
US8702514B2 (en) 2010-11-01 2014-04-22 Nintendo Co., Ltd. Controller device and controller system
US9272207B2 (en) 2010-11-01 2016-03-01 Nintendo Co., Ltd. Controller device and controller system
US8814680B2 (en) 2010-11-01 2014-08-26 Nintendo Co., Inc. Controller device and controller system
US9870093B2 (en) 2010-11-23 2018-01-16 Ge Aviation Systems Llc System and method for improving touch screen display use under vibration and turbulence
US20150370378A1 (en) * 2010-12-22 2015-12-24 Apple Inc. Integrated touch screens
US8804056B2 (en) 2010-12-22 2014-08-12 Apple Inc. Integrated touch screens
US10409434B2 (en) * 2010-12-22 2019-09-10 Apple Inc. Integrated touch screens
US9025090B2 (en) 2010-12-22 2015-05-05 Apple Inc. Integrated touch screens
US9146414B2 (en) 2010-12-22 2015-09-29 Apple Inc. Integrated touch screens
US8743300B2 (en) 2010-12-22 2014-06-03 Apple Inc. Integrated touch screens
US9727193B2 (en) * 2010-12-22 2017-08-08 Apple Inc. Integrated touch screens
US20120229308A1 (en) * 2011-03-10 2012-09-13 Chun-Liang Tsai Low power wireless keyboard
US8845426B2 (en) 2011-04-07 2014-09-30 Nintendo Co., Ltd. Input system, information processing device, storage medium storing information processing program, and three-dimensional position calculation method
US8959459B2 (en) 2011-06-15 2015-02-17 Wms Gaming Inc. Gesture sensing enhancement system for a wagering game
US9204094B2 (en) 2011-06-28 2015-12-01 Lifesize Communications, Inc. Adjusting volume of a videoconference using touch-based gestures
US8605872B2 (en) 2011-06-28 2013-12-10 Lifesize Communications, Inc. Muting a videoconference using touch-based gestures
US8605873B2 (en) 2011-06-28 2013-12-10 Lifesize Communications, Inc. Accessing settings of a videoconference using touch-based gestures
US9047007B2 (en) 2011-07-28 2015-06-02 National Instruments Corporation Semantic zoom within a diagram of a system
US8713482B2 (en) 2011-07-28 2014-04-29 National Instruments Corporation Gestures for presentation of different views of a system diagram
US8782525B2 (en) 2011-07-28 2014-07-15 National Insturments Corporation Displaying physical signal routing in a diagram of a system
US8971572B1 (en) 2011-08-12 2015-03-03 The Research Foundation For The State University Of New York Hand pointing estimation for human computer interaction
US9372546B2 (en) 2011-08-12 2016-06-21 The Research Foundation For The State University Of New York Hand pointing estimation for human computer interaction
US20130120293A1 (en) * 2011-11-14 2013-05-16 Samsung Electronics Co., Ltd. Touchscreen-enabled terminal and application control method thereof
US9524097B2 (en) * 2011-12-22 2016-12-20 International Business Machines Corporation Touchscreen gestures for selecting a graphical object
US9778841B2 (en) 2012-02-10 2017-10-03 Hand Held Products, Inc. Apparatus having random ordered keypad
US9086732B2 (en) 2012-05-03 2015-07-21 Wms Gaming Inc. Gesture fusion
US9710144B2 (en) 2012-11-27 2017-07-18 Neonode Inc. User interface for curved input device
US9092093B2 (en) 2012-11-27 2015-07-28 Neonode Inc. Steering wheel user interface
US9576422B2 (en) 2013-04-18 2017-02-21 Bally Gaming, Inc. Systems, methods, and devices for operating wagering game machines with enhanced user interfaces
US20170111297A1 (en) * 2015-10-20 2017-04-20 Line Corporation Display control method, terminal, and information processing apparatus
US10530717B2 (en) 2015-10-20 2020-01-07 Line Corporation Display control method, information processing apparatus, and terminal

Also Published As

Publication number Publication date
US7705830B2 (en) 2010-04-27
US20060125803A1 (en) 2006-06-15
US7030861B1 (en) 2006-04-18

Similar Documents

Publication Publication Date Title
USRE40153E1 (en) Multi-touch system and method for emulating modifier keys via fingertip chords
US6570557B1 (en) Multi-touch system and method for emulating modifier keys via fingertip chords
US20160124926A1 (en) Advanced methods and systems for text input error correction
US5635958A (en) Information inputting and processing apparatus
US7098896B2 (en) System and method for continuous stroke word-based text input
US10126941B2 (en) Multi-touch text input
EP1522007B1 (en) Automatically adaptable virtual keyboard
US10061510B2 (en) Gesture multi-function on a physical keyboard
US8896555B2 (en) Touch alphabet and communication system
EP2370878B1 (en) Gesture detection zones
CN103186339B (en) A kind of input method and be provided with the electronic equipment of dummy keyboard
NZ519928A (en) Keyboard system with automatic correction
JP2011530937A (en) Data entry system
EP0861485A1 (en) Multi-touch input device, method and system that minimize the need for memorization
CN104903836A (en) Method and device for typing on mobile computing devices
WO2013064915A1 (en) Single hand multi-touch surface keyboard
US11112965B2 (en) Advanced methods and systems for text input error correction
JPH06242885A (en) Document editing method
JPH10275046A (en) Device and method for word input
US20120086638A1 (en) Multi-area handwriting input system and method thereof
CN104503591A (en) Information input method based on broken line gesture
KR100477125B1 (en) Character input device using electronic pen and its control method
JP2014204137A (en) Input system of touch type display and display method for input panel
US20170024053A1 (en) Touch alphabet and communication system
JP5519546B2 (en) Handwritten character input device

Legal Events

Date Code Title Description
AS Assignment

Owner name: APPLE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:FINGERWORKS, INC.;REEL/FRAME:019849/0382

Effective date: 20070831

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 8

FPAY Fee payment

Year of fee payment: 12