US20030231219A1 - System and method for recognizing connector gestures - Google Patents

System and method for recognizing connector gestures Download PDF

Info

Publication number
US20030231219A1
US20030231219A1 US10/171,099 US17109902A US2003231219A1 US 20030231219 A1 US20030231219 A1 US 20030231219A1 US 17109902 A US17109902 A US 17109902A US 2003231219 A1 US2003231219 A1 US 2003231219A1
Authority
US
United States
Prior art keywords
connector
computer
freehand
touch
generated
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US10/171,099
Other versions
US7330184B2 (en
Inventor
Andy Leung
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Smart Technologies ULC
Original Assignee
Smart Technologies ULC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Smart Technologies ULC filed Critical Smart Technologies ULC
Priority to US10/171,099 priority Critical patent/US7330184B2/en
Priority to CA2390506A priority patent/CA2390506C/en
Assigned to SMART TECHNOLOGIES INC. reassignment SMART TECHNOLOGIES INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LEUNG, ANDY
Publication of US20030231219A1 publication Critical patent/US20030231219A1/en
Assigned to SMART TECHNOLOGIES ULC reassignment SMART TECHNOLOGIES ULC MERGER (SEE DOCUMENT FOR DETAILS). Assignors: SMART TECHNOLOGIES INC.
Publication of US7330184B2 publication Critical patent/US7330184B2/en
Application granted granted Critical
Assigned to MORGAN STANLEY SENIOR FUNDING INC. reassignment MORGAN STANLEY SENIOR FUNDING INC. SECURITY AGREEMENT Assignors: SMART TECHNOLOGIES INC., SMART TECHNOLOGIES ULC
Assigned to MORGAN STANLEY SENIOR FUNDING, INC. reassignment MORGAN STANLEY SENIOR FUNDING, INC. SECURITY AGREEMENT Assignors: SMART TECHNOLOGIES INC., SMART TECHNOLOGIES ULC
Assigned to SMART TECHNOLOGIES INC., SMART TECHNOLOGIES ULC reassignment SMART TECHNOLOGIES INC. RELEASE OF ABL SECURITY INTEREST Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to SMART TECHNOLOGIES INC., SMART TECHNOLOGIES ULC reassignment SMART TECHNOLOGIES INC. RELEASE OF TERM LOAN SECURITY INTEREST Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to SMART TECHNOLOGIES INC., SMART TECHNOLOGIES ULC reassignment SMART TECHNOLOGIES INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to SMART TECHNOLOGIES ULC, SMART TECHNOLOGIES INC. reassignment SMART TECHNOLOGIES ULC RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Adjusted expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text

Definitions

  • touch systems are well known in the art and many variations exist.
  • touch systems include a touch panel having a touch surface on which contacts are made using a pointer. Pointer contacts with the touch surface are detected and are used to generate corresponding output that represent the positions on the touch surface where contacts are made.
  • the contact position output is typically fed to a computer that executes one or more applications programs.
  • the computer generates image data that is used to present images on the touch surface.
  • the computer uses the contact position output to update the image data and thus, the images presented on the touch surface. In this manner, the images presented on the touch surface are updated to reflect the activity of the pointer on the touch surface.
  • U.S. Pat. No. 5,448,263 to Martin discloses a passive analog resistive touch panel coupled to a computer.
  • the computer provides image data to a projector that projects images onto the touch surface of the touch panel.
  • the touch panel includes a tool tray that supports a plurality of differently coloured pens.
  • the touch panel outputs signals representing the contact position on the touch surface.
  • the contact position data is conveyed to the computer and is mapped to the computer display. If a finger or other pointer is used to contact the touch surface, the touch system operates in a pointer mode and the contact position data is treated as a mouse event.
  • a computer desktop image is presented on the touch surface that includes icons representing the various applications programs available for selection.
  • icons representing the various applications programs available for selection.
  • a window for the selected applications program is opened.
  • the window typically includes a frame, one or more tool bars, optional scroll bars and an active area surrounded by the frame, tool bars and scroll bars.
  • contacts on the touch surface are treated as mouse event input to the computer desktop.
  • the computer in response to the mouse event input controls the computer desktop or selected applications program according to the touch panel output and updates the image data conveyed to the projector for display to reflect the pointer activity.
  • two-dimensional diagrams In the ink mode, users often draw two-dimensional diagrams on the touch surface such as flowcharts, schematics, process maps etc. in addition to writing text. These two-dimensional diagrams typically include a plurality of graphical objects such as rectangles, squares, diamonds, ovals and circles interconnected by straight, curved or serpentine lines. Generally, two-dimensional diagrams drawn by freehand are unclear. As a result software has been developed to assist users in the creation of two-dimensional diagrams.
  • the method further comprises the step of determining whether the start and end points of the freehand connector are proximate to graphical objects and if so, extending the computer-generated connector between the graphical objects. It is also preferred that the method further comprises the step of generating new graphical objects at the start and end points of the computer-generated connector if the start and end points of the freehand connector are not proximate to graphical objects.
  • a system for recognizing a freehand connector input into a computer comprising:
  • a touch system comprising:
  • a touch panel having a touch surface
  • a computer executing an applications program and being coupled to said touch panel and said projector, said computer being responsive to contact output generated by said touch panel in response to proximity of a pointer to said touch surface and updating image data conveyed to said projector so that images presented on said touch surface reflect pointer activity, said computer executing a connector gesture recognition routine, said connector gesture recognition routine performing recognition to convert a freehand connector on said touch surface into a computer-generated connector approximating said freehand connector.
  • a computer program product including a computer readable medium having a computer program for recognizing a freehand connector input into a computer embodied thereon, said computer program including:
  • the present invention provides advantages in that freehand connectors are automatically recognized and replaced with computer-generated connectors. As a result user input during creation of diagrams is reduced. The freehand connector simply needs to be drawn by the user.
  • FIG. 1 is a schematic diagram of an interactive display system including a touch panel having a touch surface
  • FIG. 2 is a view of a computer desktop image together with an acetate image on the touch surface of the touch panel;
  • FIGS. 3 a to 3 c are flowcharts showing the steps performed by a connector gesture recognition routine.
  • FIGS. 4 to 10 are views showing freehand connectors drawn on the touch surface and the connectors as recognized by the connector gesture recognition routine.
  • touch system 10 includes a touch panel 12 having a touch surface 14 .
  • the touch panel 12 in this embodiment is of the analog resistive type and includes a tool tray 15 that supports a plurality of differently coloured pens (not shown).
  • Touch panel 12 is responsive to contacts on the touch surface 14 made using either a finger, other pointer or pen and generates output representative of the locations on the touch surface 14 where contacts are made.
  • the computer 16 runs in a Windows environment and provides image data to the projector 18 so that a computer desktop image is presented on the touch surface 14 .
  • the computer desktop image presents one or more icons that can be selected to open associated applications programs.
  • an applications program is selected, a window for the applications program is opened.
  • FIG. 2 shows an applications program window 30 including an active area 32 bordered by a frame and one or more tool bars 34 is shown presented on the touch surface 14 .
  • an acetate image 40 identical to the computer desktop image is displayed over the computer desktop image as shown FIG. 2.
  • the acetate image 40 provides a surface for writing (commonly referred to as “ink”) so that the ink is not used by the computer 16 to update the underlying computer desktop image 30 .
  • Contact position data returned to the computer 16 by the touch panel 12 in response to pen contacts with the touch surface 14 are used by the computer to update the image data conveyed to the projector 18 so that the ink appears on the acetate image 40 .
  • the computer 16 can be conditioned to execute a connector gesture recognition routine to interpret freehand connectors drawn between graphical objects, if the user is using the pen to draw a two-dimensional diagram, by selecting an icon presented on the touch surface 14 .
  • a connector gesture recognition routine to interpret freehand connectors drawn between graphical objects, if the user is using the pen to draw a two-dimensional diagram, by selecting an icon presented on the touch surface 14 .
  • the connector gesture recognition routine examines the start point and end point of the freehand line to determine if the start point and end point of the freehand line touch graphical objects displayed on the touch surface that can be connected (step 102 ). If the start point and end point of the freehand line do not touch graphical objects that can be connected, the freehand line is maintained in its original form (step 104 ). If the start point and end point of the freehand line touch graphical objects that can be connected, a connector gesture determination is made to determine the manner by which the freehand line is to be represented (step 106 ). The freehand line is then replaced by the determined computer-generated connector and the image data conveyed to the projector 18 is updated accordingly so that the computer-generated connector approximating the freehand line is presented on the touch surface 14 (step 108 ).
  • step 102 when the start point and end point of the freehand line are examined, initially the start point is examined to determine if it touches a graphical object (step 120 ). If the start point of the freehand line does not touch a graphical object, the user is presented with a prompt to determine if a new graphical object is to be created at the start point of the freehand line (step 122 ). If the user does not wish a new graphical object to be created, the connector gesture recognition routine proceeds to step 104 .
  • the connector gesture recognition routine examines the end point of the freehand line to determine if it touches a graphical object (step 124 ). If not, the connector gesture recognition routine proceeds to step 122 to determine if the user wishes to create a new graphical object at the end of the freehand line. If the end point of the freehand line touches a graphical object or if the user conditions the connector gesture recognition routine to create a new graphical object at the end of the freehand line, the connector gesture recognition routine proceeds to step 106 to determine how to represent the freehand line. Otherwise, the connector gesture recognition routine proceeds to step 104 .
  • the connector gesture recognition routine sets the start and end points of the freehand line (step 130 ) and analyzes points along the freehand line in order to select an appropriate representation (step 132 ). Based on the analysis, if the freehand line is not a straight line and is determined to have an elbow along its length (step 134 ), the connector gesture recognition routine recognizes the freehand line as a right-angled line (step 136 ). If the line has a winding path (step 138 ), the connector gesture recognition routine recognizes the freehand line either as a polygonal, splined or circular line depending on the shape of the freehand line (step 140 ).
  • the connector gesture recognition routine recognizes the freehand line as a curved line (step 146 ). The curve is fitted using an apogee as the control knot (i.e. the furthest point out away from the graphical objects). If the freehand line is straight (step 148 ), the connector gesture recognition routine recognizes the freehand line as a straight line (step 150 ). Once the freehand line has been recognized, the connector gesture recognition routine replaces the freehand line with a computer-generated equivalent and updates the image data conveyed to the projector 18 so that the appropriate computer-generated connector is displayed on the touch surface 14 .
  • FIGS. 4 to 10 examples of connector gesture recognitions are shown.
  • a freehand line 200 interconnecting two graphical objects 202 and 204 is recognized as a straight line 206 .
  • a freehand line 210 that is not straight and that has elbows is recognized as a right-angled line 216 .
  • a non-straight freehand line 220 is recognized as a curved line 226 and in FIG. 7, a winding line 230 is recognized as a circular line 236 .
  • FIG. 8 shows a freehand straight line 240 having a start point adjacent a graphical object 242 . The end point of the freehand line 240 does not touch a graphical object.
  • the user at step 122 has requested the connector gesture recognition routine to create a new graphical object 244 at the end of the freehand line and the freehand line is recognized as a straight line 246 .
  • FIG. 9 shows relationships between graphical objects 252 and 254 in an object oriented program. As shown, three separate curved computer-generated connectors 256 a to 256 c extend between the graphical objects. The computer-generated connectors are created in manner that has been described above. It will be appreciated that any number of connectors may extend between pairs of graphical objects.
  • FIG. 10 shows a computer-generated winding line 266 extending between a pair of graphical objects 262 and 264 .
  • touch systems also include passive surface acoustic wave or capacitive-type touch systems as well as camera-based touch systems such as that disclosed in International PCT Application No. WO 02/03316 to Smart Technologies Inc. et al., the assignee of the present invention.
  • gesture connector recognition routine may be run on basically any computing device where freehand connectors joining graphical objects are entered.
  • a mouse, trackball, touch pad, or other pointing device may be used to create the freehand connectors.
  • the freehand connectors need not touch graphical objects in order for the connector gesture recognition routine to extend computer-generated connectors from them. Rather, the freehand connectors simply need to be within a threshold distance of graphical objects.
  • prompting the user to determine whether new graphical objects are to be created at the start and end points of freehand lines is optional. This feature may be disabled or designated as a default. When designated as a default, the connector gesture recognition routine creates new graphical objects at the start and end points of freehand lines automatically without requiring user input.

Abstract

A touch system includes a touch panel having a touch surface and a projector presenting images onto the touch surface. A computer executing an applications program is coupled to the touch panel and the projector. The computer is responsive to contact output generated by the touch panel in response to proximity of a pointer to the touch surface and updates image data conveyed to the projector so that images presented on the touch surface reflect pointer activity. The computer executes a gesture connector recognition routine. The gesture connector recognition routine performs recognition to convert a freehand connector on the touch surface into a computer-generated connector approximating the freehand connector.

Description

    FIELD OF THE INVENTION
  • The present invention relates generally to touch systems and in particular to a system and method for recognizing connector gestures. [0001]
  • BACKGROUND OF THE INVENTION
  • Touch systems are well known in the art and many variations exist. In all cases, touch systems include a touch panel having a touch surface on which contacts are made using a pointer. Pointer contacts with the touch surface are detected and are used to generate corresponding output that represent the positions on the touch surface where contacts are made. The contact position output is typically fed to a computer that executes one or more applications programs. The computer generates image data that is used to present images on the touch surface. The computer uses the contact position output to update the image data and thus, the images presented on the touch surface. In this manner, the images presented on the touch surface are updated to reflect the activity of the pointer on the touch surface. [0002]
  • For example, U.S. Pat. No. 5,448,263 to Martin, assigned to the assignee of the present invention, discloses a passive analog resistive touch panel coupled to a computer. The computer provides image data to a projector that projects images onto the touch surface of the touch panel. The touch panel includes a tool tray that supports a plurality of differently coloured pens. When a user contacts the touch surface either with a finger, other pointer or a pen, the touch panel outputs signals representing the contact position on the touch surface. The contact position data is conveyed to the computer and is mapped to the computer display. If a finger or other pointer is used to contact the touch surface, the touch system operates in a pointer mode and the contact position data is treated as a mouse event. This allows the user to operate the computer in a manner similar to using a computer mouse i.e. select menus, manipulate objects etc. simply by contacting the touch surface. If a pen is lifted from the tool tray and is used to contact the touch surface, the touch system operates in an ink mode and the contact position data is recorded as writing or drawing. [0003]
  • When the computer is running an applications program in a Windows environment, a computer desktop image is presented on the touch surface that includes icons representing the various applications programs available for selection. When an icon is selected, a window for the selected applications program is opened. The window typically includes a frame, one or more tool bars, optional scroll bars and an active area surrounded by the frame, tool bars and scroll bars. As mentioned above, in the pointer mode, contacts on the touch surface are treated as mouse event input to the computer desktop. The computer in response to the mouse event input controls the computer desktop or selected applications program according to the touch panel output and updates the image data conveyed to the projector for display to reflect the pointer activity. [0004]
  • In the ink mode, an acetate image identical to the computer desktop image overlies the computer desktop image to provide a surface on which ink can be drawn. When a pen contacts the touch surface, the contact position data is treated as writing or drawing (herein referred to as “writing”). In this case, the computer updates the image data conveyed to the projector for display so that the writing is displayed on the acetate image. [0005]
  • In the ink mode, users often draw two-dimensional diagrams on the touch surface such as flowcharts, schematics, process maps etc. in addition to writing text. These two-dimensional diagrams typically include a plurality of graphical objects such as rectangles, squares, diamonds, ovals and circles interconnected by straight, curved or serpentine lines. Generally, two-dimensional diagrams drawn by freehand are unclear. As a result software has been developed to assist users in the creation of two-dimensional diagrams. [0006]
  • For example, computer-aided design (CAD) software programs are available to assist users in the creation of two-dimensional diagrams. One common CAD software program is sold by Autodesk Inc. under the name “AutoCAD”. During use of this CAD software, a user creates a two-dimensional diagram either by placing an existing graphical object on a pallet that is taken from a collection or library of such graphical objects, or by creating a new graphical object. Interconnecting lines or connections, are then manually drawn from a point on or near the graphical object to the appropriate destinations. Every point along the path of the connection must be specified by the user. This is a very tedious process, and must be repeated every time the position, rotational orientation, size or other parameter of any graphical object is changed. [0007]
  • Software is also widely available that automates the manipulation and interconnection of graphical objects so that when a user changes the position or other parameter of a graphical object, all connections and/or graphical objects associated with that graphical object reconfigure themselves to maintain that association. Such software is available from Visio Corp. under the name “Visio Technical”. Complex connections of this nature can be created between graphical objects by simply picking start and end points for the connections. [0008]
  • Complex connections allow a user to edit a diagram more easily. However, the creation of complex connections requires several steps. Typically, the user must first select the graphical object where the complex connection is to originate and then select the graphical object where the complex connection is to terminate. Often, the user is required to specify the shape of the complex connection path between the two graphical objects. It is also common for the user to be required to specify the exact locations of the start and end points of the complex connection. As will be appreciated, alternative systems to enhance interpretation of connectors are desired. [0009]
  • It is therefore an object of the present invention to provide a novel system and method for recognizing connector gestures. [0010]
  • SUMMARY OF THE INVENTION
  • According to one aspect of the present invention there is provided a method of recognizing a freehand connector input into a computer comprising the steps of: [0011]
  • examining points along said freehand connector to determine the shape thereof, and [0012]
  • replacing said freehand connector with a computer-generated connector approximating said freehand connector. [0013]
  • Preferably, the method further comprises the step of determining whether the start and end points of the freehand connector are proximate to graphical objects and if so, extending the computer-generated connector between the graphical objects. It is also preferred that the method further comprises the step of generating new graphical objects at the start and end points of the computer-generated connector if the start and end points of the freehand connector are not proximate to graphical objects. [0014]
  • According to another aspect of the present invention there is provided a system for recognizing a freehand connector input into a computer comprising: [0015]
  • means for examining points along said freehand connector to determine the shape thereof; and [0016]
  • means for replacing said freehand connector with a computer-generated connector approximating said freehand connector. [0017]
  • According to yet another aspect of the present invention there is provided a touch system comprising: [0018]
  • a touch panel having a touch surface; [0019]
  • a projector presenting images onto said touch surface; and [0020]
  • a computer executing an applications program and being coupled to said touch panel and said projector, said computer being responsive to contact output generated by said touch panel in response to proximity of a pointer to said touch surface and updating image data conveyed to said projector so that images presented on said touch surface reflect pointer activity, said computer executing a connector gesture recognition routine, said connector gesture recognition routine performing recognition to convert a freehand connector on said touch surface into a computer-generated connector approximating said freehand connector. [0021]
  • According to still yet another aspect of the present invention there is provided a computer program product including a computer readable medium having a computer program for recognizing a freehand connector input into a computer embodied thereon, said computer program including: [0022]
  • computer program code for examining points along said freehand connector to determine the shape thereof; and [0023]
  • computer program code for replacing said freehand connector with a computer generated connector approximating said freehand connector. [0024]
  • The present invention provides advantages in that freehand connectors are automatically recognized and replaced with computer-generated connectors. As a result user input during creation of diagrams is reduced. The freehand connector simply needs to be drawn by the user.[0025]
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • Embodiments of the present invention will now be described more fully with reference to the accompanying drawings in which: [0026]
  • FIG. 1 is a schematic diagram of an interactive display system including a touch panel having a touch surface; [0027]
  • FIG. 2 is a view of a computer desktop image together with an acetate image on the touch surface of the touch panel; [0028]
  • FIGS. 3[0029] a to 3 c are flowcharts showing the steps performed by a connector gesture recognition routine; and
  • FIGS. [0030] 4 to 10 are views showing freehand connectors drawn on the touch surface and the connectors as recognized by the connector gesture recognition routine.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
  • Turning now to FIG. 1, an interactive touch system similar to that disclosed in U.S. Pat. No. 5,448,263 to Martin is shown and is generally identified by [0031] reference numeral 10. As can be seen, touch system 10 includes a touch panel 12 having a touch surface 14. The touch panel 12 in this embodiment is of the analog resistive type and includes a tool tray 15 that supports a plurality of differently coloured pens (not shown). Touch panel 12 is responsive to contacts on the touch surface 14 made using either a finger, other pointer or pen and generates output representative of the locations on the touch surface 14 where contacts are made. The contact position output of the touch panel 12 is fed to a computer 16 executing one or more applications programs and is treated either as mouse events or writing depending on the object used to contact the touch surface 14. Specifically, the contact position output of the touch panel 12 is treated as mouse events if a finger or other pointer is used to contact the touch surface 14. The contact position is however treated as writing when a pen is lifted from the tool tray 15 and is used to contact the touch surface 14. In this manner the touch system either operates in a pointer mode or ink mode depending on the object used to contact the touch surface 14. The computer 16 is also coupled to a front or rear projector 18 and provides image data to the projector. The projector 18 in turn presents images on the touch surface 14 of the touch panel. The touch panel 12, computer 16 and projector 18 form a closed-loop so that user contacts with the touch panel 12 can be recorded as writing or used to control execution of an applications program executed by the computer 16.
  • In the present embodiment, the [0032] computer 16 runs in a Windows environment and provides image data to the projector 18 so that a computer desktop image is presented on the touch surface 14. The computer desktop image presents one or more icons that can be selected to open associated applications programs. When an applications program is selected, a window for the applications program is opened.
  • FIG. 2 shows an [0033] applications program window 30 including an active area 32 bordered by a frame and one or more tool bars 34 is shown presented on the touch surface 14. When a pen is used to contact the touch surface 14 and the touch system is conditioned to the ink mode, an acetate image 40 identical to the computer desktop image is displayed over the computer desktop image as shown FIG. 2. The acetate image 40 provides a surface for writing (commonly referred to as “ink”) so that the ink is not used by the computer 16 to update the underlying computer desktop image 30. Contact position data returned to the computer 16 by the touch panel 12 in response to pen contacts with the touch surface 14 are used by the computer to update the image data conveyed to the projector 18 so that the ink appears on the acetate image 40.
  • When the [0034] touch system 10 is conditioned to the ink mode, the computer 16 can be conditioned to execute a connector gesture recognition routine to interpret freehand connectors drawn between graphical objects, if the user is using the pen to draw a two-dimensional diagram, by selecting an icon presented on the touch surface 14. Specifics of the connector gesture recognition routine will now be described with reference to FIGS. 3a to 3 c.
  • With the [0035] computer 16 is executing the gesture connector recognition routine, when a user draws a freehand line on the touch surface 14 (step 100) and the touch panel output is conveyed to the computer 16, the connector gesture recognition routine examines the start point and end point of the freehand line to determine if the start point and end point of the freehand line touch graphical objects displayed on the touch surface that can be connected (step 102). If the start point and end point of the freehand line do not touch graphical objects that can be connected, the freehand line is maintained in its original form (step 104). If the start point and end point of the freehand line touch graphical objects that can be connected, a connector gesture determination is made to determine the manner by which the freehand line is to be represented (step 106). The freehand line is then replaced by the determined computer-generated connector and the image data conveyed to the projector 18 is updated accordingly so that the computer-generated connector approximating the freehand line is presented on the touch surface 14 (step 108).
  • During [0036] step 102 when the start point and end point of the freehand line are examined, initially the start point is examined to determine if it touches a graphical object (step 120). If the start point of the freehand line does not touch a graphical object, the user is presented with a prompt to determine if a new graphical object is to be created at the start point of the freehand line (step 122). If the user does not wish a new graphical object to be created, the connector gesture recognition routine proceeds to step 104.
  • If the start point of the freehand line touches a graphical object or if the user conditions the connector gesture recognition routine to create a new graphical object at the start of the freehand line, the connector gesture recognition routine examines the end point of the freehand line to determine if it touches a graphical object (step [0037] 124). If not, the connector gesture recognition routine proceeds to step 122 to determine if the user wishes to create a new graphical object at the end of the freehand line. If the end point of the freehand line touches a graphical object or if the user conditions the connector gesture recognition routine to create a new graphical object at the end of the freehand line, the connector gesture recognition routine proceeds to step 106 to determine how to represent the freehand line. Otherwise, the connector gesture recognition routine proceeds to step 104.
  • During [0038] step 106, the connector gesture recognition routine sets the start and end points of the freehand line (step 130) and analyzes points along the freehand line in order to select an appropriate representation (step 132). Based on the analysis, if the freehand line is not a straight line and is determined to have an elbow along its length (step 134), the connector gesture recognition routine recognizes the freehand line as a right-angled line (step 136). If the line has a winding path (step 138), the connector gesture recognition routine recognizes the freehand line either as a polygonal, splined or circular line depending on the shape of the freehand line (step 140). If the freehand line is not straight nor winding (step 142), the connector gesture recognition routine recognizes the freehand line as a curved line (step 146). The curve is fitted using an apogee as the control knot (i.e. the furthest point out away from the graphical objects). If the freehand line is straight (step 148), the connector gesture recognition routine recognizes the freehand line as a straight line (step 150). Once the freehand line has been recognized, the connector gesture recognition routine replaces the freehand line with a computer-generated equivalent and updates the image data conveyed to the projector 18 so that the appropriate computer-generated connector is displayed on the touch surface 14.
  • Turning now to FIGS. [0039] 4 to 10, examples of connector gesture recognitions are shown. As can be seen in FIG. 4, a freehand line 200 interconnecting two graphical objects 202 and 204 is recognized as a straight line 206. In FIG. 5, a freehand line 210 that is not straight and that has elbows is recognized as a right-angled line 216. In FIG. 6, a non-straight freehand line 220 is recognized as a curved line 226 and in FIG. 7, a winding line 230 is recognized as a circular line 236. FIG. 8 shows a freehand straight line 240 having a start point adjacent a graphical object 242. The end point of the freehand line 240 does not touch a graphical object. In this case, the user at step 122 has requested the connector gesture recognition routine to create a new graphical object 244 at the end of the freehand line and the freehand line is recognized as a straight line 246.
  • FIG. 9 shows relationships between [0040] graphical objects 252 and 254 in an object oriented program. As shown, three separate curved computer-generated connectors 256 a to 256 c extend between the graphical objects. The computer-generated connectors are created in manner that has been described above. It will be appreciated that any number of connectors may extend between pairs of graphical objects. FIG. 10 shows a computer-generated winding line 266 extending between a pair of graphical objects 262 and 264.
  • It will also be appreciated that the computer-generated connectors can be created without arrow heads, with single arrow heads or with double arrow heads. In addition, the various computer-generated connectors can be displayed in different colours using a software color tool. [0041]
  • Although the touch system has been described as switching between the pointer mode and the ink mode depending on the type of pointer used to contact the touch surface, this is for illustrative purposes only. For example, transitions between the pointer mode and ink mode can be achieved by selecting appropriate buttons on a small window that remains visible on the [0042] touch surface 14. In this manner, the same pointer can be used to generate mouse events or ink. The gesture responsive ink injection routine can be used in any touch system that includes a touch panel on which a computer image is presented. Such touch systems include active touch systems that make use of special pointers that emit signals such as infrared light, visible light, ultrasonic frequencies and electromagnetic frequencies in order to activate the touch surfaces. These touch systems also include passive surface acoustic wave or capacitive-type touch systems as well as camera-based touch systems such as that disclosed in International PCT Application No. WO 02/03316 to Smart Technologies Inc. et al., the assignee of the present invention.
  • It will also be appreciated that the gesture connector recognition routine may be run on basically any computing device where freehand connectors joining graphical objects are entered. A mouse, trackball, touch pad, or other pointing device may be used to create the freehand connectors. [0043]
  • During analysis of the freehand connectors, the freehand connectors need not touch graphical objects in order for the connector gesture recognition routine to extend computer-generated connectors from them. Rather, the freehand connectors simply need to be within a threshold distance of graphical objects. In addition, prompting the user to determine whether new graphical objects are to be created at the start and end points of freehand lines is optional. This feature may be disabled or designated as a default. When designated as a default, the connector gesture recognition routine creates new graphical objects at the start and end points of freehand lines automatically without requiring user input. [0044]
  • Although a preferred embodiment of the present invention has been described, those of skill in the art will appreciate that variations and modifications may be made without departing from the spirit and scope thereof as defined by the appended claims. [0045]

Claims (14)

What is claimed is:
1. A method of recognizing a freehand connector input into a computer comprising the steps of:
examining points along said freehand connector to determine the shape thereof; and
replacing said freehand connector with a computer-generated connector approximating said freehand connector.
2. The method of claim 1 further comprising the step of determining whether the start and end points of said freehand connector are proximate to graphical objects and if so, extending the computer-generated connector between said graphical objects.
3. The method of claim 2 further comprising the step of generating a new graphical object at at least one of the start and end points of the computer-generated connector if the at least one of the start and end points of said freehand connector is not proximate to a graphical object.
4. The method of claim 3 wherein said freehand connector is input into said computer in response to drawing made on a touch panel having a touch surface.
5. A system for recognizing a freehand connector input into a computer comprising:
means for examining points along said freehand connector to determine the shape thereof; and
means for replacing said freehand connector with a computer-generated connector approximating said freehand connector.
6. A system according to claim 5 further comprising means for determining whether the start and end points of said freehand connector are proximate to graphical objects and if so, means for extending the computer-generated connector between said graphical objects.
7. A system according to claim 6 further comprising means for generating a new graphical object at at least one of the start and end points of the computer-generated connector if the at least one of the start and end points of said freehand connector is not proximate to a graphical object.
8. A system according to claim 7 wherein said freehand connector is input into said computer in response to drawing made on a touch panel having a touch surface.
9. A touch system comprising:
a touch panel having a touch surface;
a projector presenting images onto said touch surface; and
a computer executing an applications program and being coupled to said touch panel and said projector, said computer being responsive to contact output generated by said touch panel in response to proximity of a pointer to said touch surface and updating image data conveyed to said projector so that images presented on said touch surface reflect pointer activity, said computer executing a connector gesture recognition routine, said connector gesture recognition routine performing recognition to convert a freehand connector on said touch surface into a computer-generated connector approximating said freehand connector.
10. A touch system according to claim 9 wherein said connector gesture recognition routine determines whether the start and end points of said freehand connector are proximate to graphical objects and if so, extends the computer-generated connector between said graphical objects.
11. A touch system according to claim 10 wherein said connector gesture recognition routine generates a new graphical object at at least one of the start and end points of the computer-generated connector if the at least one of the start and end points of said freehand connector is not proximate to a graphical object.
12. A computer program product including a computer readable medium having a computer program for recognizing a freehand connector input into a computer embodied thereon, said computer program including:
computer program code for examining points along said freehand connector to determine the shape thereof; and
computer program code for replacing said freehand connector with a computer-generated connector approximating said freehand connector.
13. A computer program product according to claim 12 further comprising computer program code for determining whether the start and end points of said freehand connector are proximate to graphical objects and if so, means for extending the computer-generated connector between said graphical objects.
14. A computer program product according to claim 13 further comprising computer program code for generating a new graphical object at at least one of the start and end points of the computer-generated connector if the at least one of the start and end points of said freehand connector is not proximate to a graphical object.
US10/171,099 2002-06-12 2002-06-12 System and method for recognizing connector gestures Expired - Lifetime US7330184B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US10/171,099 US7330184B2 (en) 2002-06-12 2002-06-12 System and method for recognizing connector gestures
CA2390506A CA2390506C (en) 2002-06-12 2002-06-12 System and method for recognizing connector gestures

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/171,099 US7330184B2 (en) 2002-06-12 2002-06-12 System and method for recognizing connector gestures
CA2390506A CA2390506C (en) 2002-06-12 2002-06-12 System and method for recognizing connector gestures

Publications (2)

Publication Number Publication Date
US20030231219A1 true US20030231219A1 (en) 2003-12-18
US7330184B2 US7330184B2 (en) 2008-02-12

Family

ID=32178115

Family Applications (1)

Application Number Title Priority Date Filing Date
US10/171,099 Expired - Lifetime US7330184B2 (en) 2002-06-12 2002-06-12 System and method for recognizing connector gestures

Country Status (2)

Country Link
US (1) US7330184B2 (en)
CA (1) CA2390506C (en)

Cited By (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007136372A1 (en) * 2006-05-22 2007-11-29 Thomson Licensing Video system having a touch screen
US20090213070A1 (en) * 2006-06-16 2009-08-27 Ketab Technologies Limited Processor control and display system
US20090319948A1 (en) * 2008-06-20 2009-12-24 Smartdraw.Com Automated editing of graphics charts
US20110016384A1 (en) * 2006-09-29 2011-01-20 Joshua Shagam Optimizing typographical content for transmission and display
US20110185318A1 (en) * 2010-01-27 2011-07-28 Microsoft Corporation Edge gestures
US20110209103A1 (en) * 2010-02-25 2011-08-25 Hinckley Kenneth P Multi-screen hold and drag gesture
US20110209101A1 (en) * 2010-02-25 2011-08-25 Hinckley Kenneth P Multi-screen pinch-to-pocket gesture
US20120131515A1 (en) * 2010-11-22 2012-05-24 Amx, Llc Method and apparatus of error correction in resistive touch panels
US8261213B2 (en) 2010-01-28 2012-09-04 Microsoft Corporation Brush, carbon-copy, and fill gestures
US20120320061A1 (en) * 2011-06-14 2012-12-20 Nintendo Co., Ltd Drawing method
US8499236B1 (en) 2010-01-21 2013-07-30 Amazon Technologies, Inc. Systems and methods for presenting reflowable content on a display
US8539384B2 (en) 2010-02-25 2013-09-17 Microsoft Corporation Multi-screen pinch and expand gestures
US8566707B1 (en) 2006-03-29 2013-10-22 Amazon Technologies, Inc. Generating image-based reflowable files for rendering on various sized displays
US8572480B1 (en) * 2008-05-30 2013-10-29 Amazon Technologies, Inc. Editing the sequential flow of a page
US8707174B2 (en) 2010-02-25 2014-04-22 Microsoft Corporation Multi-screen hold and page-flip gesture
US8751970B2 (en) 2010-02-25 2014-06-10 Microsoft Corporation Multi-screen synchronous slide gesture
US8782516B1 (en) 2007-12-21 2014-07-15 Amazon Technologies, Inc. Content style detection
US8799827B2 (en) 2010-02-19 2014-08-05 Microsoft Corporation Page manipulations using on and off-screen gestures
US8836648B2 (en) 2009-05-27 2014-09-16 Microsoft Corporation Touch pull-in gesture
US9052820B2 (en) 2011-05-27 2015-06-09 Microsoft Technology Licensing, Llc Multi-application environment
US9075522B2 (en) 2010-02-25 2015-07-07 Microsoft Technology Licensing, Llc Multi-screen bookmark hold gesture
US9104440B2 (en) 2011-05-27 2015-08-11 Microsoft Technology Licensing, Llc Multi-application environment
US9158445B2 (en) 2011-05-27 2015-10-13 Microsoft Technology Licensing, Llc Managing an immersive interface in a multi-application immersive environment
US9229918B2 (en) 2010-12-23 2016-01-05 Microsoft Technology Licensing, Llc Presenting an application change through a tile
US9229911B1 (en) 2008-09-30 2016-01-05 Amazon Technologies, Inc. Detecting continuation of flow of a page
US9261964B2 (en) 2005-12-30 2016-02-16 Microsoft Technology Licensing, Llc Unintentional touch rejection
US9274682B2 (en) 2010-02-19 2016-03-01 Microsoft Technology Licensing, Llc Off-screen gestures to create on-screen input
US9310994B2 (en) 2010-02-19 2016-04-12 Microsoft Technology Licensing, Llc Use of bezel as an input mechanism
US9367205B2 (en) 2010-02-19 2016-06-14 Microsoft Technolgoy Licensing, Llc Radial menus with bezel gestures
US9411504B2 (en) 2010-01-28 2016-08-09 Microsoft Technology Licensing, Llc Copy and staple gestures
US9454304B2 (en) 2010-02-25 2016-09-27 Microsoft Technology Licensing, Llc Multi-screen dual tap gesture
US9477337B2 (en) 2014-03-14 2016-10-25 Microsoft Technology Licensing, Llc Conductive trace routing for display and bezel sensors
US9519356B2 (en) 2010-02-04 2016-12-13 Microsoft Technology Licensing, Llc Link gestures
US9582122B2 (en) 2012-11-12 2017-02-28 Microsoft Technology Licensing, Llc Touch-sensitive bezel techniques
US9658766B2 (en) 2011-05-27 2017-05-23 Microsoft Technology Licensing, Llc Edge gesture
US9696888B2 (en) 2010-12-20 2017-07-04 Microsoft Technology Licensing, Llc Application-launching interface for multiple modes
US9965165B2 (en) 2010-02-19 2018-05-08 Microsoft Technology Licensing, Llc Multi-finger gestures
US10254955B2 (en) 2011-09-10 2019-04-09 Microsoft Technology Licensing, Llc Progressively indicating new content in an application-selectable user interface
KR20190113741A (en) * 2016-12-15 2019-10-08 마이스크립트 System and method for management of handwritten diagram connectors
US10579250B2 (en) 2011-09-01 2020-03-03 Microsoft Technology Licensing, Llc Arranging tiles
US10969944B2 (en) 2010-12-23 2021-04-06 Microsoft Technology Licensing, Llc Application reporting in an application-selectable user interface
US11272017B2 (en) 2011-05-27 2022-03-08 Microsoft Technology Licensing, Llc Application notifications manifest

Families Citing this family (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6803906B1 (en) 2000-07-05 2004-10-12 Smart Technologies, Inc. Passive touch system and method of detecting user input
US6954197B2 (en) 2002-11-15 2005-10-11 Smart Technologies Inc. Size/scale and orientation determination of a pointer in a camera-based touch system
US7629967B2 (en) 2003-02-14 2009-12-08 Next Holdings Limited Touch screen signal processing
US8456447B2 (en) 2003-02-14 2013-06-04 Next Holdings Limited Touch screen signal processing
US8508508B2 (en) 2003-02-14 2013-08-13 Next Holdings Limited Touch screen signal processing with single-point calibration
US7532206B2 (en) 2003-03-11 2009-05-12 Smart Technologies Ulc System and method for differentiating between pointers used to contact touch surface
US7411575B2 (en) 2003-09-16 2008-08-12 Smart Technologies Ulc Gesture recognition method and touch system incorporating the same
US7274356B2 (en) 2003-10-09 2007-09-25 Smart Technologies Inc. Apparatus for determining the location of a pointer within a region of interest
US7355593B2 (en) 2004-01-02 2008-04-08 Smart Technologies, Inc. Pointer tracking across multiple overlapping coordinate input sub-regions defining a generally contiguous input region
US7460110B2 (en) * 2004-04-29 2008-12-02 Smart Technologies Ulc Dual mode touch system
US7492357B2 (en) * 2004-05-05 2009-02-17 Smart Technologies Ulc Apparatus and method for detecting a pointer relative to a touch surface
US7538759B2 (en) 2004-05-07 2009-05-26 Next Holdings Limited Touch panel display system with illumination and detection provided from a single edge
US8120596B2 (en) 2004-05-21 2012-02-21 Smart Technologies Ulc Tiled touch system
JP4771831B2 (en) * 2006-03-02 2011-09-14 富士通株式会社 Graphic display program and graphic display method
US9442607B2 (en) * 2006-12-04 2016-09-13 Smart Technologies Inc. Interactive input system and method
US8115753B2 (en) 2007-04-11 2012-02-14 Next Holdings Limited Touch screen system with hover and click input methods
US8094137B2 (en) * 2007-07-23 2012-01-10 Smart Technologies Ulc System and method of detecting contact on a display
KR20100055516A (en) * 2007-08-30 2010-05-26 넥스트 홀딩스 인코포레이티드 Optical touchscreen with improved illumination
WO2009029764A1 (en) 2007-08-30 2009-03-05 Next Holdings, Inc. Low profile touch panel systems
US20090213093A1 (en) * 2008-01-07 2009-08-27 Next Holdings Limited Optical position sensor using retroreflection
US20090207144A1 (en) * 2008-01-07 2009-08-20 Next Holdings Limited Position Sensing System With Edge Positioning Enhancement
US8405636B2 (en) * 2008-01-07 2013-03-26 Next Holdings Limited Optical position sensing system and optical position sensor assembly
US20090278794A1 (en) * 2008-05-09 2009-11-12 Smart Technologies Ulc Interactive Input System With Controlled Lighting
US20090278795A1 (en) * 2008-05-09 2009-11-12 Smart Technologies Ulc Interactive Input System And Illumination Assembly Therefor
US8902193B2 (en) * 2008-05-09 2014-12-02 Smart Technologies Ulc Interactive input system and bezel therefor
US20090277697A1 (en) * 2008-05-09 2009-11-12 Smart Technologies Ulc Interactive Input System And Pen Tool Therefor
WO2010039663A2 (en) * 2008-10-02 2010-04-08 Next Holdings, Inc. Stereo optical sensors for resolving multi-touch in a touch detection system
US20100100866A1 (en) * 2008-10-21 2010-04-22 International Business Machines Corporation Intelligent Shared Virtual Whiteboard For Use With Representational Modeling Languages
US8339378B2 (en) * 2008-11-05 2012-12-25 Smart Technologies Ulc Interactive input system with multi-angle reflector
US9335909B2 (en) * 2009-06-03 2016-05-10 Honda Motor Co., Ltd. Drawing assist device, drawing assist program, and drawing assist method
US20100229090A1 (en) * 2009-03-05 2010-09-09 Next Holdings Limited Systems and Methods for Interacting With Touch Displays Using Single-Touch and Multi-Touch Gestures
US20110095989A1 (en) * 2009-10-23 2011-04-28 Smart Technologies Ulc Interactive input system and bezel therefor
US20110199387A1 (en) * 2009-11-24 2011-08-18 John David Newton Activating Features on an Imaging Device Based on Manipulations
CN102713794A (en) * 2009-11-24 2012-10-03 奈克斯特控股公司 Methods and apparatus for gesture recognition mode control
US20110205155A1 (en) * 2009-12-04 2011-08-25 John David Newton Methods and Systems for Position Detection Using an Interactive Volume
US20110234542A1 (en) * 2010-03-26 2011-09-29 Paul Marson Methods and Systems Utilizing Multiple Wavelengths for Position Detection
US10643067B2 (en) 2015-10-19 2020-05-05 Myscript System and method of handwriting recognition in diagrams
US10417491B2 (en) 2015-10-19 2019-09-17 Myscript System and method for recognition of handwritten diagram connectors
EP3736677A1 (en) 2019-05-10 2020-11-11 MyScript A method and corresponding device for selecting and editing handwriting input elements
EP3754537A1 (en) 2019-06-20 2020-12-23 MyScript Processing text handwriting input in a free handwriting mode
EP3772015B1 (en) 2019-07-31 2023-11-08 MyScript Text line extraction
EP3796145A1 (en) 2019-09-19 2021-03-24 MyScript A method and correspond device for selecting graphical objects

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1709998A (en) * 1925-01-02 1929-04-23 Youngstown Sheet And Tube Co Method of manufacturing well points
US5299307A (en) * 1990-08-17 1994-03-29 Claris Corporation Controls for drawing images on computer displays
US5448263A (en) * 1991-10-21 1995-09-05 Smart Technologies Inc. Interactive display system
US5517604A (en) * 1991-03-20 1996-05-14 Hitachi, Ltd. Data input/output process and design supporting process
US5636297A (en) * 1992-09-10 1997-06-03 Microsoft Corporation Method and system for recognizing a graphic object's shape, line style, and fill pattern in a pen environment
US5737981A (en) * 1995-09-20 1998-04-14 Hildebrand; David Lewis Removal device for threaded connecting devices
US5969699A (en) * 1996-10-08 1999-10-19 Kaiser Aerospace & Electronics Company Stroke-to-stroke
US5999186A (en) * 1997-05-23 1999-12-07 3-Design L.L.C. Reference based parametric dimensioning method and system
US6233351B1 (en) * 1994-08-12 2001-05-15 Dassault Systemes Of America Corp. Method and apparatus for processing a freehand sketch
US6239813B1 (en) * 1999-01-20 2001-05-29 Timothy Joseph Erskine Method of defining a computer-generated intelligent symbol
US6243101B1 (en) * 1998-09-03 2001-06-05 Timothy Joseph Erskine Method of enhancing computer-generated symbols
US20020154117A1 (en) * 1999-03-29 2002-10-24 Nobuhiro Saitou Graphic editing apparatus graphic editing method and storage medium on which is recorded a program for graphic editing
US20030206169A1 (en) * 2001-09-26 2003-11-06 Michael Springer System, method and computer program product for automatically snapping lines to drawing elements

Patent Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1709998A (en) * 1925-01-02 1929-04-23 Youngstown Sheet And Tube Co Method of manufacturing well points
US5299307A (en) * 1990-08-17 1994-03-29 Claris Corporation Controls for drawing images on computer displays
US5517604A (en) * 1991-03-20 1996-05-14 Hitachi, Ltd. Data input/output process and design supporting process
US5448263A (en) * 1991-10-21 1995-09-05 Smart Technologies Inc. Interactive display system
US5636297A (en) * 1992-09-10 1997-06-03 Microsoft Corporation Method and system for recognizing a graphic object's shape, line style, and fill pattern in a pen environment
US6233351B1 (en) * 1994-08-12 2001-05-15 Dassault Systemes Of America Corp. Method and apparatus for processing a freehand sketch
US5737981A (en) * 1995-09-20 1998-04-14 Hildebrand; David Lewis Removal device for threaded connecting devices
US5969699A (en) * 1996-10-08 1999-10-19 Kaiser Aerospace & Electronics Company Stroke-to-stroke
US5999186A (en) * 1997-05-23 1999-12-07 3-Design L.L.C. Reference based parametric dimensioning method and system
US6243101B1 (en) * 1998-09-03 2001-06-05 Timothy Joseph Erskine Method of enhancing computer-generated symbols
US20010010519A1 (en) * 1998-09-03 2001-08-02 Erskine Timothy Joseph Method of converting symbols into intelligent symbols
US6239813B1 (en) * 1999-01-20 2001-05-29 Timothy Joseph Erskine Method of defining a computer-generated intelligent symbol
US20010035872A1 (en) * 1999-01-20 2001-11-01 Erskine Timothy Joseph Method of defining a computer-generated intelligent symbol
US20020154117A1 (en) * 1999-03-29 2002-10-24 Nobuhiro Saitou Graphic editing apparatus graphic editing method and storage medium on which is recorded a program for graphic editing
US20030206169A1 (en) * 2001-09-26 2003-11-06 Michael Springer System, method and computer program product for automatically snapping lines to drawing elements

Cited By (68)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9946370B2 (en) 2005-12-30 2018-04-17 Microsoft Technology Licensing, Llc Unintentional touch rejection
US9952718B2 (en) 2005-12-30 2018-04-24 Microsoft Technology Licensing, Llc Unintentional touch rejection
US9261964B2 (en) 2005-12-30 2016-02-16 Microsoft Technology Licensing, Llc Unintentional touch rejection
US10019080B2 (en) 2005-12-30 2018-07-10 Microsoft Technology Licensing, Llc Unintentional touch rejection
US9594457B2 (en) 2005-12-30 2017-03-14 Microsoft Technology Licensing, Llc Unintentional touch rejection
US8566707B1 (en) 2006-03-29 2013-10-22 Amazon Technologies, Inc. Generating image-based reflowable files for rendering on various sized displays
US20090153501A1 (en) * 2006-05-22 2009-06-18 Joseph J. Laks Thomson Licensing Llc Video System Having a Touch Screen
US9104270B2 (en) 2006-05-22 2015-08-11 Thomson Licensing Video system having a touch screen
WO2007136372A1 (en) * 2006-05-22 2007-11-29 Thomson Licensing Video system having a touch screen
US20090213070A1 (en) * 2006-06-16 2009-08-27 Ketab Technologies Limited Processor control and display system
US8723791B2 (en) * 2006-06-16 2014-05-13 Ketab Technologies Limited Processor control and display system
US20110016384A1 (en) * 2006-09-29 2011-01-20 Joshua Shagam Optimizing typographical content for transmission and display
US9208133B2 (en) 2006-09-29 2015-12-08 Amazon Technologies, Inc. Optimizing typographical content for transmission and display
US8782516B1 (en) 2007-12-21 2014-07-15 Amazon Technologies, Inc. Content style detection
US8572480B1 (en) * 2008-05-30 2013-10-29 Amazon Technologies, Inc. Editing the sequential flow of a page
US20090319948A1 (en) * 2008-06-20 2009-12-24 Smartdraw.Com Automated editing of graphics charts
US9229911B1 (en) 2008-09-30 2016-01-05 Amazon Technologies, Inc. Detecting continuation of flow of a page
US8836648B2 (en) 2009-05-27 2014-09-16 Microsoft Corporation Touch pull-in gesture
US8499236B1 (en) 2010-01-21 2013-07-30 Amazon Technologies, Inc. Systems and methods for presenting reflowable content on a display
US20110185318A1 (en) * 2010-01-27 2011-07-28 Microsoft Corporation Edge gestures
US8239785B2 (en) * 2010-01-27 2012-08-07 Microsoft Corporation Edge gestures
US8261213B2 (en) 2010-01-28 2012-09-04 Microsoft Corporation Brush, carbon-copy, and fill gestures
US9411498B2 (en) 2010-01-28 2016-08-09 Microsoft Technology Licensing, Llc Brush, carbon-copy, and fill gestures
US9411504B2 (en) 2010-01-28 2016-08-09 Microsoft Technology Licensing, Llc Copy and staple gestures
US9857970B2 (en) 2010-01-28 2018-01-02 Microsoft Technology Licensing, Llc Copy and staple gestures
US10282086B2 (en) 2010-01-28 2019-05-07 Microsoft Technology Licensing, Llc Brush, carbon-copy, and fill gestures
US9519356B2 (en) 2010-02-04 2016-12-13 Microsoft Technology Licensing, Llc Link gestures
US8799827B2 (en) 2010-02-19 2014-08-05 Microsoft Corporation Page manipulations using on and off-screen gestures
US9274682B2 (en) 2010-02-19 2016-03-01 Microsoft Technology Licensing, Llc Off-screen gestures to create on-screen input
US10268367B2 (en) 2010-02-19 2019-04-23 Microsoft Technology Licensing, Llc Radial menus with bezel gestures
US9367205B2 (en) 2010-02-19 2016-06-14 Microsoft Technolgoy Licensing, Llc Radial menus with bezel gestures
US9310994B2 (en) 2010-02-19 2016-04-12 Microsoft Technology Licensing, Llc Use of bezel as an input mechanism
US9965165B2 (en) 2010-02-19 2018-05-08 Microsoft Technology Licensing, Llc Multi-finger gestures
US8751970B2 (en) 2010-02-25 2014-06-10 Microsoft Corporation Multi-screen synchronous slide gesture
US20110209103A1 (en) * 2010-02-25 2011-08-25 Hinckley Kenneth P Multi-screen hold and drag gesture
US11055050B2 (en) 2010-02-25 2021-07-06 Microsoft Technology Licensing, Llc Multi-device pairing and combined display
US9075522B2 (en) 2010-02-25 2015-07-07 Microsoft Technology Licensing, Llc Multi-screen bookmark hold gesture
US8473870B2 (en) 2010-02-25 2013-06-25 Microsoft Corporation Multi-screen hold and drag gesture
US8707174B2 (en) 2010-02-25 2014-04-22 Microsoft Corporation Multi-screen hold and page-flip gesture
US9454304B2 (en) 2010-02-25 2016-09-27 Microsoft Technology Licensing, Llc Multi-screen dual tap gesture
US20110209101A1 (en) * 2010-02-25 2011-08-25 Hinckley Kenneth P Multi-screen pinch-to-pocket gesture
US8539384B2 (en) 2010-02-25 2013-09-17 Microsoft Corporation Multi-screen pinch and expand gestures
US9483170B2 (en) * 2010-11-22 2016-11-01 Amx Llc Method and apparatus of error correction in resistive touch panels
US20120131515A1 (en) * 2010-11-22 2012-05-24 Amx, Llc Method and apparatus of error correction in resistive touch panels
US9696888B2 (en) 2010-12-20 2017-07-04 Microsoft Technology Licensing, Llc Application-launching interface for multiple modes
US9229918B2 (en) 2010-12-23 2016-01-05 Microsoft Technology Licensing, Llc Presenting an application change through a tile
US11126333B2 (en) 2010-12-23 2021-09-21 Microsoft Technology Licensing, Llc Application reporting in an application-selectable user interface
US10969944B2 (en) 2010-12-23 2021-04-06 Microsoft Technology Licensing, Llc Application reporting in an application-selectable user interface
US10303325B2 (en) 2011-05-27 2019-05-28 Microsoft Technology Licensing, Llc Multi-application environment
US9658766B2 (en) 2011-05-27 2017-05-23 Microsoft Technology Licensing, Llc Edge gesture
US11698721B2 (en) 2011-05-27 2023-07-11 Microsoft Technology Licensing, Llc Managing an immersive interface in a multi-application immersive environment
US11272017B2 (en) 2011-05-27 2022-03-08 Microsoft Technology Licensing, Llc Application notifications manifest
US9158445B2 (en) 2011-05-27 2015-10-13 Microsoft Technology Licensing, Llc Managing an immersive interface in a multi-application immersive environment
US9104307B2 (en) 2011-05-27 2015-08-11 Microsoft Technology Licensing, Llc Multi-application environment
US9535597B2 (en) 2011-05-27 2017-01-03 Microsoft Technology Licensing, Llc Managing an immersive interface in a multi-application immersive environment
US9052820B2 (en) 2011-05-27 2015-06-09 Microsoft Technology Licensing, Llc Multi-application environment
US9104440B2 (en) 2011-05-27 2015-08-11 Microsoft Technology Licensing, Llc Multi-application environment
US9292948B2 (en) * 2011-06-14 2016-03-22 Nintendo Co., Ltd. Drawing method
US20120320061A1 (en) * 2011-06-14 2012-12-20 Nintendo Co., Ltd Drawing method
US10176613B2 (en) 2011-06-14 2019-01-08 Nintendo Co., Ltd. Drawing method
US10579250B2 (en) 2011-09-01 2020-03-03 Microsoft Technology Licensing, Llc Arranging tiles
US10254955B2 (en) 2011-09-10 2019-04-09 Microsoft Technology Licensing, Llc Progressively indicating new content in an application-selectable user interface
US10656750B2 (en) 2012-11-12 2020-05-19 Microsoft Technology Licensing, Llc Touch-sensitive bezel techniques
US9582122B2 (en) 2012-11-12 2017-02-28 Microsoft Technology Licensing, Llc Touch-sensitive bezel techniques
US9946383B2 (en) 2014-03-14 2018-04-17 Microsoft Technology Licensing, Llc Conductive trace routing for display and bezel sensors
US9477337B2 (en) 2014-03-14 2016-10-25 Microsoft Technology Licensing, Llc Conductive trace routing for display and bezel sensors
KR20190113741A (en) * 2016-12-15 2019-10-08 마이스크립트 System and method for management of handwritten diagram connectors
KR102470522B1 (en) * 2016-12-15 2022-11-23 마이스크립트 System and method for management of handwritten diagram connectors

Also Published As

Publication number Publication date
US7330184B2 (en) 2008-02-12
CA2390506A1 (en) 2003-12-12
CA2390506C (en) 2013-04-02

Similar Documents

Publication Publication Date Title
US7330184B2 (en) System and method for recognizing connector gestures
US8707217B2 (en) User interface for stylus-based user input
US8810509B2 (en) Interfacing with a computing application using a multi-digit sensor
US7058902B2 (en) Enhanced on-object context menus
US8508489B2 (en) System and method for injecting ink into an application
US5809267A (en) Apparatus and method for executing multiple-concatenated command gestures in a gesture based input system
US6928619B2 (en) Method and apparatus for managing input focus and z-order
JP3546337B2 (en) User interface device for computing system and method of using graphic keyboard
JP4809558B2 (en) High level active pen matrix
JP5542774B2 (en) Flick gesture
US7283126B2 (en) System and method for providing gesture suggestions to enhance interpretation of user input
US5872559A (en) Breakaway and re-grow touchscreen pointing device
EP3491506B1 (en) Systems and methods for a touchscreen user interface for a collaborative editing tool
US20020057263A1 (en) Simulating gestures of a pointing device using a stylus and providing feedback thereto
US20100251189A1 (en) Using gesture objects to replace menus for computer control
JPH07152527A (en) Graphic editing device
KR20040086544A (en) Dynamic feedback for gestures
TW200302977A (en) Ink gestures
CA2390503C (en) System and method for providing gesture suggestions to enhance interpretation of user input
WO2012094740A1 (en) Method for supporting multiple menus and interactive input system employing same
US20220350418A1 (en) Composite computer keyboard
Zhu et al. Ringedit: A control point based editing approach in sketch recognition systems
KRISHNAN et al. that can Integrate or Separate Operand Specification, Menu Opening and Selection, and Action Execution

Legal Events

Date Code Title Description
AS Assignment

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LEUNG, ANDY;REEL/FRAME:013273/0952

Effective date: 20020625

AS Assignment

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: MERGER;ASSIGNOR:SMART TECHNOLOGIES INC.;REEL/FRAME:020257/0187

Effective date: 20070828

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING INC., NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNORS:SMART TECHNOLOGIES ULC;SMART TECHNOLOGIES INC.;REEL/FRAME:030935/0848

Effective date: 20130731

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNORS:SMART TECHNOLOGIES ULC;SMART TECHNOLOGIES INC.;REEL/FRAME:030935/0879

Effective date: 20130731

FPAY Fee payment

Year of fee payment: 8

AS Assignment

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: RELEASE OF TERM LOAN SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040713/0123

Effective date: 20161003

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: RELEASE OF ABL SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040711/0956

Effective date: 20161003

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: RELEASE OF TERM LOAN SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040713/0123

Effective date: 20161003

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: RELEASE OF ABL SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040711/0956

Effective date: 20161003

AS Assignment

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040798/0077

Effective date: 20161003

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040798/0077

Effective date: 20161003

Owner name: SMART TECHNOLOGIES INC., CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040819/0306

Effective date: 20161003

Owner name: SMART TECHNOLOGIES ULC, CANADA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:040819/0306

Effective date: 20161003

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 12