WO2015173399A1

WO2015173399A1 - Drift compensation / optical absolute referencing

Info

Publication number: WO2015173399A1
Application number: PCT/EP2015/060774
Authority: WO
Inventors: Karl-Peter KÄMPF; Dr. Thomas LINK
Original assignee: Stabilo International Gmbh
Priority date: 2014-05-15
Filing date: 2015-05-15
Publication date: 2015-11-19
Also published as: DE102014106838A1; DE102014106838B4

Abstract

The invention relates to an electronic pen (500, 600) with pen position detection, comprising a writing rod, at least one electrical power source, at least one digital processing unit, at least one data transmission module, and an inertial positioning sensor system for detecting the position and motion of the electronic pen, and at least one optical sensor (108, 209), wherein the optical sensor (109, 209) is configured for recording and storing image data of at least one surrounding area of the writing rod tip, and the digital processing unit Is configured for creating a mapping of a writing substrate from said recorded image data and for absolute referencing of a first location signal of the electronic pen determined from the inertial positioning sensor system by referencing the determined location signal of the positioning sensor system with a second location signal determined by use of the optical sensor (109, 209) determined from the comparison of stationary features of the surrounding area of the writing rod tip identified in the image data of the current surrounding area of the writing rod tip with stationary features identified in image data of at least one already stored surrounding area of the writing rod and/or identified in a mapping of the writing substrate.

Description

Drift compensation / optical absolute referencing

Prior art

The invention relates to an electronic pen of the kind specified in the preamble of claim 1 , a method of the kind specified in the preamble of claim 10, as well as a system of the kind specified in the preamble of claim 17.

When detecting motions of an electronic pen for character recognition by inertial measurement systems, such as accelerometers or rotation rate sensors, single or dual integration of the data of said sensors must occur in order to obtain a velocity signal (first integration) or a location signal (second integration) of the electronic pen.

Small errors in the measurements of acceleration and / or angular velocities by the inertial measurement sensor mechanism of the electronic pen can with the first integration lead to greater errors when determining the velocity, which in turn can lead to even greater errors in the location signal after integration of the velocity signal.

Possible sources of error can in addition to inherent inaccuracies of numerical integration methods include, inter alia, zero point errors and non-linearities of the sensors, inaccuracies of analogue-to-digital conversions of measuring sensor signals, random interference or noise components inherent to the system. For example, errors in determining the position of the electronic pen in space as well as errors in accounting for the acceleration due to gravity lead to partly significant errors in the acceleration signal of the electronic pen, where the acceleration signal can, for example, be smaller than the acceleration due to gravity by about one to two orders of magnitude.

As, for example, a new position of the electronic pen is determined on the basis of a previously determined position, errors in determining the velocity and determining the location of the electronic pen can further accumulate and in an undesirable manner lead to a so-called drift of the motion signal or the location signal of the electronic pen.

In other words: When detecting and evaluating the motion of an electronic pen is based solely on the signals from the inertial sensors and, for example, a user after the lapse of several seconds and / or a series of other movements again moves to a point on a writing substrate that he was previously at, then an error in determining the location signal of the electronic pen is to be expected due to said drift. This can inter alia lead to problems in character recognition when a user wants to correct a previously written character or a previously written text, for example, wants to correct a T-bar and / or wants to set punctuation marks.

Task

It is an object of the invention to improve an electronic pen, in particular in terms of the accuracy with which the motions and positions of the electronic pen on a writing substrate can be detected.

Solution

This is achieved according to the invention by an electronic pen according to claim 1 , a method according to claim 10, and a system according to claim 17. Advantageous embodiments and further developments are the subject matter of the dependent claims.

An electronic pen according to the invention with pen position detection can there comprise a writing rod, at least one electrical power source, at least one digital processing unit, at least one data transmission module, and an inertial positioning sensor system for detecting the position and motion of the electronic pen, and comprise at least one optical sensor being configured such that it can capture or record and store at least one image / image data of at least one surrounding area of the writing rod tip. The digital processing unit of the electronic pen can thereby be configured such that a mapping of a writing substrate can be created from said recorded image data, and be configured such that it can perform absolute referencing of a first location signal of the electronic pen determined from the inertial positioning sensor system by referencing the determined location signal of the positioning sensor system with a second location signal determined by use of the optical sensor, where said second location signal can be determined from the comparison of stationary features in the surrounding area of the writing rod tip identified in the image data of the current surrounding area of the writing rod tip with stationary features identified in image data of at least one already stored surrounding area of the writing rod tip and / or identified in a mapping of the writing substrate.

The term absolute referencing or referencing is in this context to be understood, inter alia, as determining an absolute location of the electronic pen or determining an absolute location of the writing rod tip of the electronic pen respectively, on or relative to a two-dimensional writing substrate surface, for example, paper.

The surrounding area of the writing rod tip can in particular be understood as a region of the writing substrate which surrounds the writing rod tip when the writing rod tip contacts the writing substrate. Stationary features in a surrounding area of the writing rod tip serving as reference points, which can be identified in the image data of said writing rod tip surrounding area can be, inter alia, for example, stationary features on a writing substrate, for example, edges or corners of the writing substrate and / or marks on the writing substrate and / or characters and / or letters and / or patterns and / or microscopic details previously written with the electronic pen on the writing substrate such as the local writing substrate fine structure, for example, granularity, texture, embossing, or patterns of a paper structure.

An image or a recorded or stored image, respectively, is further to be understood to be a two- dimensional image which can be digitally stored as a two-dimensional image or as two- dimensional image data.

Images and image data or image data can hereinafter also be referred to as recordings in general or in particular as orientation recordings or mapping recordings or as mapping data or map sections.

It is for the sake of completeness to be mentioned that said image data can in particular also contain metadata regarding the point in time of recording the image of the surrounding area of the writing rod tip. For storing image data or image data, respectively, it can be advantageous to use a lossless format such as PPM, TIFF or BMP since the loss of detail of lossy compression methods would inevitably lead to a loss of accuracy in absolute referencing.

Recording and storing image data of a surrounding area of the writing rod tip can in particular comprise the option of recording and storing a plurality or a time series of images or image data, whereby a mapping of the writing substrate consisting of a combination of images or image data, respectively, can advantageously be created.

It can thereby be distinguished between a pure mapping of the writing substrate by a series of images or image data for mapping, so-called mapping recordings, and optical absolute referencing of a location signal of the pen, determined by a series of images or image data for determining the position of the pen, so-called orientation recordings.

Said optical absolute referencing of the location signal of the pen can there e.g. be performed in that a current orientation recording is compared with a previously made stored orientation recording or with previous mapping recordings or mapping sections of a mapping created and stored.

Said comparison for optical absolute referencing of the location signal of the pen, i.e. for determining the position of the pen from the image data of the optical sensor, can in particular comprise a comparison of stationary features of the surrounding area of the writing rod tip, identified in the current orientation recording, with stationary features, identified in image data of at least one already stored surrounding area of the writing rod tip, previous orientation recordings, and / or identified in a mapping of the writing substrate.

Mapping of the writing substrate and determining the position of the pen can preferably be performed in parallel, where, for example, current orientation recordings or parts of current orientation recordings can at the same time be added to a mapping. An evaluating logic of the electronic pen or an external data evaluation unit can also in steps browse last recorded image data, e.g. previous mapping or orientation recordings as well as map sections of a mapping created, for comparison with a current orientation recording or a current mapping recording.

If, for example, the user jumps to a previously written text passage, an evaluation logic can therefore, by step-wise browsing through the last recorded image data or browsing a mapping created, identify said previously written text passage and thereby enable absolute referencing of the location signal of the electronic pen.

For relative allocation, creation of a mapping, and for comparing recorded image data, such as orientation or mapping recordings, it is advantageous to have the image data overlap.

After successful relative allocation and alignment of image data, however, overlapping image data can be deleted.

This would reduce storage space and reduce the search effort for stationary features to be identified for absolute referencing of a location signal of the electronic pen determined from the image data of the optical sensor.

In order to be able to create a sufficiently accurate mapping of the writing substrate, in particular where, for example, the writing itself or the fine structure of the writing substrate are to serve as reference points or stationary features, respectively, it is advantageous if two pieces of successive imaging data have an overlap that allows unambiguous allocation or alignment of the image data.

This can be ensured, for example, in that according to the Nyquist-Shannon sampling theorem, the time interval between two pieces of successively recorded image data or images is smaller than half of the time interval arising when one divides the dimension of the image in the direction of the writing motion by the velocity with which the electronic pen is moved.

It can beyond that be useful to include further additional images or image data for aligning the image data, for example, in particular when the user has begun to write in a corner of the writing substrate and / or with the writing hand and / or his forearm covers essential parts of the writing substrate edge at the time of the first or a previous recording of an image. It has further been found that for acceptable optical absolute referencing of the images or the location signal of the pen it can suffice to select e.g. the frequency with which images, in particular orientation recordings or mapping recordings, can be recorded, according to a maximum tolerable drift of the location signal of the pen.

If the maximum tolerable drift is, for example, 1 mm/s², and the desired accuracy of the location signal maximum 1 mm, the images should be recorded by the optical sensor, or recorded and evaluated, at a frequency of at least approx.1 image per second.

An electronic pen according to the invention can therefore by means of recording and storing image data of at least one surrounding area of the writing rod tip by use of an optical sensor advantageously determine a referenced location signal of the pen that is absolute with respect to the writing substrate.

Absolute referencing of a location signal determined from the inertial positioning sensor system can be performed by use of this optically determined location signal and a possible error in a location signal determined from the inertial positioning sensor system of the electronic pen can thereby be advantageously compensated, which, for example, can arise due to integration errors of data of the inertial positioning sensor system and can cause unwanted drift in the location signal of the electronic pen determined from the positioning sensor system.

For example, if a user wants to correct an already written character or an already written text, for example, wants to correct a T-bar and / or wants to set punctuation marks, the location signal being determined from the inertial positioning sensor system can, when again moving to an already visited surrounding area or an already visited location, be corrected or calibrated relative to an absolute position on the writing substrate, in that the currently recorded image data of the surrounding area of the writing rod tip is compared with previously stored image data of the writing rod tip surrounding areas on the writing substrate.

If, namely, said currently recorded surrounding areas image data can within a predetermined tolerance be matched or put in congruence with already stored image data based on stationary features, then the current position signal of the electronic pen can be newly absolutely referenced or calibrated.

Any inaccuracies in the location signal determined from said inertial positioning sensor system possibly attributable to drift errors of the inertial positioning sensor system can be corrected.

The electronic pen can advantageously also comprise at least one optical element for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor. This allows for greater freedom in the choice of the arrangement of the optical sensor and/or optimization of the size of the imaged and recordable surrounding area of the writing rod tip, or optimization of the solid angle at which the surrounding area of the writing rod tip to be imaged and recorded can be imaged and recorded.

The size of the surrounding area of the writing rod tip that can be imaged and recorded in one recording of an optical sensor can, for example, correspond to at least 10% of the size of the writing substrate.

The size of the surrounding area of the writing rod tip that can be imaged and recorded in one recording can advantageously correspond to, for example, the size of the entire writing substrate, e.g. a writing substrate of the size 8.5 x 1 1 inches (215.9 mm x 279.4 mm) or an A4 format.

Said optical element for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor can, for example, be disposed at the end opposite to the writing rod tip of the electronic pen.

It is also conceivable that the optical sensor can be disposed near the writing rod tip at the end of the pen where the writing rod tip can exit.

It is also conceivable to employ two or more optical sensors at different positions of the pen, for example, a first optical sensor at the pen end opposite to the writing rod tip and a second sensor near the writing rod tip at the beginning of the pen

The recorded image data, i.e. the recordings from several optical sensors can then advantageously be combined in order to be able to contribute to a more accurate and more complete visual recording of the surrounding area of the writing rod tip.

Said captured image data from a plurality of optical sensors can thereby be complementing and/or overlapping.

A surface of the optical element facing the writing rod tip can thereby comprise at least one reflective surface or at least one non-reflective surface region and a reflective surface region.

The optical element can be a spherical mirror and comprise a conical or a concave-shaped reflective surface or surface region facing the writing rod tip.

The optical element can further be mounted movably and be rotatable and/or pivotable about an axis equal or parallel or not parallel to the longitudinal axis of the electronic pen. Tilting and/or pendulum and/or wobbling motions of the optical element can thereby advantageously be realized for optimizing optical scanning of the surrounding area of the writing rod tip to be imaged.

Translations along the longitudinal axis of the electronic pen or along an axis parallel to said longitudinal axis of the pen are also conceivable for further optimization of the beam path for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor.

The optical sensor for recording and storing an image of the surrounding area of the writing rod tip can be a matrix sensor or a line sensor or a point sensor and, for example, be a Charge- Coupled Device (CCD) sensor.

In addition, the optical sensor can be attached along or parallel to the longitudinal axis of the electronic pen. The optical element for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor can also be attached along or parallel to the longitudinal axis of the electronic pen.

It is also possible to dispose a second optical element, for example, a lens, between the optical sensor and the optical element for deflecting an image of the surrounding area of the writing rod tip, which can advantageously be used to improve the focus of the image of the surrounding area of the writing rod tip on the optical sensor. This can optimize making use of sensor area and sensitivity of the optical sensor, and the installation space of the arrangement can be shortened.

In a method according to the invention for detecting pen positions of an electronic pen comprising a writing rod, at least one electrical power source, at least one digital processing unit, at least one data transmission module, and an inertial positioning sensor system for detecting the position and motion of the electronic pen, and comprising at least one optical sensor, the optical sensor can therefore image at least one surrounding area of the writing rod tip of the electronic pen and store it as picture data.

The digital processing unit can create a mapping of the writing substrate, e.g. based on the image data recorded by the optical sensor.

In addition, absolute referencing of the location signal determined from the inertial positioning sensor system with a second location signal determined by the optical senor (198, 298) can in said method occur based on the stored data of the optical sensor, where said second location signal can be determined from the comparison of stationary features of the surrounding area of the writing rod tip, identified in the image data of the current surrounding area of the writing rod tip, with stationary features identified in image data from at least one already stored surrounding area of the writing rod tip and/or identified in a mapping of the writing substrate.

Said mapping of the writing substrate can be created, for example, by a digital processing unit appropriately configured for this, based on the image data recorded by the optical sensor, and can iteratively comprise the following exemplary steps:

recording a first mapping recording,

recording a second mapping recording,

examining and determining whether said two mapping recordings overlap,

in the event of determining partial overlap:

adding the non-overlapping region / the non-overlapping regions of the mapping recordings to a map or, if a map does not yet exist, determine them as a first map section /as first map sections,

in the event of determining that said two mapping recordings do no overlap:

reverse search for previous mapping recordings or map sections which provide an overlap of the mapping recordings(s) to be examined with a stored previous mapping recording or a stored previous map section, where

in the event of a successful search, the non-overlapping region of the mapping recording(s) to be examined and the encountered stored map recording / the encountered stored map sections can again be added to a map (304), and where in the event of an unsuccessful search, a next or new first mapping recording can occur, in the event of determining that said two mapping recordings do no overlap:

recording a next mapping recording,

next (312) examination and determination (303) of whether said next mapping recording (305) overlaps with a previous mapping recording.

Determining an optically referenced location signal of the electronic pen, for example, of a second location signal determined by an optical sensor, for absolute referencing of a first location signal determined from the inertial positioning sensor system of the electronic pen, can further comprise the following steps:

recording a current orientation recording by the optical sensor (109, 209), comparison of said current orientation recording with previously stored image data, for example, with a stored previous orientation recording and/or mapping recording, or map sections of a mapping, with examining and determining whether said current orientation recording overlaps with the image date previously stored,

in the event of determining partial overlap or entire overlap:

possible addition of the non-overlapping region/non-overlapping regions to a map or a mapping, determining the position of the electronic pen in relation to the writing substrate by identifying stationary features contained in previously stored image data, e.g. in a previously stored orientation recording overlapping the current orientation recording or contained in a previously stored map section overlapping the current orientation recording.

in the event of determining that no overlap is given:

reverse search in stored image data, e.g. in stored mapping recordings or in stored map sections, or previously stored orientation recordings, where

in the event of a successful search, a position of the electronic pen is again determined in relation to the writing substrate,

after determining the position (404), a recording of a next orientation recording occurs.

For advantageous simplification of the computational complexity of currently recorded image data of the surrounding area of the writing rod tip with previously stored image data of writing rod tip surrounding areas, for example, only image data of the optical sensor can be used that exhibits a contrast greater than a minimum contrast threshold.

The hand of the writer, i.e., the user of the electronic pen can in an image of the surrounding area recorded by the optical sensor create a region of lesser brightness contrast (hereinafter referred to simply as contrast) than, for example, the contrast between what is written and the writing substrate, which is unsuitable for absolute referencing of the location signal of the electronic pen because the hand and therefore said region with lesser contrast moves together with the electronic pen.

It can therefore advantageously suffice to store or record only image data with sufficient contrast.

For example, a contrast threshold between the writing substrate color, e.g. white, and the writing color of the electronic pen, e.g. blue or black, can be determined at the beginning when using the electronic pen, and an initial calibration of the writing substrate color and/or the ink color can be performed, for example, by use of a first initial recording of the writing substrate by an optical sensor of the electronic pen.

When lighting conditions change, the contrast threshold can possibly be adjusted and/or a recalibration of writing substrate color and/or the writing color can be performed.

Referencing the determined location signal of the inertial positioning sensor system with a second location signal determined from the comparison of stationary features of the surrounding area of the writing rod tip identified in the image data of the current surrounding of the writing rod tip, with stationary features identified in image data of at least one already stored surrounding area of the writing rod tip, can thereby be performed with a plurality of derived image data, where said derived image data can arise from the transformation of stored imaging or image data of the optical sensor.

These transformations can comprise rotation and/or translation and/or stretching and/or compression of the stored image data.

For an advantageous increase in the degree of efficiency with which said transformations of the stored image data can be performed, performing said transformations can be limited to, for example, non-overlapping stored image data.

It can thereby be advantageously avoided that rotations about the longitudinal axis of the electronic pen and/or changes in the angle of inclination between the longitudinal axis of the electronic pen and the writing substrate surface, as they can naturally occur during writing motions of a user with the electronic pen, do not crucially impede the comparison or recognition of same patterns or same stationary features.

For this, for example, the image data can in Cartesian coordinates be multiplied with a two- dimensional 2 x 2 transformation matrix in order to be able to obtain a rotated version of the image data of at least one already stored surrounding area of the writing rod tip and/or a rotated version of the image data of a currently recorded surround area of the writing rod tip.

Any distortion of the image of a surrounding area of the writing rod tip due to a change of the angle of inclination between the longitudinal axis of the electronic pen and the writing substrate surface can be compensated inter alia by numerical means, such as the conversion of Cartesian coordinates of the image data to barycentric coordinates.

An advantageous simplification of the computational complexity for compensating for rotational and/or distortion effects of the image data of the writing rod tip surrounding area due to the rotations about the longitudinal axis of the electronic pen and/or changes in the angle of inclination between the longitudinal axis of the electronic pen and the writing substrate surface can be achieved by determining and storing the image data with a radial scan method.

Concentric circles of image data points can thereby be sequentially imported and stored.

Said concentric circles are there from the inside out always further apart from each other so that, for example, relative to the axis in which the longitudinal axis of the optical sensor is located, they are located in equally wide angle segments.

A rotation can then be simply calculated by choosing a different starting point on the circumference of the scan, and any distortion can be easily allowed for in that one compares groups of pie segments of an equal amount of concentric circles.

When comparing the currently recorded surrounding area image data with already stored surrounding area image data by use of stationary features, required values for a rotation and/or a distortion or an equalization of image data can therefore with e.g. said calculation methods be determined within predetermined tolerance values in order to identify a match of currently recorded surrounding area image data with already stored surrounding area image data based on stationary features.

This numerical comparison of image data as well as the transformation of image data can, for example, be performed by a digital processing unit of the electronic pen.

A system according to the invention for the electronic detection of pen positions can comprise an electronic pen with all or some of the above-described features. Furthermore, said system can comprise at least one data reception module for receiving data transmitted by a data transmission module of the electronic pen, and of a data evaluation unit for evaluating and processing the data received, and a data output unit and a data storage unit.

The system can thereby be configured such that either the data evaluation unit for evaluating and processing data received or a digital processing unit of the electronic pen can, based on the data of the optical sensor, create a mapping of the writing substrate from said recorded image data and perform absolute referencing of data of the inertial positioning sensor system, in particular absolute referencing of a first location signal of the electronic pen determined from the inertial positioning sensor system, based on referencing a first location signal of the electronic pen determined from the inertial positioning sensor system, with a second location signal determined from the comparison of stationary features of the surrounding area of the writing rod tip identified in the image data of the current surrounding area of the writing rod tip, with stationary features identified in image data from at least one already stored surrounding area of the writing rod tip. It should be noted at this point that the term "region" can in particular be understood as being a region of a picture, where it should for the sake of completeness be mentioned that all image data presently discussed, in particular mapping recordings, mappings, maps, orientation recordings can be interpreted as being pictures.

The following figures by way of example illustrate

Fig. 1 a: a lateral view of an end portion of an electronic pen

Fig. 1 b: a lateral view of an alternative end portion of an electronic pen

Fig. 2a: an exemplary method for mapping the writing substrate

Fig. 2b: an exemplary method for absolute referencing

Fig. 3a: an exemplary perspective of an electronic pen on a writing substrate

Fig. 3b: an exemplary alternative perspective of an electronic pen on a writing substrate

Fig. 4a: an exemplary scanning or sampling pattern

Fig. 4b: further exemplary scanning or sampling patterns

Figs. 1a exemplifies a lateral view of an end portion 100 of an electronic pen or a partial view of an electronic pen, respectively, which can exhibit all or only some of the features described above.

For example, an optical sensor 109 with sensor casing 101 can be disposed along the longitudinal axis 105 of the electronic pen and in a first casing end portion 107 of the electronic pen.

An optical element 102 also movably attachable along the longitudinal axis 105 of the electronic pen, that can be centered relative to the longitudinal axis 105, can image a surrounding area of the writing rod tip (not shown) onto optical sensor 109, where the image of the surrounding area can be effected e.g. along the lines of sight 103, 104.

Advantageously, optical element 102 can be embodied as a spherical mirror 102 and comprise a conically shaped reflective surface 108 that faces the writing rod tip (not shown) and which, for example, can project a conical region of a surrounding area of the writing rod tip onto optical sensor 109. Optical element 102 can thereby be accommodated in a second casing end portion 106 of the electronic pen, wherein said second casing end portion 106 is transparent in order be able to guide the light from the writing rod tip surrounding area to be imaged onto optical sensor 109.

Fig. 1 b by way of example also shows a lateral view of an alternative end portion 200 of an electronic pen and represents a partial view of an alternative electronic pen, respectively, where in contrast to the example of Fig. 1 b, reflective surface 208 of the optical element facing the writing rod tip (not shown) is for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor 209 configured as a concave mirror.

Although the surrounding areas nearest to the writing rod tip or possible stationary features of the writing rod tip to be imaged are in this exemplary geometry of optical element 202 projected in the outer region of an image of the writing rod tip deflected onto the optical sensor, this, however, has no adverse effects on the quality of the optical absolute referencing according to the invention of the location signal of the electronic pen, but can advantageously lead to improved illumination of the surrounding area nearest to the writing rod tip.

Fig. 2a exemplifies an iterative method 300 for mapping a writing substrate.

Two temporally successive mapping recordings or images of a writing rod tip surrounding area recorded by an optical sensor of the electronic pen could, for example, thereby be recorded and stored.

After recording a first mapping recording 301 and a second mapping recording 302, it can be examined 303 by evaluating logic, for example, by a digital processing unit of the pen or an external data processing unit, whether said mapping recordings overlap.

Upon detection of a partial overlap 307 of successive mapping recordings, said mapping recordings 301 , 302 can be attached to each other and/or the non-overlapping region/ the non- overlapping regions of the mapping recordings 301 , 302 can be added to a map or mapping 304 or, if a map does not yet exist, be the basis for a map, for example, be defined as the first map section / first map sections.

Non-overlapping regions of mapping recordings 301 , 302, which are combined to form a map, can also be referred to as a map sections.

A next mapping recording 305 can subsequently be recorded and an examination of the overlap to a previous mapping recording can occur. In the event of a complete overlap of mapping recordings, step 304 of attaching the non- overlapping regions to a map can be skipped and a new mapping recording 305 can be directly initiated.

In the event that no overlap 308 of two successive mapping recordings can be detected, for example, a reverse search can be performed for previous mapping recordings or map sections 306 which can result in an overlap of the mapping recording(s) to be examined or the currently recorded mapping recording with a stored previous mapping recording or a stored previous map section.

If the search 306 in the stored previous mapping recordings or map section was successful 310, i.e. if determination of overlap or the examination for overlap 303 leads to determining an overlap of a mapping recording with a previous mapping recording or a previous map section, then the non-overlapping region of current mapping recording 302 and the encountered stored mapping recording / the encountered stored map section can again be attached to a map 304.

If no overlap can be determined also in the already existing stored map sections or mapping recordings, then, for example, a new mapping recording can be initiated which can be a first 301 or a subsequent mapping recording 305.

Fig. 2b represents an exemplary iterative method 400 for absolute referencing the position or a location signal of a writing rod tip of an electronic pen determined from the image data of an optical sensor of the pen.

A first or a current orientation recording 401 , i.e. an image of a surrounding area of the writing rod tip created by an optical sensor of the electronic pen, can there first be recorded and stored.

Said current orientation recording 401 can then be compared 402 with previous stored image data, for example, with a stored previous orientation recording, and/or with a stored mapping recording 301 , 302, 305, 306 or with a stored mapping recording or a stored map section of a mapping of the writing substrate.

Said comparison 402 can comprise an examination determining whether said current orientation recording 401 overlaps with previous stored image data, i.e. with stored previous orientation recordings, and/or with a stored mapping recording 301 , 302, 305, 306 or with a stored mapping or a stored map section of a mapping of the writing substrate.

In the event of partial overlap, for example, the non-overlapping region can be added 403 to a map. In other words, the methods for absolute referencing the location signal of electronic pen 400 and for mapping a writing substrate 300 can be performed in parallel, or simultaneously in a complementary manner, respectively.

If in step 407, for example, no overlap with a previous orientation recording is detected, then a reverse search 408 in stored mapping recordings 301 , 302, 305, 306, or in a stored mapping or in stored map sections of a mapping can occur. If search 406 is successful, i.e. if at least partial overlap can again be detected by a comparison 402, then the non-overlapping region can be added to a map.

In the event of a detected partial or complete overlap 409 of a current orientation recording 401 with said previous stored recordings or maps sections, absolute determination of the position 404 of the electronic pen relative to the writing substrate can occur, for example, by identification of stationary features contained in a previous stored orientation recording 402 overlapping the current orientation recording 401 or contained in a pervious stored map section overlapping the current orientation recording 401.

In other words, referencing the location of the pen can according to the invention occur purely optically, which in turn can be compared with further referencing of the location, for example, determined from the inertial positioning sensor system of the electronic pen, in order to thus ultimately be able to obtain improved absolute referencing of the location signal of the electronic pen.

Determining the position 404 of the location of the writing rod tip or the electronic pen can therefore comprise absolute referencing of a location signal of the electronic pen determined from the inertial positioning sensor by referencing said determined location signal of the inertial positioning sensor with a second location signal.

Said second location signal can thereby be determined from the comparison of stationary features of the surrounding area of the writing rod tip identified in the image data of the current surrounding area of the writing rod tip, i.e. identified in the current orientation recording 401 , with stationary features identified in image data from at least one already stored surrounding area of the writing rod tip, i.e. for example, identified in a stored previous orientation recording and/or identified in a stored mapping recording 301 , 302, 305, 306 or a stored mapping or identified in a stored map section of a mapping of the writing substrate, respectively.

Said stationary features can thereby, as already mentioned, be stationary features on a writing substrate, such as edges or corners of the writing substrate and / or marks on the writing substrate and / or characters and / or letters and / or patterns and / or microscopic details previously written with the electronic pen on the writing substrate such as the local writing substrate fine structure, for example, granularity, texture, embossing, or patterns of a paper structure.

After successfully determining the position 404 of the location of the writing rod tip or the electronic pen, respectively, a further orientation recording 405 can immediately occur.

In the event of an unsuccessful search 410 for stored image data overlapping a current orientation recording, optically determining the position 404 may not be possible 412.

In this case, it may be necessary to interrupt the writing process and perform e.g. a series of recordings with the optical sensor at different writing substrate positions of the writing rod tip for creating a first mapping, as described in Fig. 2a.

A location signal determined from an inertial positioning sensor of the pen can therefore with this optical absolute referencing of the position of the pen on the writing substrate advantageously also be absolutely referenced, and inter alia a drift of the motion signal or the location signal determined from the inertial positioning sensor can be corrected and compensated.

Fig. 3 illustrates an exemplary perspective of an electronic pen 500 on a writing substrate 505.

Axis 501 therein represents both the longitudinal axis of pen 500 or the writing rod as well as the optical axis of an optical sensor accommodated in pen end 513.

Contours or contour lines 507, 508, 509, 510 by way of example mark a current surrounding area 506 of the writing rod tip recordable by the optical sensor, i.e. a most current orientation or mapping recording, where, starting out from the writing rod tip as the origin, contour 510 can define the recordable surrounding area 506 from the inside and contour 506 the surrounding area 506 from the outside.

The current recordable surrounding area 506, in addition to part of an already written or already previously existing text 503, by way of example also comprises part 51 1 of a writing substrate edge 512 which can be considered to be stationary features of writing substrate 505.

If therefore, as described above, for example, a mapping of the writing substrate, a map section or an already previously taken mapping or orientation recording of the optical sensor has been stored, in which a stationary feature is contained, which again can be encountered partially or entirely in the current recordable surrounding area 506, and can therefore be recorded in a current orientation recording, then, as also described above, position 513 of writing rod tip 502 on the writing substrate can be absolutely referenced relative to writing substrate 505 (which, as presently shown, for example, is located at the foot of a letter T 504, and where the letter T 504, i.e. the currently written text, can itself serve as a stationary feature).

Optical referencing of the position 513 of pen 500 on the writing substrate 505 thus obtained can then, as mentioned, further serve for absolute referencing a location signal (not shown) of electronic pen 500 determined from the inertial positioning sensor system.

Fig. 3b by way of example shows a scenario in analogy to Fig. 3a from a lateral view, wherein an exemplary beam path 604 for recording orientation and/or mapping image data is illustrated.

Electronic pen 600 is shown as it, for example, just writes a text 607, for example, a letter T on writing substrate 608, where said text 607 can be regarded as a stationary feature.

A recordable surrounding area 610 of the writing rod tip, specified by way of example by the beam geometry of beam path 604, can thereby via an optical element 602 be deflected onto an optical sensor 601 , for example, accommodated in pen end 603, thereby enabling an orientation and/or mapping recording by use of optical sensor 601.

Optical sensor 601 is thereby for example arranged along the longitudinal axis of electronic pen 600.

Surrounding area 610 of writing rod tip 606 on the writing substrate illustrated by way of example includes, for example, part of an already written text 61 1 which can be regarded as a stationary feature on the writing substrate.

As described above, this stationary feature can serve as a reference point for comparisons with current orientation recordings, which, for example, also partially or completely contain this stationary feature, whereby position 605 of writing rod tip 606 or electronic pen 600 is absolutely referenced with respect to writing substrate 608.

Figure Fig. 4a by way of example illustrates scan or sampling patterns 700 with which recordings data, e.g. orientation and/or mapping recordings or surrounding areas of the writing rod tip, can by the optical sensor be recorded and processed.

It is for reasons of simplicity assumed in the figure that the optical axis is perpendicular to the plane of the figure.

Sampling pattern 700 by way of example represents a sampling pattern for a radial scan method with radial sampling contours extending around origin 704, for example, the writing rod tip. The innermost radial contour 705 and the outermost radial contour 706 are marked by way of example. The distance of the innermost radial contour from the origin can advantageously be as small as possible. While said distance can in an optical sensor disposed in the pen end be limited by the average diameter of the electronic pen, smaller distances can be realized by disposing the optical sensor near the writing rod tip.

By selecting different starting points 701 , 702 of the radial scan method, e.g. a rotational angle 703 can be defined with which, for example, simplification of the computational complexity for compensating for rotational and/or distortion effects of the image data of the writing rod tip surrounding area due to the rotations about the longitudinal axis of the electronic pen and/or changes in the angle of inclination between the longitudinal axis of the electronic pen and the writing substrate surface can be achieved.

Figure Fig. 4b by way of example represents a sampling pattern 800 alternative to scan or sampling pattern 700, where, for example, the optical axis is also perpendicular to the plane of the figure and, for example, the writing rod tip is located in origin 804.

Scan pattern 800 is an exemplary pattern for a cycloidal scan or sampling method which would also lead to simplification of the computational complexity for compensation for rotational and/or distortion effects of the image data of the writing rod tip surrounding area, and would thereby enable more efficient analysis and evaluation of said image data.

As with scan 700, a rotational angle 803 can be defined by selecting different starting points 801 , 802 of the radial scan method.

7 Pages with 8 Figures appended

The reference numerals are there allocated as follows:

100 End portion of an electronic pen / partial view of an electronic pen / pen end

101 Casing of the optical sensor

102 (First) optical element for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor

103 Line of sight

104 Line of sight

105 Longitudinal axis of the electronic pen

106 Second casing end portion of the electronic pen

107 First casing end portion of the electronic pen

108 The reflective surface of the optical element facing the writing rod tip (not shown) for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor

109 Optical sensor

200 End portion of an electronic pen / partial view of an electronic pen / pen end

201 Casing of the optical sensor

202 (First) optical element for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor

203 Line of sight

204 Line of sight

205 Longitudinal axis of the electronic pen

206 Second casing end portion of the electronic pen

207 First casing end portion of the electronic pen

208 The reflective surface of the optical element facing the writing rod tip (not shown) for deflecting an image of the surrounding area of the writing rod tip onto the optical sensor 209 Optical sensor

300 Method for mapping a writing substrate

301 First mapping recording

302 Second mapping recording

303 Examination/determination of an overlap of mapping recordings

304 Attaching non-overlapping regions of the mapping recordings to a map

305 Next mapping recording

306 Reverse search through previous / stored map segments or mapping recordings

307 Determination of partial overlap of successive mapping recordings

308 Determination that successive mapping recordings do not overlap

309 Determination of a complete overlap of successive mapping recordings

310 Re-examination/determination of an overlap of mapping recordings after successful reverse search through previous / stored map segments or mapping recordings

311 New recording of a first mapping recording in the event of non-successful reverse search through previous / stored map segments or mapping recordings

312 Next examination / determination of an overlap of mapping recordings

313 Next recording of a first mapping recording in the event of non-successful reverse

search through previous / stored map segments or mapping recordings

400 Optical absolute referencing of the position or a location signal of an electronic pen or the writing rod tip of an electronic pen, respectively

401 Current orientation recording

402 Comparison with a map, a map section, a mapping recording or a previous orientation recording

403 Attaching the non-overlapping region / non-overlapping regions to a map

404 Determination of the positioning 404 of the electronic pen in relation to the writing substrate / optical referencing absolute relative to the writing substrate, referencing the position or a location signal of an electronic pen or the writing rod tip of an electronic pen, respectively

405 Next orientation recording 406 Reverse search through previous / stored map segments or mapping recordings or orientation recording

407 Determination of partial overlap of successive orientation recordings

408 Determination that successive orientation recordings do not overlap

409 Determination of complete overlap of successive orientation recordings

410 Non-successful reverse search

411 Successful reverse search for regions from previous / stored map sections or mapping recordings or orientation recordings overlapping current orientation recordings

412 No possibility of determining the position 404 by optical referencing possible

413 Next orientation recording

500 Electronic pen / perspective by way of example of an electronic pen on a

writing substrate

501 Longitudinal axis of the electronic pen, optical axis

502 Writing rod tip

503 Exemplary stationary feature, exemplary previously written / already existing text

504 Currently written text, stationary feature

505 Writing substrate

506 Exemplary current surrounding area of the writing rod tip recordable by an optical sensor, exemplary current orientation or mapping recording

507 Contour / contour line of surrounding area 506, outermost contour of surrounding area 506

508 Contour / contour of surrounding area 506

509 Contour / contour of surrounding area 506

510 Contour / contour line of surrounding area 506, innermost contour of surrounding area 506

511 Part of writing substrate edge 512 of writing substrate 505

512 Writing substrate edge 512 of writing substrate 505

513 Current position of the writing rod tip on the writing substrate 600 Electronic pen

601 Optical sensor

602 Optical element

603 Pen end of the electronic pen

604 Exemplary beam path

605 Position of the writing rod tip

606 Writing rod tip

607 Currently written text, e.g. a letter T, stationary feature

608 Writing substrate

609 Longitudinal axis of the electronic pen, optical axis

610 Current recordable surrounding area of the writing rod tip illustrated by way of example, exemplary current orientation or mapping recording

611 Exemplary stationary feature, exemplary previously written / already existing text

700 Scanning or sampling pattern

701 (First) starting point for scanning / sampling

702 (Second / alternative) starting point for scanning / sampling

703 Rotational angle

704 Origin e.g. position of the writing rod tip

705 Innermost radial contour

706 Outermost radial contour

800 Scanning or sampling pattern

801 (First) starting point for scanning / sampling

802 (Second / alternative) starting point for scanning / sampling

803 Rotational angle

804 Origin e.g. position of the writing rod tip

Claims

1 . Electronic pen (500, 600) with pen position detection, comprising a writing rod, at least one electrical power source, at least one digital processing unit, at least one data transmission module as well as an inertial positioning sensor system for detecting the position and motion of said electronic pen (500, 600), and at least one optical sensor (109, 209),

characterized in that

said optical sensor (109, 209) is configured for recording and storing image data of at least one surrounding area of the writing rod tip (502, 606), and

said digital processing unit is configured for creating a mapping of a writing substrate from said recorded image data, and for absolute referencing of a first location signal of said electronic pen (500, 600) determined from said inertial positioning sensor system by referencing said determined location signal of said inertial positioning sensor system with a second location signal determined by use of said optical sensor (109, 209) determined from the comparison of stationary features of said surrounding area of said writing rod tip (502, 606) identified in the image data of said current surrounding area of said writing rod tip (502, 606) with stationary features identified in image data of at least one already stored surrounding area of said writing rod tip (502, 606) and / or identified in a mapping of said writing substrate (505, 608).

2. Electronic pen (500, 600) according to claim 1 ,

characterized in that

said electronic pen (500, 600) comprises at least one optical element (102, 202) for deflecting an image of said surrounding area of said writing rod tip onto said optical sensor (109, 209).

3. Electronic pen (500, 600) according to claim 2,

characterized in that

said optical element (102, 202) is disposed at the rod end (100, 200) of said electronic pen opposite to said writing rod tip.

4. Electronic pen (500, 600) according to claim 2 or claim 3,

characterized in that the surface (108, 208) of said optical element facing said writing rod tip comprises at least one reflective surface or at least one non-reflective surface region and a reflective surface region.

5. Electronic pen (500, 600) according to claim 4,

characterized in that

said optical element (102, 202) is a spherical mirror and comprises a conical or a concave- shaped reflective surface (208) facing said writing rod tip.

6. Electronic pen (500, 600) according to one of the claims 2 to 5,

characterized in that

said optical element (102, 202) is mounted movably and is rotatable and/or pivotable about an axis equal or parallel or not parallel to the longitudinal axis (105, 205) of said electronic pen.

7. Electronic pen (500, 600) according to one of the preceding claims,

characterized in that

said optical sensor (109, 209) is a matrix sensor or a line sensor or a point sensor.

8. Electronic pen (500, 600) according to one of the preceding claims,

characterized in that

said optical sensor (109, 209) is attached along or parallel to said longitudinal axis (105, 205) of said electronic pen and/or, if present, said optical element (102, 202) for deflecting an image of said surrounding area of said writing rod tip onto said optical sensor (109, 209) is attached along or parallel to said longitudinal axis (105, 205) of said electronic pen.

9. Electronic pen (500, 600) according to one of the preceding claims,

characterized in that

a second optical element, e.g., a lens, is disposed between said optical sensor (109, 209) and said optical element (102, 202) for deflecting an image of said surrounding area of said writing rod tip, for focusing said image of said surrounding area of said writing rod tip onto said optical sensor (109, 209).

10. Method for detecting motion patterns and pen positions of an electronic pen (500, 600) comprising a writing rod, at least one electrical power source, at least one digital processing unit, at least one data transmission module, as well as an inertial positioning sensor system for detecting the position and motion of said electronic pen, and at least one optical sensor, wherein said optical sensor (109, 209) images at least one surrounding area of said writing rod tip of said electronic pen (500, 600) and stores it as image data, and/or

said digital processing unit creates a mapping (300) of said writing substrate based on said image data recorded by said optical sensor (109, 209), and

absolute referencing of said first location signal of said electronic pen determined from said inertial positioning sensor system occurs based on said stored data of said optical sensor (109, 209) by

referencing said second location signal determined from said inertial positioning sensor system with a second location signal determined by use of said optical senor (109, 209), where determining said second optically referenced location signal comprises a comparison of stationary features of said surrounding area of said writing rod tip identified in said image data of said current surrounding area of said writing rod tip with stationary features identified in image data from at least one already stored surrounding area of said writing rod tip and/or identified in a mapping of said writing substrate.

Method according to claim 10, wherein said digital processing unit iteratively creates a mapping (300) of said writing substrate based on said image data recorded by said optical sensor (109, 209) and said mapping (300) comprises the following steps:

recording a first mapping recording (301 ),

recording a second mapping recording (302),

examining and determining (303) whether said two mapping recordings (301 , 302) overlap,

in the event of determining partial overlap (307):

adding the non-overlapping region / the non-overlapping regions of said mapping recordings (301 , 302) to a map or, if a map does not yet exist, determine them as a first map section /as first map sections,

in the event of determining that said two mapping recordings (301 , 302) do not overlap (308):

reverse search (306) for previous mapping recordings or map sections which provide an overlap of said mapping recordings(s) to be examined with a stored previous mapping recording or a stored previous map section, where

in the event of a successful search (310), said non-overlapping region of said mapping recording(s) to be examined and said encountered stored map recording / the encountered stored map sections can again be added to a map (304), and where in the event of an unsuccessful search (313, 31 1 ), a next or new first mapping recording can occur,

in the event of determining that said two mapping recordings (301 , 302) do no overlap (309):

recording a next mapping recording (305),

next (312) examination and determination (303) whether said next mapping recording (305) overlaps with a previous mapping recording.

12. Method according to claim 10 or 1 1 , wherein said determination of said second optically referenced location signal comprises the following steps:

recording a current orientation recording (401 ) by said optical sensor (109, 209), comparison of said current orientation recording (402) with previously stored image data, e.g. with a stored previous orientation recording or map with examining and determining whether said current orientation recording (402) overlaps said image data previously stored,

in the event of determining partial overlap (407) or entire overlap (409):

possible addition of said non-overlapping region / non-overlapping regions to a map, determination of the position (404) of said electronic pen (500, 600) in relation to said writing substrate by identifying stationary features contained in a previously stored orientation recording overlapping said current orientation recording (401 ) or contained in a previously stored map section overlapping said current orientation recording (401 ). in the event of determining that no overlap (408) is given:

reverse search (406) in stored image data, e.g. in stored mapping recordings (301 , 302, 305, 306) or in stored map sections, or previously stored orientation recordings, where in the event of a successful search (410), a position of said electronic pen is again determined (404) in relation to said writing substrate,

13. Method according to one of the claims 10 to 12, where orientation recordings and/or mapping recordings are recorded with a frequency of at least approx. 1 picture per second.

14. Method according to one of the claims 10 to 13, where only such image data of said optical sensor (109, 209) is used that exhibits a contrast greater than a minimum contrast threshold.

15. Method according to one of the claims 10 to 14, where referencing said determined location signal of said inertial positioning sensor system with a second location signal determined from the comparison of stationary features of said surrounding area of said writing rod tip identified in said image data of said current surrounding of said writing rod tip with stationary features identified in image data of at least one already stored surrounding area of said writing rod tip is performed with a plurality of derived image data, and wherein

said derived image data arises from the transformation of stored image data of said optical sensor and said transformations comprise rotation and/or translation and/or stretching and/or compression of said stored image data.

16. Method according to one of the claims 10 to 15, where stationary features of said surrounding area of said writing rod tip are stationary features on a writing substrate and edges of said writing substrate and/or markings on said writing substrate and/or characters and/or letters and/or patterns previously written with said electronic pen on said writing substrate.

17. System for electronic detection of pen positions, comprising an electronic pen (500, 600) according to one of the claims 1 to 9, at least one data reception module for receiving data transmitted by said data transmission module of said electronic pen, a data evaluation unit for evaluating and processing said data received, a data output unit and a data storage unit,

characterized in that

said system is configured such that either said data evaluation unit for evaluating and processing data received or said digital processing unit based on said data of said optical sensor can create a mapping of said writing substrate from said recorded image data and can perform absolute referencing of a first location signal of said electronic pen (500, 600) determined from said inertial positioning sensor system, based on referencing said first location signal of said electronic pen with a second location signal determined by use of said optical sensor (109, 209) determined from the comparison of stationary features of said surrounding area of said writing rod tip identified in said image data of said current surrounding area of said writing rod tip with stationary features identified in image data from at least one already stored surrounding area of said writing rod tip and/or identified in a mapping of said writing substrate.