DE102004061842B4 - Tracking system for mobile applications - Google Patents

Abstract

A method for finding the location and / or orientation of mobile devices with respect to real objects, wherein - the mobile device (400) comprises a camera (400-1) and a component is located on the mobile device, called Observer (400-2) which can examine an image of the camera, and - via a wireless network (200), the mobile device (400) receives measurement instructions (300) from an estimator (500-1) located on a server (500), - on the basis of the measuring instructions of Observer (400-2) makes measurements and sends measurement data (600) to the estimator, wherein the estimator uses the measurement data to calculate a current tracking result (700) and sends this to a graphics system, called renderer (400-3), of the mobile device sends, - by means of the tracking result of the renderer (400-3) represents augmented reality information in line, - wherein the estimator (500-1) determine the measurement instructions using a movement model (500-2) t and the measuring ranges are limited by geometric data from real objects, - whereby the calculation of the measuring instructions and the calculation of the tracking result are supported by additional sensors.

Description

Quick Facts

Optical tracking on mobile devices is not adequately solved with the current state of the art. Mobile devices have limited computing capacity, so they tried to transfer the video image over the network, to perform tracking on a server and to send back the enriched image. The large amounts of data for bidirectional transmission of the video image, however, lead to insufficient results in terms of the round-trip time.

This invention shows a way to perform tracking on the server without constantly transmitting the full video image. While the server performs the complex tracking operations, the client performs simple measurements based on server specifications. These measurements require low computer capacity and reduce the size of the sent data packets enormously.

State of the art

So-called augmented reality systems are known from the prior art. These allow the overlay of computer-generated, virtual information with visual impressions of the real environment. For this purpose, the visual impressions of the real world, preferably with semipermeable data glasses worn on the head, are mixed with virtual information. The display of the virtual information or objects can be context-dependent, d. H. adapted and derived from the respective considered real environment be executed. In principle, any kind of data such as texts, images etc. can be used as information.

Documented applications of technology envisage use in the production, service and development of complex products. Also, the use of technology from the production of aircraft is known. After the publication DE 198 32 974 A1 and DE 101 28 015 A1 is the use of augmented reality technologies in production environments known.

Other uses include, for example, the AR documentation of consumer products.

In addition, prior art optical position sensing systems are known which determine the position and or orientation of objects in a survey space. These so-called tracking systems allow, for example, the detection of up to six degrees of freedom of an object. Systems with different physical principles of action are used. Commonly used are so-called optical tracking systems that determine the position of objects located in the survey room and / or the position of the camera via the detection of the objects by various methods of computer-aided image processing.

However, optical tracking systems require relatively high computing capacity to achieve acceptable speeds (more than 5 frames per second). This is often not the case with mobile devices. One possible solution to this problem is to outsource the tracking calculations and enrich the video image to a powerful server. Approaches to this are likewise known from the prior art.

According to the prior art, a distinction is made between so-called marker-based and so-called markerless optical tracking systems. When using additional sensors, for example, intertial sensors, so-called hybrid tracking systems are used. The invention is useful for any process.

Since the free search for objects is not possible with sufficient speed thanks to today's available computing power, simplifying assumptions are made.

One way to simplify the complexity is to equip the tracking room with high-contrast, easily recognizable markers, hereafter referred to as markers. However, this approach has the disadvantage that the tracking room must be equipped and the tracking is possible only in the equipped area. Optical tracking methods using this method are generally referred to as marker-based tracking methods.

Another possibility for simplifying the complexity is the use of models, for example statistical movement models and / or additional sensors, for example acceleration sensors, which limit the search space of the image processing algorithms. Although this approach simplifies complexity without the need for room equipment, it can not help with initialization. Initialization in this context means the determination of the internal state of the system, so that the tracking process can be carried out successfully and in a sufficiently short time. The system must first be set to a valid initial state. This initial state includes the approximate position and orientation of the reality to the camera and advantageously information about the current movement of the camera.

Systems which do not rely on special markers in space are known from the following documents and are referred to below as markerless tracking systems:
Behringer, R .; Park, J .; Sundareswaran, V .: Model-Based Visual Tracking for Outdoor Augmented Reality Applications. In: International Symposium on Mixed and Augmented Reality (ISMAR'02). Darmstadt 2002, p. 277
Genc, Y; Riedel, S .; Souvannavong, F .; Akinlar, C .; Navab, N .: Marker-less Tracking for AR: A Learning-Based Approach. In: International Symposium on Mixed and Augmented Reality (ISMAR'02). Darmstadt 2002, p. 295
KarDe Chia, Adrian David Cheok, Simon JD Prince: Online 6 DOF Augmented Reality Registration from Natural Features. In: International Symposium on Mixed and Augmented Reality (ISMAR'02). Darmstadt 2002, p. 277.

Input information of the system is a two-dimensional field of red, green and blue intensities in the camera image. In addition, acceleration data can be read in by additional sensors. The output data of the system is the position and / or advantageously the orientation of one or more objects relative to the camera. The elements of reality can then, for example, span a coordinate system for augmented reality information. Position and orientation can advantageously be described by a transformation matrix. Advantageously, position and orientation are the output data of the system. Advantageously, calibration data from sensors may also be output data. Advantageously, model data of the environment may also be output data. However, the invention also relates to systems in which only position (one, two or three dimensional) or only orientation (one, two or three dimensional) or any combination of the number of dimensions represent the output data.

disadvantage

• 1. Anforderungen an Rechnerkapazität übersteigen die Möglichkeiten mobiler Systeme.
• 2. Übertragung des Videosignals zum Server und zurück erfordert eine große Datenmenge und übersteigt eine akzeptable Round-Trip-Time.
• 3. Verringerung der zu übertragenden Datenmenge durch Komprimierung des Bildes erfordert ebenfalls viel Zeit und Rechenleistung.
• 4. Bei hoher Auslastung des Netzwerks wird dies für zusätzliche Funktionen, wie Geometriedaten oder die Kommunikation mit einem entfernten Experten, etc. eingeschränkt.
• 5. Verfahren, welche die Position des mobilen Gerätes zum Beispiel durch Netzwerkeigenschaften (Mobilfunkzelle, etc. ) erhalten sind sehr ungenau genau.

A disadvantage of the known methods for the implementation of AR on mobile systems

• 1. Computer capacity requirements exceed the capabilities of mobile systems.
• 2. Transmission of the video signal to the server and back requires a large amount of data and exceeds an acceptable round-trip time.
• 3. Reducing the amount of data to be transferred by compressing the image also requires a lot of time and computing power.
• 4. When the network is heavily used, this is restricted for additional functions, such as geometry data or communication with a remote expert, etc.
• 5. Method, which obtained the position of the mobile device, for example, by network properties (mobile radio cell, etc.) are very inaccurate exactly.

In Wagner, D., Schmalstieg, D .; Seventh IEEE International Symposium on Wearable Computers, ISWC 2003, pages 127-135, October 2003, describes mobile AR devices that exchange data in communication with a server, the problem being large Data should be avoided when transferring raw video data. This is related to 2 B proposed to outsource the tracking to the server, whereby special preprocessed video data is transferred from the handheld to the server for this purpose. Preprocessing is accomplished by transitioning from a color image to a black and white image and also performing compression so that WLAN communication can be performed. In a "downstream communication" corresponding pose information is then provided by the server to the handheld.

In Cruz, DS et al., "Region of Interest Coding in JPEG 2000 for interactive client / server applications" in: 1999 IEEE 3rd Workshop on Multimedia Signal Processing, pp. 389-394, 1999, an approach is described which provides an efficient "on- the-fly "decoding of regions of interest in an already encoded image, without the need for a complete decoding / encoding process. This approach is particularly advantageous in interactive client / server applications interconnected by narrowband networks in which the client may request the server to perform the transmission of the desired information more efficiently at a lower processing cost.

Object of the invention

The invention is based on the object of enabling augmented reality or other applications on mobile devices which require position and or orientation of the device in relation to real objects. To relieve the mobile device tasks are outsourced to servers. Processes outsourced to servers should minimize the load on the transmission system. The round trip time from the mobile device to the server is greatly reduced by the invention.

Description of the invention

The previously derived and resulting from the prior art task is inventively achieved by methods in which the transmission of the entire video image is avoided or reduced to a minimum.

The invention provides methods for determining the position and / or orientation of mobile devices relative to real devices or for determining the position and / or orientation of real objects relative to a mobile device, wherein an image containing the real object is captured by a camera associated with the mobile device and wherein said determination is made by comparison between the captured image and stored information about the real object,
the methods comprising the steps of
that a metering instruction is sent to the mobile device from a server separate from the mobile device;
that the mobile device makes a measurement in the image based on the measurement instruction;
and that the measurement result is sent from the mobile device to the server.

The real object can be a marker. But you can also work without markers and instead work with one or more succinct, real-life objects that store information. The latter can also be done by initializing the system.

It is possible to send several measurement instructions one after the other, to take several measurements in succession, and to send several measurement results in succession. A typical example of this would be the measurement along several measurement lines or in several partial measurement areas within the image.

It is possible to carry out several position and / or orientation determinations in succession. In this case, the method provides a continuous or quasi-continuous position and / or orientation determination.

The appended claims 1 to 31 include methods of the invention and preferred embodiments thereof. The features disclosed therein can also be realized in the methods mentioned in the four preceding paragraphs.

In the method according to the invention, the server informs the client which measurement processes are to be carried out in the video image. Measurement processes are minimized by the use of intelligent algorithms.

shows an advantageous structure for carrying out the method according to the invention. A camera ( 400-1 ) creates a video image ( 100 ) the real environment in which real objects ( 50 ), advantageously markers. On the mobile device is a component, the so-called Observer ( 400-2 ), which can examine the camera image. Via a digital transmission system ( 200 ), here a wireless network, receives the mobile device ( 400 ), also referred to as client, measuring instructions ( 300 ) from the estimator ( 500-1 ), which is located on the server ( 500 ) is located. Based on the measurement instructions, the observer makes measurements and sends the measurement data ( 600 ) to the estimator. The estimator uses the measurement data to calculate the current tracking result ( 700 ) and sends this advantageously to the renderer ( 400-3 ) of the mobile device. By means of the tracking result, the renderer can display AR information in the correct position. Advantageously, the AR information ( 800 ) is mixed with the video image and output via an output system ( 400-4 ) the user ( 1000 ) is displayed. Advantageously, the output system is a head-mounted screen system.

The estimator determines measurement instructions using a motion model ( 500-2 ). The measuring ranges are limited by geometry data of the real objects, which are taken from a database ( 500-3 ). The calculation of measurement instructions and the calculation of the tracking result are supported by additional sensors. Advantageously, these are intertial sensors ( 400-5 ), which are attached to the display device of the mobile device and acceleration and / or speed data ( 900-1 ) to the estimator. Advantageously, this is a Global Positioning System ( 400-6 ), which is attached to the display device of the mobile device and position and / or acceleration and / or speed data ( 900-2 ) sends to the estimator. Advantageously, the calculation of measurement instructions and the calculation of the tracking result are supported by information from the provider of the transmission system. Advantageously, the provider of the transmission system ( 1100 ) Position information ( 900-3 ) known.

The measuring instructions ( 300 ) and thus the measured data ( 600 ) are advantageously very different and adapted to the state of the overall system. For example, to initialize it may be necessary to have a complete image, e.g. As a digital image with reduced resolution and / or color depth or a binary image to transfer. After initialization, it may be sufficient to transfer color intensities of individual dots in the image.

Advantages and applications of the method

Advantages of the described method for finding the position and / or the orientation of mobile devices with respect to real objects is the significant reduction of the round-trip-time by the variable requirement of only required measurement results by the server. This connection is in qualitatively clarified. While the transfer of the video image in 2 directions constantly generates large amounts of data, these are lower by the selected work of the client, after initialization even much lower.

Possible applications of the method include, in particular, applications of augmented reality technology in the areas of service and maintenance, production and applications in the mobile environment.

LIST OF REFERENCE NUMBERS

50

Real object

100

video image

200

transmission system

300

measuring instructions

400

mobile device

400-1

camera

400-2

Observer

400-3

Renderer (graphics system)

400-4

output system

400-5

inertial sensors

400-6

Global Positioning System

400-7

Data store for AR information

500

server

500-1

Estimator

500-2

movement model

500-3

Database

600

measurement data

700

tracking results

800

AR Information

900-1

Data of the Intertial Sensor

900-2

Data from the Global Positioning System

900-3

Data of the provider of the transmission system

1000

user

1100

Providers of Übertragungssytems, advantageously network providers

Claims (31)

Method for finding the position and / or orientation of mobile devices with respect to real objects, wherein - the mobile device ( 400 ) a camera ( 400-1 ) and there is a component on the mobile device as Observer ( 400-2 ), which can examine an image of the camera, and - via a wireless network ( 200 ) the mobile device ( 400 ) Measurement instructions ( 300 ) from an estimator ( 500-1 ), which resides on a server ( 500 ) due to the measuring instructions of the Observer ( 400-2 ) Makes measurements and measurement data ( 600 ) to the estimator, whereby the estimator uses the measurement data to obtain a current tracking result ( 700 ) and this to a graphics system, as a renderer ( 400-3 ), the mobile device sends, - by means of the tracking result of the renderer ( 400-3 ) Represents augmented reality information according to the situation, - whereby the estimator ( 500-1 ) the measurement instructions using a motion model ( 500-2 ) and the measurement ranges are limited by geometry data from real objects, - whereby the calculation of the measurement instructions and the calculation of the tracking result are supported by additional sensors. A method according to claim 1, characterized in that the calculation of an image of the augmented reality information takes place on the mobile device and thus only the tracking result has to be transmitted from the server to the mobile device. A method for finding the position and / or orientation of a camera associated with a mobile device with respect to real objects, wherein - on the mobile device ( 400 ) a component is located as Observer ( 400-2 ), which is an image of the camera ( 400-1 ), and - via a wireless network ( 200 ) the mobile device ( 400 ) Measurement instructions ( 300 ) from an estimator ( 500-1 ), which resides on a server ( 500 ) due to the measuring instructions of the Observer ( 400-2 ) Makes measurements and measurement data ( 600 ) to the estimator, whereby the estimator uses the measurement data to obtain a current tracking result ( 700 ) and this to a graphics system, as a renderer ( 400-3 ), the mobile device sends, - by means of the tracking result of the renderer ( 400-3 ) Represents augmented reality information according to the situation, - whereby the estimator ( 500-1 ) the measurement instructions using a motion model ( 500-2 ) and the measurement ranges are limited by geometry data from real objects, - whereby the calculation of the measurement instructions and the calculation of the tracking result are supported by additional sensors. Method according to one of claims 1 to 3, characterized in that the communication between the mobile device and server is wireless with different, each adapted to the field of application techniques, in particular with Bluetooth for the local area and wireless LANs or GSM / GPRS / UMTS for the remote area. Method according to one of the preceding claims, characterized in that the measurement instructions and thus the measurement results sent by the mobile device to the server can vary in form and scope. A method according to claim 1 or 3, characterized in that the measurement instructions represent image coordinates and the measurement results individual pixel intensities to image coordinates. Method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions represent straight lines of arbitrary length and the measurement results represent pixel intensities on these straight lines. Method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions represent straight lines of arbitrary length and the measurement results represent point coordinates. A method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions straight lines of any length and the measurement results represent distances. Method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions represent rectangles of any area and the measurement results represent pixel intensities of image coordinates within these rectangles. Method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions represent rectangles of any surface and the measurement results represent point coordinates. Method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions represent rectangles of any surface and the measurement results represent distances. Method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions represent free-form surfaces and the measurement results represent pixel intensities of image coordinates within these surfaces. Method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions represent free-form surfaces and the measurement results represent point coordinates, A method according to claim 1 or 3, characterized in that, expressed in image coordinates, the measurement instructions free-form surfaces and the measurement results represent distances. Method according to claim 1 or 3, characterized in that geometry models of real objects are used to calculate the measurement instructions. A method according to claim 1 or 3, characterized in that for the calculation of the measurement statements assumptions about the position and orientation of real objects are made relative to the camera. A method according to claim 1 or 3, characterized in that for the calculation of the measurement instructions several assumptions about the position and orientation of a real object relative to the camera are taken and the number of assumptions in position and orientation areas increases, for which a higher probability was assumed. A method according to claim 1 or 3, characterized in that for the calculation of the measurement statements assumptions about the position and orientation of real objects relative to the camera by means of a motion model are coupled to a Kalman filter. A method according to claim 1 or 3, characterized in that the movement model is adjusted due to the tracking results at runtime. Method according to claim 1, 2 or 3, characterized in that the tracking results are 2- or 3-dimensional position information and / or 1-, 2- or 3-dimensional orientation information and / or information about camera parameters. A method according to claim 1 or 3, characterized in that the estimator makes assumptions about the uncertainty of the last tracking results and increases in the measurement instructions, the number of pixels to be examined with increasing uncertainty. A method according to claim 1 or 3, characterized in that the same time the tracking results and new measurement instructions are transmitted to the mobile device. Method according to Claim 1 or 3, characterized in that data from intertial sensors are used to calculate the measurement instructions and / or to calculate the tracking result. Method according to claim 1 or 3, characterized in that position data of the provider of the transmission system are used to calculate the measurement instructions and / or to calculate the tracking result. A method according to claim 1 or 3, characterized in that data of a Global Positioning System are used to calculate the measurement instructions and / or to calculate the tracking result. A method according to claim 1 or 3, characterized in that the measurement instructions or the measurement results are compressed. A method according to claim 1, characterized in that the calculation of the augmented reality information is done on the server, is transmitted to the mobile device and enriched on the mobile device with a video image. A method according to claim 1, characterized in that the calculation of the augmented reality information is done on the server, is transmitted to the mobile device and displayed by means of a semipermeable display. The method of claim 1, 2 or 3, characterized in that the server automatically supplies the renderer of the mobile device with context-dependent AR content due to the tracking results when the network load required by the tracking is low and the results show that new content is required soon become. A method according to claim 1 or 3, characterized in that the server, depending on the current required by the tracking capacity of the transmission medium, buffers additional data required by the mobile device and flexibly transmits.

Images