US20150103183A1 - Method and apparatus for device orientation tracking using a visual gyroscope - Google Patents
Method and apparatus for device orientation tracking using a visual gyroscope Download PDFInfo
- Publication number
- US20150103183A1 US20150103183A1 US14/051,282 US201314051282A US2015103183A1 US 20150103183 A1 US20150103183 A1 US 20150103183A1 US 201314051282 A US201314051282 A US 201314051282A US 2015103183 A1 US2015103183 A1 US 2015103183A1
- Authority
- US
- United States
- Prior art keywords
- orientation
- frame
- keyframe
- visual
- map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
-
- G06T7/0044—
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
- G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
- G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
- G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
- G01C21/1656—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments with passive imaging devices, e.g. cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Abstract
A method for tracking device orientation on a portable device is disclosed. The method comprises initializing a device orientation to a sensor orientation, wherein the sensor orientation is based on information from an inertial measurement unit (IMU) sensor. It also comprises initiating visual tracking using a camera on the portable device and capturing a frame. Next, it comprises determining a plurality of visual features in the frame and matching the frame to a keyframe, wherein capture of the keyframe precedes capture of the frame. Subsequently, it comprises computing a rotation amount between the frame and the keyframe. Responsive to a determination that a rotational distance between the frame and the keyframe exceeds a predetermined threshold, promoting the frame to a keyframe status and adding it to a first orientation map and adjusting the frame with all prior captured keyframes.
Description
- The present application is related to U.S. patent application Ser. No. ______, filed ______, entitled “A METHOD AND APPARATUS FOR LONG TERM IMAGE EXPOSURE WITH IMAGE STABILIZATION ON A MOBILE DEVICE,” naming Syed Zahir Bokari, Josh Abbott, and Jim van Welzen as inventors, and having attorney docket number NVID-PDU-13-0254-US1. That application is incorporated herein by reference in its entirety and for all purposes.
- Embodiments according to the present invention generally relate to augmented reality systems and more specifically to device orientation tracking for augmented reality systems.
- While augmented reality technologies have improved considerably in recent years, the improvements typically come at the cost of computationally intensive procedures implemented on expensive hardware. The high cost of implementing effective augmented reality systems is a barrier to entry that prevents casual users from having access to such systems on everyday devices that have relatively low processing capability, e.g., tablet computers, phones and other hand-held devices.
- A critical component in successfully implementing augmented reality systems is device orientation tracking. In other words, tracking the orientation of users and objects in the scene is critical for developing augmented reality applications. Procedures implemented in conventional augmented reality applications for tracking device orientation with expected levels of robustness, speed and precision are computationally expensive. Accordingly, they are not ideal for handheld mobile devices.
- For example, prior approaches for tracking device orientation use full Simultaneous Localization and Mapping (SLAM) algorithms, which attempt to determine a device's position as well as orientation. SLAM is a technique used by augmented reality applications build a map of the environment of the device while at the same time keeping track of the device's current location, where the current location includes both the device's position and also its orientation. SLAM generally works by creating geometrically consistent maps of the environment using inputs from different types of sensors, e.g., 2D cameras, 3D sonar sensors, single dimensional beams or 2D sweeping laser rangefinders etc. Building a 3D map of the environment can be significantly intensive from a computation standpoint, in part, because it involves tracking the device's position as well as orientation. Because of the considerable computational requirements for implementing SLAM, it is not a suitable procedure for implementation on a mobile device, such as a smart phone.
- Other conventional augmented reality systems are unsuitable for determining device orientation on mobile devices because they suffer from problems such as drift error. For example, one approach used in conventional devices for tracking device orientation uses only Inertial Measurement Unit (IMU) sensors. An IMU is an electronic device that can measure and report on a device's velocity, orientation or gravitational forces, using a combination of inputs from other devices such an accelerometer, gyroscope and magnetometer. A major disadvantage of using IMUs is that they typically suffer from accumulated error. Because the guidance system is continually adding detected changes to its previously-calculated positions, any errors in measurement, however small, are accumulated from point to point. This leads to drift error, or an ever-increasing difference between where the system thinks it is located and the actual location, where, as stated above, location includes both the device's position and also its orientation. Stated differently, drift is a problem because integration of the orientation tracking's relative measurements accumulates the small errors in each measurement. This can, consequently, create significant differences between the estimated and actual orientation. Further, measurements from IMU sensors tend to be error prone and noisy. Thus, IMU sensors are typically not suitable for an immersive augmented reality environment.
- Conventional methods of tracking device orientation for augmented reality systems, therefore, are unsuitable for use on mobile devices because they are either too expensive and computationally intensive or simply unsuitable as a result of problems such as noise and drift error.
- Accordingly, a need exists for a system and a method for tracking device orientation on a mobile device that has a smaller compute footprint. For example, embodiments of the present invention track device orientation without constructing an entire map of the environment or even trying to calculate the device's position. Further, embodiments of the present invention advantageously track device orientation without requiring a user to change the position of the camera on the mobile device. As a result, device orientation can be tracked more efficiently and quickly using smaller and more affordable electronic components.
- Further, a need exists for systems and methods for a vision-based orientation tracking procedure on a mobile device that uses the camera on the device to determine orientation by identifying and tracking landmarks (or image features) in a natural environment. As a result, embodiments of the present invention advantageously provide a robust, fast and precise orientation tracking solution while avoiding the pitfalls of noise and drift error that is prevalent in conventional IMU sensor-based systems. Moreover, embodiments of the present invention use markerless tracking to provide the most accurate device orientation over naïve sensor-based approaches. In one embodiment of the present invention, IMU sensors are utilized as a fallback option if vision-based tracking fails.
- In one embodiment, a method for tracking device orientation on a portable device is disclosed. The method comprises initializing a device orientation to a sensor orientation, wherein the sensor orientation is based on information from an inertial measurement unit (IMU) sensor. It also comprises initiating visual tracking using a camera on the portable device and capturing a frame. Next, it comprises determining a plurality of visual features in the frame and matching the frame to a keyframe, wherein capture of the keyframe precedes capture of the frame. Subsequently, it comprises computing a rotation amount between the frame and the keyframe. Responsive to a determination that a rotational distance between the frame and the keyframe exceeds a predetermined threshold, promoting the frame to a keyframe status and adding it to a first orientation map and adjusting the frame with all prior captured keyframes.
- In another embodiment, a computer-readable storage medium having stored thereon instructions that, if executed by a computer system cause the computer system to perform a method for tracking device orientation on a portable device is disclosed. The method comprises initializing a device orientation to a sensor orientation, wherein the sensor orientation is based on information from an inertial measurement unit (IMU) sensor. It also comprises initiating visual tracking using a camera on the portable device and capturing a frame. Next, it comprises determining a plurality of visual features in the frame and matching the frame to a keyframe, wherein capture of the keyframe precedes capture of the frame. Subsequently, it comprises computing a rotation amount between the frame and the keyframe. Responsive to a determination that a rotational distance between the frame and the keyframe exceeds a predetermined threshold, promoting the frame to a keyframe status and adding it to a first orientation map and adjusting the frame with all prior captured keyframes.
- In a different embodiment, a system for tracking device orientation on a portable device is presented. The system comprises a display screen; a memory; a camera; and a processor configured to implement a visual gyroscope, wherein the visual gyroscope performs a method for tracking device orientation on the portable device. The method comprises initializing a device orientation to a sensor orientation, wherein the sensor orientation is based on information from an inertial measurement unit (IMU) sensor. It also comprises initiating visual tracking using a camera on the portable device and capturing a frame. Next, it comprises determining a plurality of visual features in the frame and matching the frame to a keyframe, wherein capture of the keyframe precedes capture of the frame. Subsequently, it comprises computing a rotation amount between the frame and the keyframe. Responsive to a determination that a rotational distance between the frame and the keyframe exceeds a predetermined threshold, promoting the frame to a keyframe status and adding it to a first orientation map and adjusting the frame with all prior captured keyframes.
- The following detailed description together with the accompanying drawings will provide a better understanding of the nature and advantages of the present invention.
- Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements.
-
FIG. 1 shows an exemplary computer system with a camera used to implement a visual gyroscope for tracking device orientation in accordance with one embodiment of the present invention. -
FIG. 2 shows an exemplary operating environment of a mobile device capable of tracking device orientation in accordance with one embodiment of the present invention. -
FIG. 3 illustrates a use case for the visual gyroscope in accordance with one embodiment of the present invention. -
FIG. 4 is a high level block diagram illustrating the elements of the orientation tracking system proposed in accordance with an embodiment of the present invention. -
FIG. 5 is a diagram of a visual gyroscope capturing an initial keyframe in accordance with an embodiment of the present invention. -
FIG. 6 is an illustration of the visual gyroscope capturing a second keyframe following a user rotation from an initial direction in accordance with an embodiment of the present invention. -
FIG. 7 is an illustration of the visual gyroscope starting a new map based on IMU sensor data in accordance with an embodiment of the present invention. -
FIG. 8 is an illustration of the visual gyroscope creating a new map based on IMU sensor data until an overlap is found with the map created from vision-based data in accordance with an embodiment of the present invention. -
FIG. 9 depicts a flowchart of an exemplary computer implemented process of tracking device orientation in accordance with one embodiment of the present invention. -
FIG. 10 depicts a flowchart of an exemplary computer implemented process of using sensors for tracking device orientation when visual tracking is lost in accordance with one embodiment of the present invention. - In the figures, elements having the same designation have the same or similar function.
- Reference will now be made in detail to the various embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. While described in conjunction with these embodiments, it will be understood that they are not intended to limit the disclosure to these embodiments. On the contrary, the disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the disclosure as defined by the appended claims. Furthermore, in the following detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be understood that the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.
- Notation and Nomenclature
- Some portions of the detailed descriptions that follow are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present application, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those utilizing physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as transactions, bits, values, elements, symbols, characters, samples, pixels, or the like.
- It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present disclosure, discussions utilizing terms such as “capturing,” “determining,” “matching,” “promoting,” “bundling,” “storing,” or the like, refer to actions and processes (e.g.,
flowchart 900 ofFIG. 9 ) of a computer system or similar electronic computing device or processor (e.g.,system 100 ofFIG. 1 ). The computer system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within the computer system memories, registers or other such information storage, transmission or display devices. - Embodiments described herein may be discussed in the general context of computer-executable instructions residing on some form of computer-readable storage medium, such as program modules, executed by one or more computers or other devices. By way of example, and not limitation, computer-readable storage media may comprise non-transitory computer-readable storage media and communication media; non-transitory computer-readable media include all computer-readable media except for a transitory, propagating signal. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or distributed as desired in various embodiments.
- Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory or other memory technology, compact disk ROM (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can accessed to retrieve that information.
- Communication media can embody computer-executable instructions, data structures, and program modules, and includes any information delivery media. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media. Combinations of any of the above can also be included within the scope of computer-readable media.
-
FIG. 1 shows an exemplary computer system with a camera used to implement a visual gyroscope for tracking device orientation in accordance with one embodiment of the present invention.Computer system 100 depicts the components of a generic computer system in accordance with embodiments of the present invention providing the execution platform for certain hardware-based and software-based functionality. In general,computer system 100 comprises at least oneCPU 101, asystem memory 115, and at least one graphics processor unit (GPU) 110. TheCPU 101 can be coupled to thesystem memory 115 via a bridge component/memory controller (not shown) or can be directly coupled to thesystem memory 115 via a memory controller (not shown) internal to theCPU 101. TheGPU 110 may be coupled to adisplay 112. One or more additional GPUs can optionally be coupled tosystem 100 to further increase its computational power. The GPU(s) 110 is coupled to theCPU 101 and thesystem memory 115. TheGPU 110 can be implemented as a discrete component, a discrete graphics card designed to couple to thecomputer system 100 via a connector (e.g., AGP slot, PCI-Express slot, etc.), a discrete integrated circuit die (e.g., mounted directly on a motherboard), or as an integrated GPU included within the integrated circuit die of a computer system chipset component (not shown). Additionally, alocal graphics memory 114 can be included for theGPU 110 for high bandwidth graphics data storage. - The
CPU 101 and theGPU 110 can also be integrated into a single integrated circuit die and the CPU and GPU may share various resources, such as instruction logic, buffers, functional units and so on, or separate resources may be provided for graphics and general-purpose operations. The GPU may further be integrated into a core logic component. Accordingly, any or all the circuits and/or functionality described herein as being associated with theGPU 110 can also be implemented in, and performed by, a suitably equippedCPU 101. Additionally, while embodiments herein may make reference to a GPU, it should be noted that the described circuits and/or functionality can also be implemented and other types of processors (e.g., general purpose or other special-purpose coprocessors) or within a CPU. -
System 100 can be implemented as, for example, a desktop computer system or server computer system having a powerful general-purpose CPU 101 coupled to a dedicatedgraphics rendering GPU 110. In such an embodiment, components can be included that add peripheral buses, specialized audio/video components, IO devices, and the like. Similarly,system 100 can be implemented as a handheld device (e.g., cell-phone, tablet computer, MP3 player, etc.), direct broadcast satellite (DBS)/terrestrial set-top box or a set-top video game console device such as, for example, the Xbox® or the PlayStation3®.System 100 can also be implemented as a “system on a chip”, where the electronics (e.g., thecomponents - A Method and Apparatus for Device Orientation Tracking Using a Visual Gyroscope
- Embodiments of the present invention provide a system and a method for tracking device orientation on a mobile device that has a smaller compute footprint. For example, embodiments of the present invention track device orientation without constructing an entire map of the environment or even trying to calculate the device's position. Further, embodiments of the present invention advantageously track device orientation without requiring a user to change the position of the camera on the mobile device. As a result, device orientation can be tracked more efficiently and quickly using smaller and more affordable electronic components.
- Embodiments of the present invention provide a system and a method for a vision-based orientation tracking procedure on a mobile device that uses the camera on the device to determine orientation by identifying and tracking landmarks (or image features) in natural environment. In one embodiment, the present invention is a visual gyroscope that is operable to precisely determine a handheld device's orientation using vision-based techniques. As a result, embodiments of the present invention advantageously provide a robust, fast and precise orientation tracking solution while avoiding the pitfalls of noise and drift error prevalent in conventional IMU sensor-based systems. Embodiments of the present invention use markerless tracking to provide the most accurate device orientation over naïve sensor-based approaches. In one embodiment of the present invention, IMU sensors are utilized simply as a fallback option if vision-based tracking fails.
-
FIG. 2 shows an exemplary operating environment of a mobile device capable of tracking device orientation in accordance with one embodiment of the present invention.System 200 includescamera 202, image signal processor (ISP) 204,memory 206, IMU sensor 240,input module 208, central processing unit (CPU) 210,display 212, communications bus 214, andpower source 220.Power source 220 provides power tosystem 200 and may be a DC or AC power source.System 200 depicts the components of an exemplary system in accordance with embodiments of the present invention providing the execution platform for certain hardware-based and software-based functionality. Although specific components are disclosed insystem 200, it should be appreciated that such components are examples. That is, embodiments of the present invention are well suited to having various other components or variations of the components recited insystem 200. It is appreciated that the components insystem 200 may operate with other components other than those presented, and that not all of the components ofsystem 200 may be required to achieve the goals ofsystem 200. -
CPU 210 and theISP 204 can also be integrated into a single integrated circuit die andCPU 210 andISP 204 may share various resources, such as instruction logic, buffers, functional units and so on, or separate resources may be provided for image processing and general-purpose operations.System 200 can be implemented as, for example, a digital camera, cell phone camera, portable device (e.g., audio device, entertainment device, handheld device), webcam, video device (e.g., camcorder) or any other device with a front or back facing camera. - In one embodiment,
camera 202 captures light via a front-facing or back-facing lens (depending on how the user typically holds the device), and converts the light received into a signal (e.g., digital or analog).Camera 202 may comprise any of a variety of optical sensors including, but not limited to, complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) sensors.Camera 202 is coupled to communications bus 214 and may provide image data received over communications bus 214.Camera 202 may comprise functionality to determine and configure optical properties and settings including, but not limited to, focus, exposure, color or white balance, and areas of interest (e.g., via a focus motor, aperture control, etc.). In one embodiment,camera 202 may also represent a front facing and a back facing camera both of which are operable to capture images contemporaneously. - Image signal processor (ISP) 204 is coupled to communications bus 214 and processes the signal generated by
camera 204, as described herein. More specifically,image signal processor 204 may process data fromcamera 202 for storing inmemory 206. For example,image signal processor 204 may process frames of visual data captured usingcamera 202 to be stored withinmemory 206. -
Input module 208 allows entry of commands intosystem 200 which may then, among other things, control the sampling of data bycamera 202 and subsequent processing byISP 204.Input module 208 may include, but it not limited to, navigation pads, keyboards (e.g., QWERTY), up/down buttons, touch screen controls (e.g., via display 212) and the like. - Central processing unit (CPU) 210 receives commands via
input module 208 and may control a variety of operations including, but not limited to, sampling and configuration ofcamera 202, processing byISP 204, and management (e.g., addition, transfer, and removal) of images and/or video frommemory 206. - Inertial Measurement Unit (IMU) module 240 can detect the current rate of acceleration of the
device 200 using one or more accelerometers in device 200 (not shown). Accelerometers detect acceleration forces along a single axis, three are often combined to provide acceleration detection along the x, y and z axis. When the accelerometer is at rest, the axis pointing down will read one due to the force of gravity and the two horizontal axis will read zero. - The IMU module 240 can also detect changes in rotational attributes like pitch, roll, and yaw using one or more gyroscopes in device 200 (not shown). A gyroscope detects the rotational change of a device. Finally, IMU module 240 may also receive data from a magnetometer (not shown), which is a sensor for measuring the strength and direction of magnetic fields, and can be used for tracking magnetic north, thereby, acting like a compass. Alternatively, the IMU module 240 may receive direction data from a digital compass.
-
FIG. 3 illustrates a use case for the visual gyroscope in accordance with one embodiment of the present invention. As thestationary user 310 pans the camera on his handheld device around, the visual gyroscope of the present invention creates a spherical map of the local environment using vision data from the camera by dynamically identifying and tracking features in the environment. In order to create the spherical map, the visual gyroscope needs to track the orientation of the device. Stated differently, the visual gyroscope of the present invention can track features or landmarks in the environment and use information from the tracking to determine the handheld device's orientation, which is used in order to create a spherical map of the environment for identification thereof. The map can then be used in, for example, augmented reality applications, where the user could obtain more information regarding the landmarks in the scene identified through the visual gyroscope. For example,user 310 inFIG. 3 could identify the building that the camera pans across insection 320. Or the user may, for example, be able to determine whether a particular restaurant in the building is open or closed by receiving real time information concerning the identified restaurant. By way of further example, the user may be able to overlay certain graphics on landmarks within the field of view with information regarding the identified businesses, e.g., restaurant menus on restaurants, menu of services rendered by a spa, etc. - Unlike conventional SLAM based and other approaches, the user is not required to change the position of the camera to build a model of the scene and construct an entire map of the environment. A stationary user can capture the feature map with natural panning motion.
-
FIG. 4 is a high level block diagram illustrating the elements of theorientation tracking system 405 proposed in accordance with an embodiment of the present invention. The basic approach employed in the present invention is using vision-based methods with a visual gyroscope for image feature recognition, but also combining vision data with IMU sensor data for robustness. As explained above, only the IMU sensor data lead to drift, which caused marker misplacement. Thus, embodiments of the present invention use vision data to correct for drift, but also use IMU sensor data as a back-up when acquisition of visual data is lost or breaks down. - As shown in
FIG. 4 , data from thegyroscope 414,accelerometer 416 andmagnetometer 418 feeds into the IMU sensor 420. The output of thecamera 412 and the IMU sensor 420 is transmitted to the user application 422. The time-stamped camera data andIMU sensor data 410 from user application 422 is directed to thevisual gyroscope module 480. Thevisual gyroscope module 480 uses the time-stamped camera and sensor data to create time-stampeddevice orientation data 490. Theorientation data 490 is fed back into the user application 422. In one embodiment, user application 422 can be an augmented reality application used, for example, to locate nearby restaurants. -
FIG. 5 is a diagram of a visual gyroscope capturing an initial keyframe in accordance with an embodiment of the present invention. The visual gyroscope module creates anorientation map 524 by capturing keyframes. A keyframe is a snapshot taken by the camera in the direction the camera is pointing in that is at a sufficient degree of rotation from a prior keyframe. This keyframe is saved as part of an orientation map and can be found if the device gets lost while the user is panning around. As shown inFIG. 5 , in order to createorientation map 524, the device captures aninitial keyframe 522 atuser direction 526. - When the
system 200 is first started, the visual gyroscope module is initialized to the absolute orientation of the device read from the IMU sensor 420. This initialization plays an important role when visual tracking fails and will be explained further below. - After every captured frame, including the initial keyframe, the visual gyroscope procedure computes visual features that are repeatable and distinct within the frame. Any number of feature detecting procedures can be employed for this purpose, e.g., Harris & Stephens, Plessey, Shi-Tomasi, Moravec corner detection, Wang and Brady corner detection, SUSAN (smallest univalue segment assimilating nucleus) corner detector etc. In one embodiment, the visual gyroscope uses FAST (Features from Accelerated Segment Test) for feature detection. FAST is a well-known corner detection algorithm. Corner detection is an approach used within computer vision systems to extract certain kinds of features and infer the contents of an image.
- The visual gyroscope procedure then performs feature description for the features found from feature detection using a well-known procedure, e.g., SURF (Speeded Up Robust Feature), Scale Invariant Feature Transform (SIFT), or BRIEF (Binary Robust Independent Elementary Features). In one embodiment, the visual gyroscope module uses BRIEF for feature description. Feature description comprises detecting and describing local features in images. For example, for any object in an image, interesting points on the object can be extracted to provide a feature description of the object. This description, extracted from a training image, can then be used to identify the object when attempting to locate the object in a test image containing many other objects. Accordingly, the visual gyroscope procedure, in one embodiment, can use BRIEF for feature description of objects within the captured frame.
- The feature detection procedure, e.g., FAST and the feature description procedure, e.g., BRIEF are run on every captured frame including the initial keyframe. Feature detection procedures find features in the image while feature description procedures describe the feature in the sequence of bits so as to compare it with similar features in other frames. In one embodiment, the smoothing operation in BRIEF can be removed to speed up the procedure to run real-time.
- In one embodiment, the feature points are projected onto the image plane as seen by the “ideal” camera using a camera inverse matrix (K−1)*(x, y, 1) column vector. In one embodiment, the newly transformed points are normalized to make a spherical representation of the points. All features in every frame will be warped in this fashion.
-
FIG. 6 is an illustration of the visual gyroscope capturing a second frame following a user orientation rotation from an initial direction in accordance with an embodiment of the present invention. If the procedure determines that the user has panned around sufficiently and that there are enough matching visual features between the second frame and the initial keyframe, the visual gyroscope module will promoteframe 523 atuser direction 527 to keyframe status. - In order to determine if the user has panned around sufficiently, the visual gyroscope procedure matches features between the current frame and the prior keyframe and then computes a rotation between the two if the features match. Stated differently, the procedure computes a relative rotation between the two consecutive frames from the differences in position of a matching set of feature points in the two images. In one embodiment, the procedure may build a grid for faster matching. This way only grid cells can be matched to grid cells rather than brute-force matching of the entire set of features. Matching features from the prior keyframe to the new frame allows the procedure to determine which locations on the sphere map to new locations. When matches are found, the visual gyroscope procedure can use Horn's procedure with RANSAC, as will be explained below, to estimate a pure rotation from the matched points.
- The vision data from the camera enables the visual gyroscope to determine an approximation of the user rotation. In one embodiment, the well-known Random Sample Consensus (RANSAC) procedure can be used to match a frame to a prior keyframe. Horn's procedure as described in the following: “B. Horn. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America, 1987”, all of which is incorporated herein by reference, is used to compute a rotation between two sets of three points, which are then used for RANSAC sampling. Also Horn's method demonstrates how to compute a rotation between two sets of all matched points. This is used to then compute the final rotation between frames once the RANSAC procedure has provided information regarding if the rotation computed with the two sets of three points provides enough inliers for all points. While RANSAC and Horn's procedure can be used to determine the rotation between keyframes, the embodiments of the present invention are not limited to solely these procedures. For example, in one embodiment, changes or deltas in absolute sensor orientation received from the IMU sensor can also be used to approximate user rotation.
- In one embodiment, keyframes are captured at approximately every 20 degrees of user rotation. If the procedure determines that the user has panned above a certain threshold, e.g., 20 degrees, and that there are enough matching visual features in captured
frame 523, it will promote the captured frame to a keyframe status. Conversely, if it is determined that the user has not panned a distance sufficiently far frominitial keyframe 522, the procedure will not promote the captured frame to a keyframe. The new keyframe will mach to the nearest prior keyframe based, in one embodiment, on a dot product lookup. If, however, for example, no keyframe is near, then it will match to the last frame and save that as a keyframe, if possible. - In one embodiment, a “bundle adjustment” is performed when a new keyframe is added to the map. A bundle adjustment, which is a well-known method, comprises globally adjusting every keyframe to minimize orientation error of each keyframe every time a new keyframe is added to the map. Global alignment (bundle adjustment) is based on the difference between the angle of neighboring keyframes and what a brute force match provides as an angle.
-
FIG. 7 is an illustration of the visual gyroscope starting a new map based on IMU sensor data in accordance with an embodiment of the present invention. Continuing with visual tracking, the visual gyroscope is able to save additional keyframes beyondkeyframe 523. As mentioned before, the visual gyroscope is initialized to the absolute orientation of the device as read from the IMU sensor 420 on start-up. Accordingly, the visual gyroscope is able to continually update the orientation as the user moves if the environment permits. - However, sometimes vision-based tracking may fail for any of several reasons, e.g., insufficient texture in environment, not enough landmarks available, or highly dynamic content as a result of a user panning too quickly etc. In
FIG. 7 , for instance, the user has lost visual tracking because panning too fast results in motion blur created in the visual data captured by the camera. As a result of motion blur, the camera is no longer able to match features to prior frames and therefore loses its orientation. - When visual tracking is lost, the procedure starts a new map with
keyframe 703 using data from IMU sensor 420. At this point, the absolute sensor orientation and the visual gyroscope orientation may show different readings and, occasionally, even vastly different readings even though they both were initialized with the same orientation value because of the drift in the IMU sensor 420. Thus, instead of using the absolute sensor orientation, the visual gyroscope uses the deltas (or differences) of absolute sensor orientation to calculate a relative orientation traveled, which is to be combined into the visual gyroscope orientation reading from where the visual tracking fails. For example, inFIG. 7 , assuming visual tracking failed atuser direction 527 right afterkeyframe 523 was captured and the user pans around to user direction 528, then the relative orientation difference betweenuser direction 527 and user direction 528 is calculated using delta values from the absolute sensor orientation. The relative orientation is then combined with the visual gyroscope orientation reading obtained atuser direction 527 to determine the orientation at user direction 528. This creates a smooth experience and the user would not see the transition from visual tracking to sensor tracking. - After determining orientation by combining the last known visual gyroscope reading with the delta values from the absolute sensor orientation, the visual gyroscope starts a new map at user direction 528 by capturing
keyframe 703. -
FIG. 8 is an illustration of the visual gyroscope creating a new map based on IMU sensor data until an overlap is found with the map created from vision-based data in accordance with an embodiment of the present invention. As shown inFIG. 8 , a new map is created comprisingkeyframes 808 using IMU sensor 420 data when visual tracking fails. Because of feature tracking in each keyframe, the visual gyroscope can determine if the user has panned back to a location for which keyframes based on visual tracking already exist. In other words, the visual gyroscope can recognize when there is an overlap between the initial map created using visual tracking (comprisingkeyframes 522 and 523) and the newmap comprising keyframes 808. - In one embodiment, when the overlap is found, the visual gyroscope deletes the secondary map created using sensor data, e.g., map comprising frames 808. In other words, the sensor data is only used when visual tracking has failed. When the procedure recognizes a keyframe from before that was created using visual tracking, it immediately reverts back to visual tracking and discontinues use of sensor tracking at that time. Moreover, it deletes the map obtained through sensor tracking.
- In a different embodiment, however, when an overlap is found, the visual gyroscope will merge the new map created through sensor tracking with the prior map created using visual tracking. The combined map will then comprise the prior
map comprising keyframes map comprising keyframes 808. - In one embodiment of the present invention, the visual gyroscope is able to turn on certain rejection zones in the camera's field of view. Rejection zones are areas in the spherical map that are not allowed to save keyframes. This is important because the map can experience significant drift if keyframes are saved based on features that are too close to the viewer. Thus, the visual gyroscope turns on dead-zones for angles that are pointed down, e.g., the ground. Also it may be turned on in areas that are featureless, e.g., the sky. Accordingly, for the rejection zones, precision is not important, and, therefore, the visual gyroscope relies on the tracking from the IMU sensor 420.
- In one embodiment, the present invention takes advantage of the fact that many scenes can be approximated as objects at infinity, e.g., in a panoramic model. This simplification is leveraged by the visual gyroscope in order to simplify the procedures it is implementing.
-
FIG. 9 depicts a flowchart of an exemplary computer implement process of tracking device orientation using visual tracking in accordance with one embodiment of the present invention. While the various steps in this flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps can be executed in different orders and some or all of the steps can be executed in parallel. Further, in one or more embodiments of the invention, one or more of the steps described below can be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown inFIG. 9 should not be construed as limiting the scope of the invention. Rather, it will be apparent to persons skilled in the relevant art(s) from the teachings provided herein that other functional flows are within the scope and spirit of the present invention.Flowchart 900 may be described with continued reference to exemplary embodiments described above, though the method is not limited to those embodiments. - At step 910, the device orientation is initialized using absolute sensor orientation. At
step 912, the camera on the handheld device captures a frame from the camera or any other visual capture device. Atstep 914, the features in the frame are determined using procedures such as FAST AND BRIEF as explained above. Atstep 916, the frame is matched to a prior keyframe and a rotation is computed between the current frame and prior keyframe. In one embodiment, RANSAC and Horn's procedure are used to perform the matching and rotation computation. If it is found that the user has rotated orientation of the handheld device over a certain threshold, e.g., 20 degrees, then the current frame is promoted to a keyframe. It should be noted, however, that there are other considerations that, in one embodiment, may also be taken into account before promoting a frame to a keyframe status, e.g., ascertaining that the frame has enough visual features, that the frame is not in a restricted zone, and also that it is at an appropriate distance away from another key-frame. Finally, at step 920, bundle adjustment is performed on the newly added keyframe with all the other keyframes. -
FIG. 10 depicts a flowchart of an exemplary computer implemented process of using sensors for tracking device orientation when visual tracking is temporarily lost in accordance with one embodiment of the present invention. While the various steps in this flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps can be executed in different orders and some or all of the steps can be executed in parallel. Further, in one or more embodiments of the invention, one or more of the steps described below can be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown inFIG. 10 should not be construed as limiting the scope of the invention. Rather, it will be apparent to persons skilled in the relevant art(s) from the teachings provided herein that other functional flows are within the scope and spirit of the present invention.Flowchart 1000 may be described with continued reference to exemplary embodiments described above, though the method is not limited to those embodiments. - At
step 1012, after visual tracking is lost, embodiments of the present invention determine device orientation by combining delta values obtained from absolute sensor orientation to a last computed orientation reading from the visual gyroscope, wherein the last computed orientation reading is based on visual tracking. - At
step 1013, a new keyframe is saved based on the delta values from the IMU sensor. This acts as the initial keyframe in a new orientation map that is generated based on IMU sensor data. In other words, relative orientation between the first map based on visual tracking and the second map is calculated using IMU delta values. Accordingly, at step 10 14 a new map is built based on the sensor data. It should be noted that this new orientation map based on IMU sensor data is only built if visual tracking is lost and the visual gyroscope module that the incoming new frames are not close enough to match to the first map. In other words, the second orientation map is only created if the prior map is “lost.” - At
step 1015, as new keyframes are added to the new orientation map, the visual gyroscope maintains matching features to determine if there is an overlap between the prior map based on visual tracking and the new map based on sensor data. - At step 1016, when an overlap is found, in one embodiment, the new map based on sensor data is deleted and the visual gyroscope continues to build the prior map based on visual tracking data. In a different embodiment, however, when an overlap is found, the prior map and the new map based on sensor data are merged and the visual gyroscope continues to build the map based on visual tracking data.
- While the foregoing disclosure sets forth various embodiments using specific block diagrams, flowcharts, and examples, each block diagram component, flowchart step, operation, and/or component described and/or illustrated herein may be implemented, individually and/or collectively, using a wide range of hardware, software, or firmware (or any combination thereof) configurations. In addition, any disclosure of components contained within other components should be considered as examples because many other architectures can be implemented to achieve the same functionality.
- The process parameters and sequence of steps described and/or illustrated herein are given by way of example only. For example, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various example methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.
- While various embodiments have been described and/or illustrated herein in the context of fully functional computing systems, one or more of these example embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. These software modules may configure a computing system to perform one or more of the example embodiments disclosed herein. One or more of the software modules disclosed herein may be implemented in a cloud computing environment. Cloud computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a Web browser or other remote interface. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.
- The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as may be suited to the particular use contemplated.
- Embodiments according to the invention are thus described. While the present disclosure has been described in particular embodiments, it should be appreciated that the invention should not be construed as limited by such embodiments, but rather construed according to the below claims.
Claims (23)
1. A method for tracking device orientation on a portable device, said method comprising:
initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;
initiating visual tracking using a camera on said portable device;
capturing a frame using said camera;
determining a plurality of visual features in said frame;
matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;
computing a rotation amount between said frame and said keyframe;
responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and
performing an adjustment of said frame with all prior captured keyframes.
2. The method of claim 1 , wherein said determining comprises:
computing repeatable and distinct features in said frame using a feature detecting procedure; and
detecting and describing said features using a feature description procedure.
3. The method of claim 2 , wherein said feature detecting procedure is selected from a group consisting of: FAST, Harris & Stephens, Plessey, Shi-Tomasi, Moravec corner detection, Wang and Brady corner detection, and SUSAN corner detector.
4. The method of claim 2 , wherein said feature description procedure is selected from a group consisting of: SURF, SIFT and BRIEF.
5. The method of claim 1 , further comprising:
responsive to a determination that a rotational distance between said frame and said keyframe is within said predetermined threshold, continuing to search for next keyframe.
6. The method of claim 1 , wherein said matching is performed using RANSAC.
7. The method of claim 1 , wherein said computing is performed using Horn's procedure.
8. The method of claim 1 further comprising:
responsive to a loss of said visual tracking, determining device orientation by combining delta values from said IMU sensor with a last known orientation measurement from said visual tracking; and
building a second orientation map, wherein said second orientation map is created using data from said IMU sensor.
9. The method of claim 8 , further comprising:
determining if said second orientation map overlaps with keyframes from said orientation map; and
responsive to a determination of overlap, deleting said second orientation map and continuing to build said first orientation map.
10. The method of claim 8 , further comprising:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of overlap, merging said second orientation map with said first orientation map.
11. The method of claim 8 , further comprising:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of no overlap, continuing to build said second orientation map.
12. A computer-readable storage medium having stored thereon instructions that, if executed by a computer system cause the computer system to perform a method for tracking device orientation on a portable device, said method comprising:
initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;
initiating visual tracking using a camera on said portable device;
capturing a frame using said camera;
determining a plurality of visual features in said frame;
matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;
computing a rotation amount between said frame and said keyframe;
responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and
performing an adjustment of said frame with all prior captured keyframes.
13. The computer-readable medium as described in claim 12 , wherein said determining comprises:
computing repeatable and distinct features in said frame using a feature detecting procedure; and
detecting and describing said features using a feature description procedure.
14. The computer-readable medium as described in claim 13 , wherein said feature detecting procedure is selected from a group consisting of: FAST, Harris & Stephens, Plessey, Shi-Tomasi, Moravec corner detection, Wang and Brady corner detection, and SUSAN corner detector.
15. The computer-readable medium as described in claim 13 , wherein said feature description procedure is selected from a group consisting of: SURF, SIFT and BRIEF.
16. The computer-readable medium as described in claim 12 , further comprising:
responsive to a determination that a rotational distance between said frame and said keyframe is within said predetermined threshold, continuing to search for next keyframe.
17. The computer-readable medium as described in claim 12 , wherein said matching is performed using RANSAC.
18. The computer-readable medium as described in claim 12 , wherein said computing is performed using Horn's procedure.
19. The computer-readable medium as described in claim 12 , wherein said method further comprises:
responsive to a loss of said visual tracking, determining device orientation by combining delta values from said IMU sensor with a last known orientation measurement from said visual tracking; and
building a second orientation map, wherein said second orientation map is created using data from said IMU sensor.
20. The computer-readable medium as described in claim 19 , wherein said method further comprises:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of overlap, deleting said second orientation map and continuing to build said first orientation map.
21. The computer-readable medium as described in claim 19 , further comprising:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of overlap, merging said second orientation map with said first orientation map.
22. The computer-readable medium as described in claim 19 , further comprising:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of no overlap, continuing to build said second orientation map.
23. A system for tracking device orientation on a portable device, said system comprising:
a display screen;
a memory;
a camera; and
a processor configured to implement a visual gyroscope, wherein said visual gyroscope performs a method for tracking device orientation on said portable device, wherein said method comprises:
initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;
initiating visual tracking using a camera on said portable device;
capturing a frame using said camera;
determining a plurality of visual features in said frame;
matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;
computing a rotation amount between said frame and said keyframe;
responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and
performing an adjustment of said frame with all prior captured keyframes.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/051,282 US20150103183A1 (en) | 2013-10-10 | 2013-10-10 | Method and apparatus for device orientation tracking using a visual gyroscope |
US14/051,307 US9479709B2 (en) | 2013-10-10 | 2013-10-10 | Method and apparatus for long term image exposure with image stabilization on a mobile device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/051,282 US20150103183A1 (en) | 2013-10-10 | 2013-10-10 | Method and apparatus for device orientation tracking using a visual gyroscope |
Publications (1)
Publication Number | Publication Date |
---|---|
US20150103183A1 true US20150103183A1 (en) | 2015-04-16 |
Family
ID=52809333
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/051,282 Abandoned US20150103183A1 (en) | 2013-10-10 | 2013-10-10 | Method and apparatus for device orientation tracking using a visual gyroscope |
US14/051,307 Active US9479709B2 (en) | 2013-10-10 | 2013-10-10 | Method and apparatus for long term image exposure with image stabilization on a mobile device |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/051,307 Active US9479709B2 (en) | 2013-10-10 | 2013-10-10 | Method and apparatus for long term image exposure with image stabilization on a mobile device |
Country Status (1)
Country | Link |
---|---|
US (2) | US20150103183A1 (en) |
Cited By (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150103193A1 (en) * | 2013-10-10 | 2015-04-16 | Nvidia Corporation | Method and apparatus for long term image exposure with image stabilization on a mobile device |
US20150235378A1 (en) * | 2014-02-17 | 2015-08-20 | Samsung Electronics Co., Ltd. | Apparatus and method for estimating camera pose |
CN105049674A (en) * | 2015-07-01 | 2015-11-11 | 中科创达软件股份有限公司 | Video image processing method and system |
US20160364867A1 (en) * | 2015-06-11 | 2016-12-15 | Fujitsu Limited | Camera pose estimation device and control method |
US9704265B2 (en) * | 2014-12-19 | 2017-07-11 | SZ DJI Technology Co., Ltd. | Optical-flow imaging system and method using ultrasonic depth sensing |
CN108111760A (en) * | 2017-12-26 | 2018-06-01 | 北京理工大学 | A kind of electronic image stabilization method and system |
JP2018124067A (en) * | 2017-01-30 | 2018-08-09 | 日本電信電話株式会社 | Azimuth estimating device, photographing/map display device, method, and program |
CN109509206A (en) * | 2017-09-11 | 2019-03-22 | 苏宁云商集团股份有限公司 | The localization method and system of superposition of data in a kind of augmented reality |
US10262462B2 (en) * | 2014-04-18 | 2019-04-16 | Magic Leap, Inc. | Systems and methods for augmented and virtual reality |
TWI660156B (en) * | 2017-04-28 | 2019-05-21 | 宏達國際電子股份有限公司 | Tracking system and method thereof |
CN109863510A (en) * | 2016-10-24 | 2019-06-07 | 斯纳普公司 | Redundancy tracking system |
CN110060277A (en) * | 2019-04-30 | 2019-07-26 | 哈尔滨理工大学 | A kind of vision SLAM method of multiple features fusion |
CN110168476A (en) * | 2017-01-09 | 2019-08-23 | 斯纳普公司 | Augmented reality object manipulation |
US10404915B1 (en) * | 2016-04-07 | 2019-09-03 | Scott Zhihao Chen | Method and system for panoramic video image stabilization |
KR20190103322A (en) * | 2017-01-09 | 2019-09-04 | 스냅 인코포레이티드 | Surface recognition lens |
CN110832546A (en) * | 2017-07-07 | 2020-02-21 | 三星电子株式会社 | System and method for device tracking |
US10593116B2 (en) | 2016-10-24 | 2020-03-17 | Snap Inc. | Augmented reality object manipulation |
US10643347B2 (en) * | 2016-02-29 | 2020-05-05 | Canon Kabushiki Kaisha | Device for measuring position and orientation of imaging apparatus and method therefor |
US20200167953A1 (en) * | 2017-07-28 | 2020-05-28 | Qualcomm Incorporated | Image Sensor Initialization in a Robotic Vehicle |
KR20210063995A (en) * | 2019-11-25 | 2021-06-02 | 한국전자통신연구원 | System, apparatus and method for recognizing motion of multi-users |
US11030813B2 (en) | 2018-08-30 | 2021-06-08 | Snap Inc. | Video clip object tracking |
US11189098B2 (en) | 2019-06-28 | 2021-11-30 | Snap Inc. | 3D object camera customization system |
US11210850B2 (en) | 2018-11-27 | 2021-12-28 | Snap Inc. | Rendering 3D captions within real-world environments |
US11232646B2 (en) | 2019-09-06 | 2022-01-25 | Snap Inc. | Context-based virtual object rendering |
EP3886048A4 (en) * | 2019-02-01 | 2022-03-02 | Guangzhou Xiaopeng Motors Technology Co., Ltd. | Slam map joining method and system |
US20220178692A1 (en) * | 2017-12-21 | 2022-06-09 | Mindmaze Holding Sa | System, method and apparatus of a motion sensing stack with a camera system |
US11429337B2 (en) * | 2017-02-08 | 2022-08-30 | Immersive Robotics Pty Ltd | Displaying content to users in a multiplayer venue |
US11501499B2 (en) | 2018-12-20 | 2022-11-15 | Snap Inc. | Virtual surface modification |
US11553187B2 (en) | 2017-11-21 | 2023-01-10 | Immersive Robotics Pty Ltd | Frequency component selection for image compression |
US20230117498A1 (en) * | 2020-10-16 | 2023-04-20 | SLAMcore Limited | Visual-inertial localisation in an existing map |
US11636657B2 (en) | 2019-12-19 | 2023-04-25 | Snap Inc. | 3D captions with semantic graphical elements |
US11810220B2 (en) | 2019-12-19 | 2023-11-07 | Snap Inc. | 3D captions with face tracking |
US11973979B2 (en) | 2021-09-30 | 2024-04-30 | Immersive Robotics Pty Ltd | Image compression for digital reality |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2017215664A (en) * | 2016-05-30 | 2017-12-07 | サン電子株式会社 | Terminal device |
JP6762913B2 (en) * | 2017-07-11 | 2020-09-30 | キヤノン株式会社 | Information processing device, information processing method |
CN110211239B (en) * | 2019-05-30 | 2022-11-08 | 杭州远传新业科技股份有限公司 | Augmented reality method, apparatus, device and medium based on label-free recognition |
US11301967B2 (en) | 2019-08-27 | 2022-04-12 | Samsung Electronics Company, Ltd. | Intelligence-based editing and curating of images |
CN112444820B (en) * | 2019-08-28 | 2023-06-09 | 深圳市优必选科技股份有限公司 | Robot pose determining method and device, readable storage medium and robot |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6970591B1 (en) * | 1999-11-25 | 2005-11-29 | Canon Kabushiki Kaisha | Image processing apparatus |
US20110164810A1 (en) * | 2008-01-04 | 2011-07-07 | Tong Zang | Image signatures for use in motion-based three-dimensional reconstruction |
WO2013167901A1 (en) * | 2012-05-09 | 2013-11-14 | Ncam Technologies Limited | A system for mixing or compositing in real-time, computer generated 3d objects and a video feed from a film camera |
WO2014020317A1 (en) * | 2012-07-30 | 2014-02-06 | Sony Computer Entertainment Europe Limited | Localisation and mapping |
US20140285619A1 (en) * | 2012-06-25 | 2014-09-25 | Adobe Systems Incorporated | Camera tracker target user interface for plane detection and object creation |
US20140320593A1 (en) * | 2013-04-30 | 2014-10-30 | Qualcomm Incorporated | Monocular visual slam with general and panorama camera movements |
Family Cites Families (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6134346A (en) | 1998-01-16 | 2000-10-17 | Ultimatte Corp | Method for removing from an image the background surrounding a selected object |
JP4596220B2 (en) * | 2001-06-26 | 2010-12-08 | ソニー株式会社 | Image processing apparatus and method, recording medium, and program |
US8698924B2 (en) * | 2007-03-05 | 2014-04-15 | DigitalOptics Corporation Europe Limited | Tone mapping for low-light video frame enhancement |
US8896725B2 (en) * | 2007-06-21 | 2014-11-25 | Fotonation Limited | Image capture device with contemporaneous reference image capture mechanism |
US7071948B2 (en) * | 2003-10-21 | 2006-07-04 | Adobe Systems Incorporated | Adjusting images based on previous modifications |
US20050129324A1 (en) | 2003-12-02 | 2005-06-16 | Lemke Alan P. | Digital camera and method providing selective removal and addition of an imaged object |
GB0409463D0 (en) | 2004-04-28 | 2004-06-02 | Ibm | Method for removal of moving objects from a video stream |
TWI241127B (en) | 2004-08-27 | 2005-10-01 | Univ Nat Cheng Kung | Image-capturing device and method for removing strangers |
EP1944732A3 (en) * | 2007-01-12 | 2010-01-27 | Sanyo Electric Co., Ltd. | Apparatus and method for blur detection, and apparatus and method for blur correction |
EP2051205B1 (en) | 2007-10-17 | 2012-03-14 | Deutsches Krebsforschungszentrum | Method, computer program and workstation for removing undesirable objects for a digital medical image |
CA2707104C (en) * | 2007-11-30 | 2018-06-19 | Searidge Technologies Inc. | Airport target tracking system |
JP5213670B2 (en) * | 2008-01-16 | 2013-06-19 | 三洋電機株式会社 | Imaging apparatus and blur correction method |
US8340351B2 (en) | 2009-07-01 | 2012-12-25 | Texas Instruments Incorporated | Method and apparatus for eliminating unwanted objects from a streaming image |
US8823808B2 (en) * | 2009-10-27 | 2014-09-02 | Intellectual Ventures Fund 83 Llc | Method for improved digital video image quality |
US20120320237A1 (en) | 2011-06-15 | 2012-12-20 | Wei-Ting Liu | Camera with function of removing unwanted object and method thereof |
US8885941B2 (en) * | 2011-09-16 | 2014-11-11 | Adobe Systems Incorporated | System and method for estimating spatially varying defocus blur in a digital image |
CN103514583B (en) * | 2012-06-30 | 2016-08-24 | 华为技术有限公司 | Image sharpening method and equipment |
US9349218B2 (en) * | 2012-07-26 | 2016-05-24 | Qualcomm Incorporated | Method and apparatus for controlling augmented reality |
KR101926490B1 (en) * | 2013-03-12 | 2018-12-07 | 한화테크윈 주식회사 | Apparatus and method for processing image |
US20140294366A1 (en) | 2013-04-01 | 2014-10-02 | Michael-Ryan FLETCHALL | Capture, Processing, And Assembly Of Immersive Experience |
US9564175B2 (en) * | 2013-04-02 | 2017-02-07 | International Business Machines Corporation | Clustering crowdsourced videos by line-of-sight |
US20150103183A1 (en) * | 2013-10-10 | 2015-04-16 | Nvidia Corporation | Method and apparatus for device orientation tracking using a visual gyroscope |
-
2013
- 2013-10-10 US US14/051,282 patent/US20150103183A1/en not_active Abandoned
- 2013-10-10 US US14/051,307 patent/US9479709B2/en active Active
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6970591B1 (en) * | 1999-11-25 | 2005-11-29 | Canon Kabushiki Kaisha | Image processing apparatus |
US20110164810A1 (en) * | 2008-01-04 | 2011-07-07 | Tong Zang | Image signatures for use in motion-based three-dimensional reconstruction |
WO2013167901A1 (en) * | 2012-05-09 | 2013-11-14 | Ncam Technologies Limited | A system for mixing or compositing in real-time, computer generated 3d objects and a video feed from a film camera |
GB2503563A (en) * | 2012-05-09 | 2014-01-01 | Ncam Technologies Ltd | Combining computer generated objects with real video |
US20150084951A1 (en) * | 2012-05-09 | 2015-03-26 | Ncam Technologies Limited | System for mixing or compositing in real-time, computer generated 3d objects and a video feed from a film camera |
US20140285619A1 (en) * | 2012-06-25 | 2014-09-25 | Adobe Systems Incorporated | Camera tracker target user interface for plane detection and object creation |
WO2014020317A1 (en) * | 2012-07-30 | 2014-02-06 | Sony Computer Entertainment Europe Limited | Localisation and mapping |
GB2506338A (en) * | 2012-07-30 | 2014-04-02 | Sony Comp Entertainment Europe | A method of localisation and mapping |
US20150209673A1 (en) * | 2012-07-30 | 2015-07-30 | Sony Computer Entertainment Europe Limited | Localisation and mapping |
US20140320593A1 (en) * | 2013-04-30 | 2014-10-30 | Qualcomm Incorporated | Monocular visual slam with general and panorama camera movements |
Cited By (68)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150103193A1 (en) * | 2013-10-10 | 2015-04-16 | Nvidia Corporation | Method and apparatus for long term image exposure with image stabilization on a mobile device |
US9479709B2 (en) * | 2013-10-10 | 2016-10-25 | Nvidia Corporation | Method and apparatus for long term image exposure with image stabilization on a mobile device |
US9916689B2 (en) * | 2014-02-17 | 2018-03-13 | Samsung Electronics Co., Ltd. | Apparatus and method for estimating camera pose |
US20150235378A1 (en) * | 2014-02-17 | 2015-08-20 | Samsung Electronics Co., Ltd. | Apparatus and method for estimating camera pose |
US10665018B2 (en) | 2014-04-18 | 2020-05-26 | Magic Leap, Inc. | Reducing stresses in the passable world model in augmented or virtual reality systems |
US10909760B2 (en) | 2014-04-18 | 2021-02-02 | Magic Leap, Inc. | Creating a topological map for localization in augmented or virtual reality systems |
US10846930B2 (en) | 2014-04-18 | 2020-11-24 | Magic Leap, Inc. | Using passable world model for augmented or virtual reality |
US10825248B2 (en) | 2014-04-18 | 2020-11-03 | Magic Leap, Inc. | Eye tracking systems and method for augmented or virtual reality |
US10262462B2 (en) * | 2014-04-18 | 2019-04-16 | Magic Leap, Inc. | Systems and methods for augmented and virtual reality |
US11205304B2 (en) | 2014-04-18 | 2021-12-21 | Magic Leap, Inc. | Systems and methods for rendering user interfaces for augmented or virtual reality |
US9704265B2 (en) * | 2014-12-19 | 2017-07-11 | SZ DJI Technology Co., Ltd. | Optical-flow imaging system and method using ultrasonic depth sensing |
US20160364867A1 (en) * | 2015-06-11 | 2016-12-15 | Fujitsu Limited | Camera pose estimation device and control method |
US10088294B2 (en) * | 2015-06-11 | 2018-10-02 | Fujitsu Limited | Camera pose estimation device and control method |
CN105049674A (en) * | 2015-07-01 | 2015-11-11 | 中科创达软件股份有限公司 | Video image processing method and system |
US10643347B2 (en) * | 2016-02-29 | 2020-05-05 | Canon Kabushiki Kaisha | Device for measuring position and orientation of imaging apparatus and method therefor |
US10404915B1 (en) * | 2016-04-07 | 2019-09-03 | Scott Zhihao Chen | Method and system for panoramic video image stabilization |
KR20190075977A (en) * | 2016-10-24 | 2019-07-01 | 스냅 인코포레이티드 | Duplicate tracking system |
KR102491191B1 (en) * | 2016-10-24 | 2023-01-20 | 스냅 인코포레이티드 | Redundant tracking system |
US11481978B2 (en) | 2016-10-24 | 2022-10-25 | Snap Inc. | Redundant tracking system |
EP3529750A4 (en) * | 2016-10-24 | 2019-10-16 | Snap Inc. | Redundant tracking system |
KR20210129273A (en) * | 2016-10-24 | 2021-10-27 | 스냅 인코포레이티드 | Redundant tracking system |
US10593116B2 (en) | 2016-10-24 | 2020-03-17 | Snap Inc. | Augmented reality object manipulation |
CN109863510A (en) * | 2016-10-24 | 2019-06-07 | 斯纳普公司 | Redundancy tracking system |
KR20230014886A (en) * | 2016-10-24 | 2023-01-30 | 스냅 인코포레이티드 | Redundant tracking system |
KR102317167B1 (en) * | 2016-10-24 | 2021-10-27 | 스냅 인코포레이티드 | Duplicate Tracking System |
US10692285B2 (en) | 2016-10-24 | 2020-06-23 | Snap Inc. | Redundant tracking system |
US10803664B2 (en) | 2016-10-24 | 2020-10-13 | Snap Inc. | Redundant tracking system |
US11580700B2 (en) | 2016-10-24 | 2023-02-14 | Snap Inc. | Augmented reality object manipulation |
EP4296827A1 (en) * | 2016-10-24 | 2023-12-27 | Snap Inc. | Redundant tracking system |
KR102643741B1 (en) * | 2016-10-24 | 2024-03-06 | 스냅 인코포레이티드 | Redundant tracking system |
CN110168476A (en) * | 2017-01-09 | 2019-08-23 | 斯纳普公司 | Augmented reality object manipulation |
KR20210063442A (en) * | 2017-01-09 | 2021-06-01 | 스냅 인코포레이티드 | Surface aware lens |
KR20190103322A (en) * | 2017-01-09 | 2019-09-04 | 스냅 인코포레이티드 | Surface recognition lens |
KR102257167B1 (en) * | 2017-01-09 | 2021-05-27 | 스냅 인코포레이티드 | Surface recognition lens |
KR102367928B1 (en) * | 2017-01-09 | 2022-03-02 | 스냅 인코포레이티드 | Surface aware lens |
US11195338B2 (en) | 2017-01-09 | 2021-12-07 | Snap Inc. | Surface aware lens |
US11704878B2 (en) | 2017-01-09 | 2023-07-18 | Snap Inc. | Surface aware lens |
JP2018124067A (en) * | 2017-01-30 | 2018-08-09 | 日本電信電話株式会社 | Azimuth estimating device, photographing/map display device, method, and program |
US11429337B2 (en) * | 2017-02-08 | 2022-08-30 | Immersive Robotics Pty Ltd | Displaying content to users in a multiplayer venue |
TWI660156B (en) * | 2017-04-28 | 2019-05-21 | 宏達國際電子股份有限公司 | Tracking system and method thereof |
CN110832546A (en) * | 2017-07-07 | 2020-02-21 | 三星电子株式会社 | System and method for device tracking |
US20200167953A1 (en) * | 2017-07-28 | 2020-05-28 | Qualcomm Incorporated | Image Sensor Initialization in a Robotic Vehicle |
US11080890B2 (en) * | 2017-07-28 | 2021-08-03 | Qualcomm Incorporated | Image sensor initialization in a robotic vehicle |
CN109509206A (en) * | 2017-09-11 | 2019-03-22 | 苏宁云商集团股份有限公司 | The localization method and system of superposition of data in a kind of augmented reality |
US11553187B2 (en) | 2017-11-21 | 2023-01-10 | Immersive Robotics Pty Ltd | Frequency component selection for image compression |
US20220178692A1 (en) * | 2017-12-21 | 2022-06-09 | Mindmaze Holding Sa | System, method and apparatus of a motion sensing stack with a camera system |
CN108111760A (en) * | 2017-12-26 | 2018-06-01 | 北京理工大学 | A kind of electronic image stabilization method and system |
US11715268B2 (en) | 2018-08-30 | 2023-08-01 | Snap Inc. | Video clip object tracking |
US11030813B2 (en) | 2018-08-30 | 2021-06-08 | Snap Inc. | Video clip object tracking |
US11210850B2 (en) | 2018-11-27 | 2021-12-28 | Snap Inc. | Rendering 3D captions within real-world environments |
US11620791B2 (en) | 2018-11-27 | 2023-04-04 | Snap Inc. | Rendering 3D captions within real-world environments |
US11836859B2 (en) | 2018-11-27 | 2023-12-05 | Snap Inc. | Textured mesh building |
US20220044479A1 (en) | 2018-11-27 | 2022-02-10 | Snap Inc. | Textured mesh building |
US11501499B2 (en) | 2018-12-20 | 2022-11-15 | Snap Inc. | Virtual surface modification |
EP3886048A4 (en) * | 2019-02-01 | 2022-03-02 | Guangzhou Xiaopeng Motors Technology Co., Ltd. | Slam map joining method and system |
CN110060277A (en) * | 2019-04-30 | 2019-07-26 | 哈尔滨理工大学 | A kind of vision SLAM method of multiple features fusion |
US11823341B2 (en) | 2019-06-28 | 2023-11-21 | Snap Inc. | 3D object camera customization system |
US11443491B2 (en) | 2019-06-28 | 2022-09-13 | Snap Inc. | 3D object camera customization system |
US11189098B2 (en) | 2019-06-28 | 2021-11-30 | Snap Inc. | 3D object camera customization system |
US11232646B2 (en) | 2019-09-06 | 2022-01-25 | Snap Inc. | Context-based virtual object rendering |
KR102564849B1 (en) | 2019-11-25 | 2023-08-09 | 한국전자통신연구원 | System, apparatus and method for recognizing motion of multi-users |
KR20210063995A (en) * | 2019-11-25 | 2021-06-02 | 한국전자통신연구원 | System, apparatus and method for recognizing motion of multi-users |
US11810220B2 (en) | 2019-12-19 | 2023-11-07 | Snap Inc. | 3D captions with face tracking |
US11636657B2 (en) | 2019-12-19 | 2023-04-25 | Snap Inc. | 3D captions with semantic graphical elements |
US11908093B2 (en) | 2019-12-19 | 2024-02-20 | Snap Inc. | 3D captions with semantic graphical elements |
US11830218B2 (en) * | 2020-10-16 | 2023-11-28 | SLAMcore Limited | Visual-inertial localisation in an existing map |
US20230117498A1 (en) * | 2020-10-16 | 2023-04-20 | SLAMcore Limited | Visual-inertial localisation in an existing map |
US11973979B2 (en) | 2021-09-30 | 2024-04-30 | Immersive Robotics Pty Ltd | Image compression for digital reality |
Also Published As
Publication number | Publication date |
---|---|
US20150103193A1 (en) | 2015-04-16 |
US9479709B2 (en) | 2016-10-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20150103183A1 (en) | Method and apparatus for device orientation tracking using a visual gyroscope | |
JP6043856B2 (en) | Head pose estimation using RGBD camera | |
EP3420530B1 (en) | A device and method for determining a pose of a camera | |
US9031283B2 (en) | Sensor-aided wide-area localization on mobile devices | |
US9406137B2 (en) | Robust tracking using point and line features | |
JP5950973B2 (en) | Method, apparatus and system for selecting a frame | |
CN108805917B (en) | Method, medium, apparatus and computing device for spatial localization | |
US9135514B2 (en) | Real time tracking/detection of multiple targets | |
US9824450B2 (en) | Localisation and mapping | |
EP2640057B1 (en) | Image processing device, image processing method and program | |
US10154190B2 (en) | Balancing exposure and gain at an electronic device based on device motion and scene distance | |
CN110148178B (en) | Camera positioning method, device, terminal and storage medium | |
EP3206163A1 (en) | Image processing method, mobile device and method for generating a video image database | |
US11042984B2 (en) | Systems and methods for providing image depth information | |
US9582896B2 (en) | Line tracking with automatic model initialization by graph matching and cycle detection | |
CN112087728A (en) | Method and device for acquiring Wi-Fi fingerprint spatial distribution and electronic equipment | |
CN111489376A (en) | Method and device for tracking interactive equipment, terminal equipment and storage medium | |
US20230162375A1 (en) | Method and system for improving target detection performance through dynamic learning | |
CN116862979A (en) | Repositioning method and related equipment | |
CN114422736A (en) | Video processing method, electronic equipment and computer storage medium | |
CN112818748A (en) | Method and device for determining plane in video, storage medium and electronic equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: NVIDIA CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ABBOTT, JOSH;HEINLY, JARED;VAN WELZEN, JIM;REEL/FRAME:031385/0230 Effective date: 20131008 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |