WO2017088678A1

WO2017088678A1 - Long-exposure panoramic image shooting apparatus and method

Info

Publication number: WO2017088678A1
Application number: PCT/CN2016/105705
Authority: WO
Inventors: 李嵩
Original assignee: 努比亚技术有限公司
Priority date: 2015-11-24
Filing date: 2016-11-14
Publication date: 2017-06-01
Also published as: CN105430263A

Abstract

Disclosed is a long-exposure panoramic image shooting apparatus. The apparatus comprises: an image acquisition module configured to acquire an image regularly collected by a mobile terminal viewfinder when receiving a long-exposure shooting instruction; an image projection module configured to project the acquired image to a pre-set projection space to form a corresponding projection image; a registration and fusion module configured to perform position registration and pixel fusion on subsequent projection images at multiple perspectives satisfying a pre-set projection condition one by one with a reference projection image, so as to form a new reference projection image until all the acquired subsequent projection images are traversed; and a back projection module configured to back-project an output projection image to a coordinate space for the mobile terminal viewfinder to capture the image, so as to generate a long-exposure panoramic image. The solution enhances the ability of a mobile terminal to adapt to a jitter in a long-exposure shooting mode, thereby making the use of a long-exposure shooting function of the mobile terminal more convenient.

Description

Long exposure panoramic image capturing device and method

Technical field

The present application relates to, but is not limited to, the field of image processing technologies, and in particular, to a long exposure panoramic image capturing apparatus and method.

Background technique

With the development of electronic technology, mobile terminals such as smart phones and digital cameras on the market generally have long exposure shooting functions. The so-called long exposure shooting function is to open the shutter of the mobile terminal and use the long exposure technology to capture the camera for a long time. The environment, often used to shoot water, nebula star trails, night road light trails, etc., can capture stunning time and space images.

The long-exposure photographing method in the related art is to fix the mobile terminal having the camera by using a fixing device such as a tripod, and then perform exposure shooting. Since the shooting method requires the photographer to carry the fixing device with him, the mobile terminal can not be shaken for a long time, but when the mobile terminal user shoots outdoors, it is very inconvenient to carry a large-sized and heavy-weight fixing device, resulting in The long exposure shooting function of the mobile terminal is inconvenient to use.

Summary of the invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

The main purpose of the present application is to provide a long exposure panoramic image capturing apparatus and method, aiming at solving the technical problem of inconvenient use of the long exposure shooting function of the mobile terminal.

An embodiment of the present invention provides a long exposure panoramic image capturing apparatus, where the long exposure panoramic image capturing apparatus includes:

The image acquisition module is configured to acquire an image acquired by the mobile terminal viewfinder at a time when receiving the long exposure shooting instruction;

An image projection module is configured to project the acquired image to a preset projection space to form a corresponding projection image, wherein the acquired projection image corresponding to the first frame image is used as a reference projection image, The projected image corresponding to the acquired subsequent image is used as a subsequent projected image;

The registration fusion module is configured to perform position registration and pixel fusion of the subsequent projection images of the plurality of viewing angles satisfying the preset projection conditions one by one with the reference projection image to form a new reference projection image until the traversal is obtained. Subsequently projecting an image to obtain an output projected image;

A back projection module is configured to backproject the output projected image to a coordinate space of the image acquired by the mobile terminal viewfinder to generate a long exposure panoramic image.

As an implementation manner, the image projection module includes:

a parameter obtaining unit, configured to acquire a motion parameter of the mobile terminal, where the motion parameter includes a motion direction and a tilt angle;

a space determining unit configured to determine a projection space for image projection according to the acquired motion parameter;

And a projection unit configured to project the acquired image to the determined projection space to form a corresponding projection image.

As an implementation manner, the parameter obtaining unit is configured to acquire a motion parameter of the mobile terminal by:

The parameter acquisition unit controls an accelerometer and a gyroscope installed inside the mobile terminal to sense a movement direction and a tilt angle of the user holding the mobile terminal, or use an image registration method to extract features from the current image, find an alignment position, and determine a user. Hold the moving direction and tilt angle of the mobile terminal.

As an implementation manner, the projection space includes a cylindrical projection space, a spherical projection space, and a cube projection space.

As an implementation manner, the registration fusion module includes:

a feature extraction unit configured to extract a reference feature from the reference projection image, and extract matching features corresponding to the reference feature from the subsequent projection images of the plurality of perspectives satisfying the preset projection condition one by one;

The registration unit is configured to combine the subsequent projection image that meets the preset projection condition with the reference projection image according to the positional relationship between the matching feature and the reference feature;

The merging unit is configured to perform pixel merging and overlapping of the overlapping area of the reference projection image satisfying the preset projection condition with the non-overlapping area to form a new reference projection image until the acquired subsequent projection image is traversed And the resulting reference projection image is taken as an output projection image.

As an embodiment, the feature extraction unit is configured to extract features in one of the following ways: feature point matching, optical flow method, cross-correlation.

As an implementation manner, the registration unit is configured to obtain a registration parameter according to a positional relationship between the matching feature and the reference feature, and the subsequent projection image and the reference projection satisfying the preset projection condition according to the registration parameter Image registration combined.

As an implementation manner, the image acquiring module is further configured to acquire a motion parameter of the mobile terminal when acquiring an image acquired by the mobile terminal viewfinder timing, where the motion parameter includes a moving direction and a tilting angle;

The registration unit is further configured to:

Determining, according to a motion parameter of the mobile terminal, a combination direction of the subsequent projected image that satisfies a preset projection condition and the reference projected image;

And matching the subsequent projection image satisfying the preset projection condition with the reference projection image according to the bonding direction and the positional relationship between the matching feature and the reference feature.

As an implementation manner, the fusion unit is further configured to:

Obtaining an input color value, a reference color value, and a number of fusion times of each pixel position in an overlapping area of the subsequent projection image and the reference projection image that satisfy a preset projection condition;

And adjusting a color value of each pixel in the overlapping area to complete pixel fusion according to the input color value, the reference color value, and the number of fusions, wherein the input color value is a color of a pixel position of the subsequent projected image in the overlapping area. The value and the reference color value are color values of the pixel position of the reference projection image in the overlapping area, and the number of times of fusion is the number of times the pixel of the reference projection image is pixel-fused in the overlapping area.

As an embodiment, the fusion unit is configured to calculate a color value of each pixel in the overlapping region according to a weighted average method.

An embodiment of the present invention further provides a long exposure panoramic image capturing method, the long exposure panoramic image Photo shooting methods include:

Obtaining an image acquired by the mobile terminal viewfinder at a time when receiving a long exposure shooting instruction;

Projecting the acquired image to a preset projection space to form a corresponding projection image, wherein the acquired projection image corresponding to the first frame image is used as a reference projection image, and the acquired projection image corresponding to the subsequent image is used as a subsequent projection image;

The subsequent projection images of the plurality of viewing angles satisfying the preset projection condition are position-aligned and pixel-fused with the reference projection image one by one to form a new reference projection image, until the acquired subsequent projection image is traversed to obtain an output projection image;

The output projection image is back projected to a coordinate space of the mobile terminal viewfinder acquisition image to generate a long exposure panoramic image.

As an implementation manner, the step of projecting the acquired image into a preset projection space to form a corresponding projection image includes:

Obtaining a motion parameter of the mobile terminal, where the motion parameter includes a motion direction and a tilt angle;

Determining a projection space for image projection based on the acquired motion parameters;

The acquired image is projected onto the determined projection space to form a corresponding projected image.

As an implementation manner, the acquiring the motion parameter of the mobile terminal includes:

The accelerometer and the gyroscope installed inside the mobile terminal are controlled to sense the moving direction and the tilting angle of the user holding the mobile terminal, or the image registration method is used to extract the feature from the current image, and the alignment position is found to determine the user holding the mobile terminal. Direction of movement and angle of inclination.

As an implementation manner, the subsequent projection images of the plurality of viewing angles satisfying the preset projection condition are position-aligned and pixel-fused with the reference projection image one by one to form a new reference projection image, and the subsequent steps of the acquisition are traversed. The steps of projecting an image to obtain an output projected image include:

Extracting reference features from the reference projection image, and extracting matching features corresponding to the reference features from the subsequent projection images of the plurality of perspectives satisfying the preset projection conditions one by one;

According to the positional relationship between the matching feature and the reference feature, the preset projection condition will be met. The subsequent projected image is combined with the reference projected image;

Performing pixel fusion on the overlapping area of the predetermined projection image and the reference projection image, and directly splicing the non-overlapping area to form a new reference projection image until the acquired subsequent projection image is traversed;

The resulting reference projection image is taken as an output projection image.

As an implementation manner, the extracting reference features from the reference projection image, and extracting matching features corresponding to the reference features from the subsequent projection images of the plurality of perspectives satisfying the preset projection conditions one by one, including : Feature extraction in one of the following ways: feature point matching, optical flow method, cross-correlation.

In an embodiment, the registering the subsequent projection image that meets the preset projection condition with the reference projection image according to the positional relationship between the matching feature and the reference feature comprises: matching the reference feature and the reference feature The positional relationship between the two results in a registration parameter, and the subsequent projection image that satisfies the preset projection condition is combined with the reference projection image according to the registration parameter.

As an embodiment, the long-exposure panoramic image capturing method further includes: acquiring a motion parameter of the mobile terminal when acquiring an image acquired by the mobile terminal viewfinder timing, where the motion parameter includes a moving direction and a tilting angle;

The step of combining the subsequent projection image satisfying the preset projection condition with the reference projection image according to the positional relationship between the matching feature and the reference feature comprises:

As an implementation manner, the step of performing pixel fusion on the overlapping area of the subsequent projection image that meets the preset projection condition and the reference projection image includes:

And adjusting a color value of each pixel in the overlapping area to complete pixel fusion according to the input color value, the reference color value, and the number of times of fusion, wherein the input color value is heavy in the subsequent projection image The color value of the pixel position in the overlap region, the reference color value is the color value of the pixel position of the reference projection image in the overlapping region, and the number of fusion times is the number of times the pixel of the reference projection image is pixel-fused in the overlapping region.

In one embodiment, the color value of each pixel in the overlapping area is adjusted according to the input color value, the reference color value, and the number of times of fusion to complete pixel fusion, including: calculating an overlapping area according to a weighted average method. The color value of each pixel in the middle.

The embodiment of the invention further provides a computer readable storage medium storing computer executable instructions, which are implemented to implement the long exposure panoramic image capturing method described above.

When receiving the long exposure shooting instruction, the embodiment obtains an image captured by the mobile terminal viewfinder at a time; and then projects the acquired image to a preset projection space to form a corresponding projection image, where the acquired first frame image corresponds to The projected image is used as a reference projection image, and the acquired projection image corresponding to the subsequent image is used as a subsequent projection image; then the subsequent projection images are position-aligned and pixel-fused with the reference projection image one by one to form a new reference projection image, until the traversal is obtained. Subsequent projection image to obtain an output projection image; finally, the output projection image is backprojected to a coordinate space of the mobile terminal viewfinder acquisition image to generate a long exposure panoramic image, thereby registering the position of the subsequent projection image and the reference projection image Pixel fusion, the image acquired by the mobile terminal in a slight jitter state (for example, the user holds the mobile terminal for long exposure shooting) can also perform long exposure synthesis, thereby enhancing the adaptability of the mobile terminal to jitter in the long exposure shooting mode. So that the user is outdoors Exposure shooting, without carrying fixing means, so long exposure shooting function of the mobile terminal more convenient to use.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF abstract

1 is a schematic structural diagram of hardware of an optional mobile terminal that implements various embodiments of the present application;

Figure 2 is a block diagram showing the electrical structure of the camera of Figure 1;

3 is a schematic diagram of functional modules of a first embodiment of a long exposure panoramic image capturing apparatus of the present application;

4 is a schematic diagram of a refinement function module of an image projection module in a second embodiment of the long exposure panoramic image capturing apparatus of the present application;

5 is a schematic diagram of a refinement function module of a registration fusion module in a third embodiment of the long exposure panoramic image capturing apparatus of the present application;

6 is a schematic flow chart of a first embodiment of a long exposure panoramic image capturing method according to the present application;

7 is a refinement flow diagram of a step of projecting an acquired image into a preset projection space to form a corresponding projection image in a second embodiment of the long exposure panoramic image capturing method of the present application;

8 is a third embodiment of the long exposure panoramic image capturing method of the present application, in which a subsequent projection image of a plurality of viewing angles satisfying a preset projection condition is subjected to position registration and pixel fusion one by one with a reference projection image to form a new reference projection image. A detailed flow diagram of the steps up to the step of traversing the acquired subsequent projected image.

The implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings.

Embodiments of the invention

It is understood that the specific embodiments described herein are merely illustrative of the application and are not intended to be limiting.

The long exposure panoramic image capturing apparatus of the embodiment of the present invention can be applied to a terminal. A terminal implementing various embodiments of the present application will now be described with reference to the accompanying drawings. In the following description, the use of suffixes such as "module", "component" or "unit" for indicating an element is merely an explanation for facilitating the present application, and does not have a specific meaning per se. Therefore, "module" and "component" can be used in combination.

The mobile terminal can be implemented in various forms. For example, the terminals described in this application may include, for example, mobile phones, smart phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablets), PMPs (Portable Multimedia Players), navigation devices, and the like. Mobile terminals and fixed terminals such as digital TVs, desktop computers, and the like. In the following, it is assumed that the terminal is a mobile terminal. However, those skilled in the art will appreciate that configurations in accordance with embodiments of the present application can be applied to fixed type terminals in addition to elements that are specifically for mobile purposes.

FIG. 1 is a schematic structural diagram of hardware of an optional mobile terminal implementing various embodiments of the present application.

The mobile terminal 100 may include a wireless communication unit 110, an A/V (audio/video) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, an interface unit 170, The controller 180, the power supply unit 190, the long exposure panoramic image capturing device 200, and the like. Figure 1 illustrates a mobile terminal having various components, but it should be understood that not all illustrated components are required to be implemented. More or fewer components can be implemented instead. The elements of the mobile terminal will be described in detail below.

Wireless communication unit 110 typically includes one or more components that permit radio communication between mobile terminal 100 and a wireless communication device or network.

The A/V input unit 120 is for receiving an audio or video signal. The A/V input unit 120 may include a camera 121 that processes image data of still pictures or video obtained by an image capturing device in a video capturing mode or an image capturing mode. The processed image frame can be displayed on the display unit 151. The image frames processed by the camera 121 may be stored in the memory 160 (or other storage medium) or transmitted via the wireless communication unit 110, and two or more cameras 121 may be provided according to the configuration of the mobile terminal.

The user input unit 130 may generate key input data according to a command input by the user to control various operations of the mobile terminal. The user input unit 130 allows the user to input various types of information, and may include a keyboard, a pot, a touch pad (eg, a touch sensitive component that detects changes in resistance, pressure values, capacitance, etc. due to contact), Roller, rocker, etc. In particular, when the touch panel is superimposed on the display unit 151 in the form of a layer, a touch screen can be formed.

The sensing unit 140 detects the current state of the mobile terminal 100 (eg, the open or closed state of the mobile terminal 100), the location of the mobile terminal 100, the presence or absence of contact (ie, touch input) by the user with the mobile terminal 100, and the mobile terminal. The orientation of 100, the direction of movement and the tilt angle of the mobile terminal 100, and the like, and generates commands or signals for controlling the operation of the mobile terminal 100. For example, the sensing unit 140 includes an accelerometer 141 for detecting a real-time acceleration of the mobile terminal 100 to derive a moving direction of the mobile terminal 100, and a gyroscope 142 for detecting the mobile terminal 100 relative to its location The angle of inclination of the plane. In addition, the sensing unit 140 can detect whether the power supply unit 190 provides power or whether the interface unit 170 is coupled to an external device.

The interface unit 170 serves as an interface through which at least one external device can connect with the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, and an audio input/output. (I/O) port, video I/O port, headphone port, and more. The identification module may be stored to verify various information used by the user using the mobile terminal 100 and may include User Identification Module (UIM), Customer Identification Module (SIM), Universal Customer Identification Module (USIM), and more. In addition, the device having the identification module (hereinafter referred to as "identification device") may take the form of a smart card, and thus the identification device may be connected to the mobile terminal 100 via a port or other connection device. The interface unit 170 can be configured to receive input from an external device (eg, data information, power, etc.) and transmit the received input to one or more components within the mobile terminal 100 or can be used at the mobile terminal and external device Transfer data between.

Output unit 150 is configured to provide an output signal (eg, an audio signal, a video signal, a vibration signal, etc.) in a visual, audio, and/or tactile manner. The output unit 150 may include a display unit 151, an audio output module 152, and the like.

The display unit 151 can display information processed in the mobile terminal 100. For example, when the mobile terminal 100 is in a phone call mode, the display unit 151 can display a user interface (UI) or a graphical user interface (GUI) related to a call or other communication (eg, text messaging, multimedia file download, etc.). When the mobile terminal 100 is in a video call mode or an image capturing mode, the display unit 151 may display a captured image and/or a received image, a UI or GUI showing a video or image and related functions, and the like.

Meanwhile, when the display unit 151 and the touch panel are superposed on each other in the form of a layer to form a touch screen, the display unit 151 can function as an input device and an output device. The display unit 151 may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), an organic light emitting diode (OLED) display, a flexible display, a three-dimensional (3D) display, and the like. Some of these displays may be configured to be transparent to allow a user to view from the outside, which may be referred to as a transparent display, and a typical transparent display may be, for example, a TOLED (Transparent Organic Light Emitting Diode) display or the like. According to a particular desired embodiment, the mobile terminal 100 may include two or more display units (or other display devices), for example, the mobile terminal may include an external display unit (not shown) and an internal display unit (not shown) . The touch screen can be used to detect touch input pressure values as well as touch input positions and touch input areas.

The audio output module 152 may convert audio data received by the wireless communication unit 110 or stored in the memory 160 when the mobile terminal is in a call signal receiving mode, a call mode, a recording mode, a voice recognition mode, a broadcast receiving mode, and the like. The audio signal is output as sound. Moreover, the audio output module 152 can provide audio output (eg, call signal reception sound, message reception sound, etc.) associated with a particular function performed by the mobile terminal 100. Audio output module 152 It may include a pickup, a buzzer, and the like.

The memory 160 may store a software program or the like for processing and control operations performed by the controller 180, or may temporarily store data (for example, a phone book, a message, a still image, a video, etc.) that has been output or is to be output. Moreover, the memory 160 can store data regarding vibrations and audio signals of various manners that are output when a touch is applied to the touch screen.

The memory 160 may include at least one type of storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (eg, SD or DX memory, etc.), a random access memory (RAM), a static random access memory ( SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, and the like. Moreover, the mobile terminal 100 can cooperate with a network storage device that performs a storage function of the memory 160 through a network connection.

The controller 180 typically controls the overall operation of the mobile terminal. For example, the controller 180 performs the control and processing associated with voice calls, data communications, video calls, and the like. In addition, the controller 180 may include a multimedia module 181 for reproducing (or playing back) multimedia data, for example, the multimedia module 181 may be used to display a long-exposure panoramic image generated in real time in real time, and the multimedia module 181 may be constructed in the controller 180, or may It is configured to be separated from the controller 180. The controller 180 may perform a pattern recognition process to recognize a handwriting input or a picture drawing input performed on the touch screen as a character or an image.

The power supply unit 190 receives external power or internal power under the control of the controller 180 and provides appropriate power required to operate the various components and components.

The various embodiments described herein can be implemented in a computer readable medium using, for example, computer software, hardware, or any combination thereof. For hardware implementations, the embodiments described herein may be through the use of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays ( An FPGA, a processor, a controller, a microcontroller, a microprocessor, at least one of the electronic units designed to perform the functions described herein, in some cases, such an embodiment may be at the controller 180 Implemented in the middle. For software implementations, implementations such as procedures or functions may be implemented with separate software modules that permit the execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, which can be stored in memory 160 and The controller 180 executes.

The long exposure panoramic image capturing device 200 includes an image acquiring module 10, an image projecting module 20, a registration fusion module 30, and a back projection module 40, wherein

The image obtaining module 10 is configured to acquire an image periodically collected by the viewfinder of the mobile terminal when receiving the long exposure shooting instruction;

The image projection module 20 is configured to project the acquired image to a preset projection space to form a corresponding projection image, wherein the acquired projection image corresponding to the first frame image is used as a reference projection image, and the acquired projection corresponding to the subsequent image is used. The image as a subsequent projected image;

The registration fusion module 30 is configured to perform position registration and pixel fusion of the subsequent projection images one by one with the reference projection image to form a new reference projection image until the acquired subsequent projection image is traversed to obtain an output projection image;

The back projection module 40 is configured to backproject the output projected image to a coordinate space of the mobile terminal viewfinder captured image to generate a long exposure panoramic image.

In addition, since the present invention mainly relates to a long-exposure panoramic image capturing apparatus and method, the electrical structure of the camera 121 in the hardware structure of the mobile terminal 100 in FIG. 1 is described in detail. Referring to FIG. 2, FIG. 2 is an electrical structure of the camera of FIG. block diagram.

The photographic lens 1211 is composed of a plurality of optical lenses for forming a subject image, and is a single focus lens or a zoom lens. The photographic lens 1211 is movable in the optical axis direction under the control of the lens driver 1221, and the lens driver 1221 controls the focus position of the photographic lens 1211 in accordance with a control signal from the lens driving control circuit 1222, and can also be controlled in the case of the zoom lens. Focus distance. The lens drive control circuit 1222 performs drive control of the lens driver 1221 in accordance with a control command from the microcomputer 1217.

An imaging element 1212 is disposed on the optical axis of the photographic lens 1211 near the position of the subject image formed by the photographic lens 1211. The imaging element 1212 is for capturing an image of a subject and acquiring captured image data. Photodiodes constituting each pixel are arranged two-dimensionally and in a matrix on the imaging element 1212. Each photodiode generates a photoelectric conversion current corresponding to the amount of received light, which is subjected to charge accumulation by a capacitor connected to each photodiode. The front surface of each pixel is provided with a Bayer array of RGB color filters.

The imaging element 1212 is connected to the imaging circuit 1213. The imaging circuit 1213 performs charge accumulation control and image signal readout control in the imaging element 1212, and performs waveform shaping after reducing the reset noise of the read image signal (analog image signal). Further, gain improvement or the like is performed to obtain an appropriate signal level.

The imaging circuit 1213 is connected to an A/D converter 1214 that performs analog-to-digital conversion on the analog image signal and outputs a digital image signal (hereinafter referred to as image data) to the bus 1227.

The bus 1227 is a transmission path for transmitting various data read or generated inside the camera. The A/D converter 1214 is connected to the bus 1227, and an image processor 1215, a JPEG processor 1216, a microcomputer 1217, a SDRAM (Synchronous Dynamic Random Access Memory) 1218, and a memory interface are also connected. (hereinafter referred to as memory I/F) 1219, LCD (Liquid Crystal Display) driver 1220.

The image processor 1215 performs various kinds of images such as OB subtraction processing, white balance adjustment, color matrix calculation, gamma conversion, color difference signal processing, noise removal processing, simultaneous processing, edge processing, and the like on the image data based on the output of the imaging element 1212. deal with. The JPEG processor 1216 compresses the image data read out from the SDRAM 1218 in accordance with the JPEG compression method when the image data is recorded on the recording medium 1225. Further, the JPEG processor 1216 performs decompression of JPEG image data for image reproduction display. At the time of decompression, the file recorded on the recording medium 1225 is read, and after the compression processing is performed in the JPEG processor 1216, the decompressed image data is temporarily stored in the SDRAM 1218 and displayed on the LCD 1226. Further, in the present embodiment, the JPEG method is adopted as the image compression/decompression method. However, the compression/decompression method is not limited thereto, and other compression/decompression methods such as MPEG, TIFF, and H.264 may be used.

The microcomputer 1217 functions as a control unit of the entire camera, and collectively controls various processing sequences of the camera. The microcomputer 1217 is connected to the operation unit 1223 and the flash memory 1224.

The operating unit 1223 includes, but is not limited to, a physical button or a virtual button, and the entity or virtual button may be a power button, a camera button, an edit button, a dynamic image button, a reproduction button, a menu button, a cross button, an OK button, a delete button, an enlarge button An operation control such as various input buttons and various input keys detects an operation state of these operation controls, and outputs the detection result to the microcomputer 1217. Further, a touch panel is provided on the front surface of the LCD 1226 as a display, and the touch position of the user is detected, and the touch position is output to the microcomputer 1217. Microcomputer 1217 based on operation The detection result of the operation position of the unit 1223 performs various processing sequences corresponding to the operation of the user.

The flash memory 1224 stores programs for executing various processing sequences of the microcomputer 1217. The microcomputer 1217 performs overall control of the camera in accordance with the program. Further, the flash memory 1224 stores various adjustment values of the camera, and the microcomputer 1217 reads out the adjustment value, and performs control of the camera in accordance with the adjustment value.

The SDRAM 1218 is an electrically rewritable volatile memory for temporarily storing image data or the like. The SDRAM 1218 temporarily stores image data output from the A/D converter 1214 and image data processed in the image processor 1215, the JPEG processor 1216, and the like.

The memory interface 1219 is connected to the recording medium 1225, and performs control for writing image data and a file header attached to the image data to the recording medium 1225 and reading out from the recording medium 1225. The recording medium 1225 is, for example, a recording medium such as a memory card that can be detachably attached to the camera body. However, the recording medium 1225 is not limited thereto, and may be a hard disk or the like built in the camera body.

The LCD driver 1220 is connected to the LCD 1226, and stores image data processed by the image processor 1215 in the SDRAM 1218. When display is required, the image data stored in the SDRAM 1218 is read and displayed on the LCD 1226, or the image data stored in the JPEG processor 1216 is compressed. In the SDRAM 1218, when display is required, the JPEG processor 1216 reads the compressed image data of the SDRAM 1218, decompresses it, and displays the decompressed image data through the LCD 1226.

The LCD 1226 is configured to display an image on the back of the camera body. However, it is not limited thereto, and various display panels (LCD 1226) such as an organic EL may be used.

Based on the hardware structure of the mobile terminal and the electrical structure diagram of the camera, an embodiment of the long exposure panoramic image capturing apparatus of the present application is proposed, and the long exposure panoramic image capturing apparatus is a part of the mobile terminal.

Referring to FIG. 3, the present application provides a long exposure panoramic image capturing apparatus. In a first embodiment of a long exposure panoramic image capturing apparatus, the apparatus includes:

When the user needs to take a long exposure photo with the mobile terminal, input a long exposure shooting instruction to the mobile terminal to control the mobile terminal to enter the long exposure mode, the camera shutter of the mobile terminal is open, the camera The viewfinder captures changes in light and shadow in the scene over a long period of time. When the mobile terminal camera receives the long exposure shooting instruction, the viewfinder of the mobile terminal starts to collect images at a timing, for example, one frame image is acquired every 0.5 s, and the image acquisition module 10 collects the image acquired by the mobile terminal viewfinder at a time.

The image projection module 20 projects the acquired image acquired by the mobile terminal viewfinder to a preset projection space to form a corresponding projected image. Projecting the acquired image may have multiple execution sequences. For example, each frame image acquired by the mobile terminal viewfinder may be captured, and the acquired image of the frame may be projected to a preset projection space; or each mobile terminal viewfinder may be acquired. Collecting preset frames (for example, 5 frames), and then projecting the acquired preset frame images one by one to the preset projection space; in addition, all the acquired frame images may be acquired once the image is captured by the mobile terminal viewfinder. The projections are projected one by one to the preset projection space. The execution sequence of the projection operation on the acquired image can be flexibly set according to requirements. For example, the image processing capability of the mobile terminal is strong, and after the image capture by the mobile terminal viewfinder is completed, all the acquired frame images are projected one by one to the preset projection. space. The projection image formed by the projection of the first frame image captured by the mobile terminal viewfinder in the preset projection space is used as a reference projection image, and the projection of the subsequent frame image collected by the mobile terminal viewfinder in the preset projection space is obtained. The projected image is used as a subsequent projected image. The projection space includes a cylindrical projection space, a spherical projection space, a cube projection space, and the like.

The registration fusion module 30 is configured to perform position registration and pixel fusion of the subsequent projection images of the plurality of viewing angles that meet the preset projection conditions one by one with the reference projection image to form a new reference projection image until the acquired subsequent projection image is traversed. To obtain an output projection image;

The acquired subsequent projection image includes a plurality of viewing angles. For example, when the user moves the mobile terminal horizontally for long exposure panoramic shooting, the acquired image is an image of multiple viewing angles in the same horizontal direction, so that the projected subsequent projected image is also relative to the reference projected image. An image of multiple perspectives. After acquiring the subsequent projection image, the registration fusion module 30 first determines whether the acquired subsequent projection satisfies the preset projection condition. For example, the preset projection condition includes that the tilt angle of the subsequent projection image relative to the reference projection image translation or rotation cannot be greater than the preset angle, The subsequent projection image is vertically shifted relative to the reference projection image by a preset value or the like; then the reference projection image is used as a reference image, and the acquired subsequent projection image is pressed Position registration and pixel fusion with the reference projection image one by one according to the order of projection to the preset projection space until the mobile terminal viewfinder stops capturing images, that is, acquiring subsequent projections of all images acquired by the mobile terminal viewfinder in the preset projection space After the image is registered with the reference projection image and the pixels are fused, the reference projection image is registered and the image after all the subsequent projection images are fused as an output projection image, thereby synthesizing the subsequent projection images of the plurality of viewing angles and the reference projection image into one large image. A projected image of the angle of view, thereby enabling back projection to obtain a long exposure panoramic image of a large viewing angle.

The subsequent projection image is firstly registered with the reference projection image, and then pixel-fused with the reference projection image to form a new reference projection image, and then the next subsequent projection image is registered with the new reference projection image and pixel fusion is performed. Position registration is based on image registration technology, and pixel fusion is based on fusion technology.

The above image registration technology refers to a process of matching and superimposing two or more images acquired at different times, different sensors (imaging devices) or under different conditions (weather, illumination, imaging position, angle, etc.). It is widely used in remote sensing data analysis, computer vision, image processing and other fields. The general flow of image registration technology is as follows: firstly, feature extraction is performed on two images to obtain feature points; matching feature point pairs are found by performing similarity measure; then image space coordinate transformation parameters are obtained by matching feature point pairs; Transform parameters for image registration.

The above image fusion technology refers to: image data of the same target collected by the multi-source channel is subjected to image processing and computer technology to extract the favorable information in the respective channels to the maximum extent, and finally integrated into a high-quality image to improve The utilization of image information, improving the accuracy and reliability of computer interpretation, and improving the spatial resolution and spectral resolution of the original image are beneficial for monitoring. The image to be fused is already registered and the pixel width is consistent.

In order to better understand the long exposure panoramic image capturing method of the present application, the present application is explained below with a specific example: for example, after receiving the long exposure shooting instruction, the image acquisition module 10 acquires three frames of images acquired by the mobile terminal viewfinder. , in order of image 1, image 2 and image 3, first image projection module 20 projects image 1 to a preset projection space to form a corresponding projection image 1 , and projection image 1 as a reference projection image; then image projection module 20 will acquire Image 2 is projected onto a preset projection space to form a corresponding projection image 2, and projection image 2 is used as a subsequent projection image; further, registration fusion module 30 will project image 2 (ie, subsequent projection image) and projection image 1 (ie, reference projection) Image) performing position registration and pixel fusion to form a new reference projection image (set as projection image 1a); The image projection module 20 then projects the acquired image 3 to a preset projection space to form a corresponding projection image 3, and uses the projection image 3 as a new subsequent projection image; the registration fusion module 30 then positions the projection image 3 and the projection image 1a. Registration and pixel fusion form the latest reference projection image, and since the acquired images are all projected, registered, and fused, the latest reference projection image is set as the output projection image.

The back projection module 40 performs back projection on the synthesized output image of the large angle of view, that is, back projection of the output projection image from the preset projection space to the coordinate space of the image captured by the mobile terminal finder to generate a user operating the mobile terminal camera A long exposure panoramic image of a large viewing angle of an image viewing angle is acquired.

In this embodiment, when the long-exposure shooting instruction is received by the image acquiring module 10, the image captured by the mobile terminal viewfinder is acquired; then the image projection module 20 projects the acquired image to the preset projection space to form a corresponding projection image. The projection image corresponding to the acquired first frame image is used as a reference projection image, and the acquired projection image corresponding to the subsequent image is used as a subsequent projection image; the registration fusion module 30 further follows the multiple perspectives that satisfy the preset projection condition. The projected image is position-aligned and pixel-fused with the reference projection image one by one to form a new reference projection image until the acquired subsequent projection image is traversed to obtain an output projection image; finally, the reverse projection module 40 reverses the output projection image. Projecting to the coordinate space of the image captured by the mobile terminal viewfinder to generate a long-exposure panoramic image, thereby aligning the position of the subsequent projected image with the reference projected image and pixel fusion, and the mobile terminal is in a slight jitter state (for example, the user holds the mobile terminal) For long exposure shooting) the captured images can also enter The line length exposure synthesis enhances the adaptability of the mobile terminal to the jitter in the long exposure shooting mode, so that the user does not need to carry the fixing device when performing long exposure shooting outdoors, so that the long exposure shooting function of the mobile terminal is more convenient to use. And, combining the subsequent projection images of the plurality of viewing angles with the reference projection image into one output projection image of a large viewing angle, and then performing back projection on the output projection image to obtain a long-exposure panoramic image with a large viewing angle, that is, by taking long exposure shooting and panoramic The combination of the projection techniques of the shooting enables the user to capture a large-angle long-exposure panoramic image beyond the range of the camera's viewing angle, thereby overcoming the disadvantages of the related art method that the fixed mobile terminal camera can only capture a fixed viewing angle scene and has a limited shooting angle.

Further, based on the first embodiment of the long exposure panoramic image capturing apparatus of the present application, A second embodiment of the long exposure panoramic image capturing device is proposed. In the second embodiment, referring to FIG. 4, the image projection module 20 includes:

The parameter obtaining unit 21 is configured to acquire a motion parameter of the mobile terminal, where the motion parameter includes a motion direction and a tilt angle;

When acquiring the image acquired by the mobile terminal viewfinder, the parameter acquisition unit 21 controls the accelerometer and the gyroscope installed inside the mobile terminal to sense the moving direction and the tilt angle of the user holding the mobile terminal; or use the image registration method to the current The image extraction feature, the parameter acquisition unit 21 finds the alignment position to determine the moving direction and the tilt angle at which the user holds the mobile terminal.

The space determining unit 22 is configured to determine a projection space for image projection according to the acquired motion parameters;

The space determining unit 22 determines an applicable projection space according to the moving direction and the tilting angle of the mobile terminal; for example, when the tilting angle is smaller than the preset angle, and the moving direction is only rotated 360 degrees in the horizontal direction, determining the cylindrical projection space as the image projection amount The preset projection space; when the tilt angle of the mobile terminal is greater than the preset angle, or the vertical direction shift is greater than the preset value, the user is prompted or the image that is tilted too much is discarded. When the horizontal and vertical directions are greater than the preset value, the spherical projection space is determined as the projection space of the image projection, and the spherical projection space can synthesize the scene in which the mobile terminal camera moves in a three-dimensional manner.

The projection unit 23 is arranged to project the acquired image to the determined projection space to form a corresponding projection image.

After the image captured by the user holding the mobile terminal moving process is projected into the determined projection space, the subsequent registration process and pixel fusion process can be performed in the same projection space. The projection image corresponding to the acquired first frame image is used as a reference projection image, and the acquired projection image corresponding to the subsequent image is used as a subsequent projection image.

In this embodiment, the motion parameter of the mobile terminal is acquired by the parameter acquisition unit 21, and the motion tendency of the user to hold the mobile terminal for long exposure shooting is determined, so that the space determining unit 22 selects an appropriate projection space, thereby improving image projection to determination. The projection effect of the projection space improves the integrity of the projected image, and facilitates the registration and fusion between subsequent projected images, thereby improving the light and shadow effect of the image obtained by the mobile terminal for long-exposure panoramic image capturing.

Further, based on the first embodiment of the long exposure panoramic image capturing apparatus of the present application, a third embodiment of the long exposure panoramic image capturing apparatus is proposed. In the third embodiment, referring to FIG. 5, the registration fusion module is provided. 30 includes:

The feature extraction unit 31 is configured to extract reference features from the reference projection image, and extract matching features corresponding to the reference features from the subsequent projection images of the plurality of perspectives satisfying the preset projection conditions one by one;

In the image registration process, we first choose a transformation model as a hypothesis. The global transformation of the image can select geometric transformation, similar transformation, affine transformation, and projective transformation. The image local transformation can divide the image into different parts and calculate for each part. A separate registration parameter that is used to align the reference projection image with subsequent projection images. Then, the feature extraction unit 31 extracts the registration features, and the methods for extracting the registration features are: feature point matching, optical flow method, cross-correlation, and the like. among them:

Feature point matching means: extracting a number of feature points from a reference frame (ie, a reference projection image), and then extracting or searching for corresponding feature points from the to-be-registered frame (ie, a subsequent projected image), and solving the problem according to the position of the feature point as data The registration parameters of the registration frame relative to the reference frame.

The optical flow method refers to estimating the instantaneous velocity of the moving motion of the moving object on the imaging plane by using the variation of the pixel in the image sequence and the correlation between adjacent frames to find the previous frame and the current Correspondence between frames, thereby calculating motion information of objects between adjacent frames.

Cross-correlation means: transforming the image into the frequency domain by Fourier transform, and then calculating the correlation of each position of the to-be-registered frame in the spatial domain by using the cross-correlation formula, taking the largest position as the relative reference frame to be registered. Registration parameters.

The registration unit 32 is configured to combine the subsequent projection image that meets the preset projection condition with the reference projection image according to the positional relationship between the matching feature and the reference feature;

The registration unit 32 obtains a registration parameter according to the positional relationship between the matching feature and the reference feature (for example, the registration parameter is a two-dimensional or three-dimensional motion vector), and the subsequent projection that satisfies the preset projection condition according to the registration parameter The image is registered in conjunction with the reference projection image (ie, aligned). For example, there are three matching features. According to the positional relationship between the three matching features and the three corresponding reference features, the moving direction and the moving distance of the matching feature to the reference feature are obtained, and the subsequent projected image is moved according to the moving direction and the moving distance. Each pixel point, the subsequent projection image is moved every pixel point The subsequent position, such that the matching features of the subsequent projected image are aligned with the reference features of the reference projected image, and also causes the subsequent projected image to coincide with the same image region of the reference projected image.

Taking the affine transformation model as an example, the transformation has 6 parameters for 2*3 matrix representation.

A can be found by a registration algorithm. Then a point for the current input image frame (ie, the subsequent projected image) on the projected cylinder

Coordinated position

Similarly, the coordinate positions of all the pixel points of the current input image after registration are obtained, so that the current input image is aligned with the reference projection image according to the coordinate positions of all the pixel points after registration.

The merging unit 33 is configured to perform pixel merging and overlapping of the overlapping regions of the reference projection image and the non-overlapping region to form a new reference projection image until the acquired subsequent projection image is traversed. The resulting reference projection image is taken as an output projection image.

For each pixel on the input frame image (ie, the subsequent projected image) that currently meets the preset projection condition, it is first determined whether the pixel overlaps with the pixel of the previous panoramic image (ie, the reference projected image). The method of judging is that the color of each pixel position is initially 0 when the panorama is initialized. Each pixel of the new input frame image is judged at the composite position if the colors of the original pixels are all 0, and then overlap; if the colors are not all 0, they overlap.

After the subsequent projection image that meets the preset projection condition is aligned with the reference projection image, the fusion unit 33 performs pixel fusion on the overlapping region of the subsequent projection image and the reference projection image, that is, based on the image fusion technique, the pixel of the subsequent projection image according to the overlapping region The color value and the color value of the reference projection image pixel point are pixel-fused to obtain the fused pixel point color value; the merging unit 33 directly splices the subsequent projection image and the reference projection image non-overlapping region, that is, if the reference projection image is used The area has a subsequent projection image, indicating that the mobile terminal acquires a new image due to the movement, and directly splicing the newly acquired subsequent projection image with the corresponding position of the reference projection image; thus, the image formed by the pixel fusion and the image mosaic is used as a new reference projection. image. In the same way, the same process is performed on other subsequent projected images by using the new reference projection image, and the loop is repeated until the traversal or processing is completed. Finished subsequent projection images. In addition, the image after the subsequent projection image is registered and registered with the reference projection image may be synchronously displayed in the display area of the mobile terminal for the user to view.

After the mobile terminal viewfinder stops acquiring the image, and all the acquired subsequent projection images are registered and fused with the reference projection image, the finally formed new reference projection image, the fusion unit 33 uses the latest reference projection image as an output projection. image.

In this embodiment, by comparing the reference features in the reference projection image with the matching features in the subsequent projection images, the positional relationship between the reference projection image and the subsequent projection image is obtained, that is, the registration of the subsequent projection image with respect to the reference projection image is obtained. Parameter, and then aligning the subsequent projection image with the reference projection image according to the positional relationship between the reference projection image and the subsequent projection image, and then pixel merging the overlapping regions of the subsequent projection image with the reference projection image, directly splicing the non-overlapping regions to form a new one Referring to the projected image until the acquired subsequent projected image is traversed; finally, the finally formed reference projected image is used as an output projected image, thereby registering and merging each subsequent projected image with the updated reference projected image to form a long exposure. The panoramic output projection image corresponding to the captured image can be combined into a panoramic projection output image of a large viewing angle image, and can realize a long exposure shooting function when the mobile terminal is shaken at a certain amplitude, and the mobile terminal is expanded. Long exposure shooting Wai.

Further, based on the third embodiment of the long exposure panoramic image capturing device of the present application, a fourth embodiment of the long exposure panoramic image capturing device is proposed. In the fourth embodiment,

The image obtaining module 10 is further configured to acquire a motion parameter of the mobile terminal when acquiring an image captured by the mobile terminal viewfinder, and the motion parameter includes a moving direction and a tilting angle;

When the motion parameter starts to acquire the image acquired by the mobile terminal viewfinder, the accelerometer and the gyroscope installed inside the mobile terminal are used to sense the moving direction and the tilt angle of the user holding the mobile terminal; or the current image is extracted by using the image registration method. Feature, find the alignment position to determine the direction and tilt angle at which the user holds the mobile terminal.

The registration unit 32 is also configured to:

Determining, according to the motion parameter of the mobile terminal, a combination direction of the subsequent projection image that satisfies the preset projection condition and the reference projection image;

According to the combination direction, and the positional relationship between the matching feature and the reference feature, the preset will be satisfied. Subsequent projection images of the projection conditions are combined with the reference projection image registration.

The registration unit 32 can roughly determine the combination direction of the subsequent projection image and the reference projection image that meet the preset projection condition according to the motion parameter of the mobile terminal. For example, if the mobile terminal moves horizontally to the right, the subsequent projection image that satisfies the preset projection condition should be The combination is made on the right side of the reference projection image, that is, the combined direction of the subsequent projection image and the reference projection image is right-handed. After determining the combination direction of the subsequent projection image and the reference projection image, the registration unit 32 can quickly align the subsequent projection image with the reference projection image and perform registration and combination according to the positional relationship between the matching feature and the reference feature.

In this embodiment, when the image acquired by the mobile terminal viewfinder is acquired, the motion parameter of the mobile terminal is acquired, and then the combination direction of the subsequent projection image that meets the preset projection condition and the reference projection image is determined according to the moving direction of the mobile terminal. After determining the combination direction of the subsequent projection image and the reference projection image satisfying the preset projection condition, according to the positional relationship between the matching feature and the reference feature, the subsequent projection image can be quickly aligned with the reference projection image and the registration is combined. The efficiency of registering the position of the subsequent projection image and the reference projection image is improved.

Further, based on the third embodiment of the long exposure panoramic image capturing device of the present application, a fifth embodiment of the long exposure panoramic image capturing device is proposed. In the fifth embodiment,

The fusion unit 33 is also configured to:

Obtaining an input color value, a reference color value, and a fusion number of each pixel position in an overlapping area of the subsequent projection image and the reference projection image that satisfy the preset projection condition;

Adjusting the color value of each pixel in the overlapping area according to the input color value, the reference color value, and the number of fusions to complete pixel fusion, wherein the input color value is a color value, a reference color of a pixel position of the subsequent projected image in the overlapping area. The value is the color value of the pixel position of the reference projected image in the overlapping area, and the number of times of fusion is the number of times the pixel of the reference projection image is pixel-fused at the pixel position in the overlapping area.

The fusion unit 33 acquires an input color value of each pixel point of the overlapping area of the reference projection image in the subsequent projection image that satisfies the preset projection condition, and acquires a reference color of each pixel point in the overlapping area of the reference projection image and the subsequent projection image. The number of fusions of the value and the pixel fusion; the fusion unit 33 adjusts the color value of each pixel in the overlapping region according to the input color value, the reference color value, and the number of fusions of each pixel position of the overlapping region to complete pixel fusion. For example, according to the weighted average method, the color value of each pixel in the overlapping region is calculated, and the number of times of fusion of a certain pixel in the overlapping region is F, the reference color value is C _F , and the input color value is C _P , then synthesized The color values are:

The number of fusions after synthesis is recorded as F incremented by 1. The color value in this embodiment is the value in the hexadecimal color code.

In this embodiment, the input color value, the reference color value, and the number of fusions of each pixel position in the overlapping area of the subsequent projection image and the reference projection image are obtained, and then based on the weighted average algorithm, and according to the input color value and the reference color. The value and the number of fusions are adjusted to adjust the color value of each pixel in the overlapping area to complete the pixel fusion, so that the color value fusion of the pixel in the overlapping area is correlated with the color value of the corresponding pixel of all subsequent projected images, so that the pixel fusion effect is better.

The present application also provides a long exposure panoramic image capturing method. In the first embodiment of the long exposure panoramic image capturing method, referring to FIG. 6, the method includes:

Step S10, when receiving the long exposure shooting instruction, acquiring an image periodically collected by the mobile terminal viewfinder;

When the user needs to take a long exposure photo with the mobile terminal, a long exposure shooting instruction is input to the mobile terminal to control the mobile terminal to enter the long exposure mode, the camera shutter of the mobile terminal is opened, and the camera viewfinder captures the change of the light and shadow in the scene for a long time. When the mobile terminal camera receives the long exposure shooting instruction, the viewfinder of the mobile terminal starts to acquire images at a timing, for example, one frame image is acquired every 0.5 s, and the long exposure panoramic image capturing device collects images acquired by the mobile terminal viewfinder at a time.

Step S20: Projecting the acquired image to a preset projection space to form a corresponding projection image, wherein the acquired projection image corresponding to the first frame image is used as a reference projection image, and the acquired projection image corresponding to the subsequent image is used as a subsequent projection image. ;

The acquired image acquired by the mobile terminal viewfinder is projected to a preset projection space to form a corresponding projected image. Projecting the acquired image may have multiple execution sequences. For example, each frame image acquired by the mobile terminal viewfinder may be captured, and the acquired image of the frame may be projected to a preset projection space; or each mobile terminal viewfinder may be acquired. Collecting preset frames (for example, 5 frames), and then projecting the acquired preset frame images one by one to the preset projection space; in addition, all the acquired frame images may be acquired once the image is captured by the mobile terminal viewfinder. The projections are projected one by one to the preset projection space. Correct The execution order of the acquired image for the projection operation can be flexibly set according to requirements. For example, the image processing capability of the mobile terminal is strong, and after the image capture by the mobile terminal viewfinder is completed, all the acquired frame images are projected one by one to the preset projection space. . The projection image formed by the projection of the first frame image captured by the mobile terminal viewfinder in the preset projection space is used as a reference projection image, and the projection of the subsequent frame image collected by the mobile terminal viewfinder in the preset projection space is obtained. The projected image is used as a subsequent projected image. The projection space includes a cylindrical projection space, a spherical projection space, a cube projection space, and the like.

Step S30, performing position registration and pixel fusion of the subsequent projection images of the plurality of viewing angles that meet the preset projection conditions one by one with the reference projection image to form a new reference projection image, until the acquired subsequent projection image is traversed to obtain the output projection image. ;

The acquired subsequent projection image includes a plurality of viewing angles. For example, when the user moves the mobile terminal horizontally for long exposure panoramic shooting, the acquired image is an image of multiple viewing angles in the same horizontal direction, so that the projected subsequent projected image is also relative to the reference projected image. An image of multiple perspectives. After obtaining the subsequent projection image, it is first determined whether the acquired subsequent projection satisfies the preset projection condition. For example, the preset projection condition includes that the tilt angle of the subsequent projection image relative to the reference projection image translation or rotation cannot be greater than the preset angle, and the subsequent projection image is relative to the reference. The vertical displacement of the projected image is greater than a preset value or the like; then the reference projection image is used as a reference image, and the acquired subsequent projection images are position-aligned and pixel-fused with the reference projection image one by one according to the order in which they are projected to the preset projection space until The mobile terminal viewfinder stops acquiring images, that is, after acquiring the image of all the images captured by the mobile terminal viewfinder in the preset projection space and the reference projection image position registration and pixel fusion, the reference projection image is registered and merged with all subsequent projections. The image after the image is used as an output projection image, thereby synthesizing the subsequent projection image of the plurality of angles of view and the reference projection image into a projection image of a large angle of view, so that the long-exposure panoramic image of the large angle of view can be back-projected.

The above image registration technology refers to matching two or more images acquired at different times, different sensors (imaging devices) or under different conditions (weather, illumination, imaging position, angle, etc.), The superposition process has been widely used in remote sensing data analysis, computer vision, image processing and other fields. The general flow of image registration technology is as follows: firstly, feature extraction is performed on two images to obtain feature points; matching feature point pairs are found by performing similarity measure; then image space coordinate transformation parameters are obtained by matching feature point pairs; Transform parameters for image registration.

In order to better understand the long exposure panoramic image capturing method of the present application, the present application is explained below with a specific example: for example, after receiving the long exposure shooting instruction, acquiring three frames of images captured by the mobile terminal viewfinder, which are images in sequence 1. Image 2 and image 3, first project image 1 to a preset projection space to form a corresponding projection image 1, and use projection image 1 as a reference projection image; then project the acquired image 2 to a preset projection space to form a corresponding projection. Image 2, the projected image 2 is used as a subsequent projected image; further, the projected image 2 (ie, the subsequent projected image) is position-registered and pixel-fused with the projected image 1 (ie, the reference projected image) to form a new reference projected image ( Set to the projected image 1a); then project the acquired image 3 to the preset projection space to form a corresponding projected image 3, and use the projected image 3 as a new subsequent projected image; and then positionally map the projected image 3 and the projected image 1a. Blending with the pixels to form the latest reference projection image, and since the acquired images are all projected, registered, and fused, the latest reference is obtained. Shadow image is set to output a projection image.

In step S40, the output projection image is back-projected to the coordinate space of the image acquired by the mobile terminal viewfinder to generate a long-exposure panoramic image.

Performing back projection on the synthesized output image of the large viewing angle, that is, back projection of the output projection image from the preset projection space to the coordinate space of the image captured by the mobile terminal finder to generate a large angle of view of the user operating the mobile terminal camera to capture the image Long exposure panoramic image of perspective.

In this embodiment, when the long exposure shooting instruction is received, the image captured by the mobile terminal viewfinder is acquired; then the acquired image is projected to the preset projection space to form a corresponding projection image, wherein the first image to be acquired is obtained. The projection image corresponding to the frame image is used as the reference projection image, and the projection image corresponding to the acquired subsequent image is used as the subsequent projection image; Subsequent projection images of the viewing angle are position-aligned and pixel-fused with the reference projection image one by one to form a new reference projection image until the acquired subsequent projection image is traversed to obtain an output projection image; finally, the output projection image is back projected The mobile terminal finder collects the coordinate space of the image to generate a long-exposure panoramic image, so that the mobile terminal is in a slight jitter state by the position registration and pixel fusion of the subsequent projected image and the reference avatar image (for example, the user holds the mobile terminal for long Exposure shooting) The captured image can also be combined with long exposure, which enhances the ability of the mobile terminal to adapt to the jitter in the long exposure shooting mode, so that the user does not need to carry a fixing device when performing long exposure shooting outdoors, so that the mobile terminal is long. The exposure shooting function is more convenient to use. And, combining the subsequent projection images of the plurality of viewing angles with the reference projection image into one output projection image of a large viewing angle, and then performing back projection on the output projection image to obtain a long-exposure panoramic image with a large viewing angle, that is, by taking long exposure shooting and panoramic The combination of the projection techniques of the shooting enables the user to capture a large-angle long-exposure panoramic image beyond the range of the camera's viewing angle, thereby overcoming the disadvantages of the related art method that the fixed mobile terminal camera can only capture a fixed viewing angle scene and has a limited shooting angle.

Further, based on the first embodiment of the long exposure panoramic image capturing method of the present application, a second embodiment of the long exposure panoramic image capturing method is proposed. In the second embodiment, referring to FIG. 7, step S20 includes:

Step S21, acquiring a motion parameter of the mobile terminal, where the motion parameter includes a motion direction and a tilt angle;

When acquiring the image acquired by the mobile terminal viewfinder, the accelerometer sense and the gyroscope installed inside the mobile terminal are used to sense the moving direction and the tilt angle of the mobile terminal, or the image registration method is used to extract the current image. Find the alignment position to determine the direction and tilt angle at which the user holds the mobile terminal.

Step S22, determining a projection space for image projection according to the acquired motion parameters;

Determining the applicable projection space according to the moving direction and the tilting angle of the mobile terminal; for example, when the tilting angle is smaller than the preset angle, and the moving direction is only rotated 360 degrees in the horizontal direction, determining the cylindrical projection space as the preset projection space of the image projection amount When the tilt angle of the mobile terminal is greater than the preset angle, or when the vertical direction shift is greater than the preset value, the user is prompted or the image that is tilted too much is discarded. When the horizontal and vertical directions are greater than the preset value, the spherical projection space is determined as the projection space of the image projection, and the spherical projection space can synthesize the scene in which the mobile terminal camera moves in a three-dimensional manner.

Step S23, projecting the acquired image to the determined projection space to form a corresponding projection image.

In this embodiment, by acquiring the motion parameter of the mobile terminal, determining the motion tendency of the user to hold the mobile terminal for long exposure shooting, thereby selecting an appropriate projection space, thereby improving the projection effect of the image projection to the determined projection space, and improving The completeness of the projected image facilitates the registration and fusion between subsequent projected images, thereby improving the light and shadow effect of the image obtained by the mobile terminal for long-exposure panoramic image capturing.

Further, based on the first embodiment of the long exposure panoramic image capturing method of the present application, a third embodiment of the long exposure panoramic image capturing method is proposed. In the third embodiment, referring to FIG. 8, step S30 includes:

Step S31, extracting reference features from the reference projection image, and extracting matching features corresponding to the reference features from the subsequent projection images of the plurality of perspectives satisfying the preset projection conditions one by one;

In the image registration process, we first choose a transformation model as a hypothesis. The global transformation of the image can select geometric transformation, similar transformation, affine transformation, and projective transformation. The image local transformation can divide the image into different parts and calculate for each part. A separate registration parameter that is used to align the reference projection image with subsequent projection images. Then, the registration features are extracted, and the methods for extracting the registration features are: feature point matching, optical flow method, cross-correlation, and the like. among them:

Cross-correlation means: transforming an image into the frequency domain using Fourier transform, and then using the cross-correlation formula The correlation of each position of the registration frame in the spatial domain is calculated, and the largest position is taken as the registration parameter of the to-be-registered frame relative to the reference frame.

Step S32, according to the positional relationship between the matching feature and the reference feature, the subsequent projection image that satisfies the preset projection condition is combined with the reference projection image;

According to the positional relationship between the matching feature and the reference feature, the registration parameter is obtained (for example, the registration parameter is a two-dimensional or three-dimensional motion vector), and the subsequent projection image and the reference projection satisfying the preset projection condition according to the registration parameter are obtained. Image registration is combined (ie aligned). For example, there are three matching features. According to the positional relationship between the three matching features and the three corresponding reference features, the moving direction and the moving distance of the matching feature to the reference feature are obtained, and the subsequent projected image is moved according to the moving direction and the moving distance. For each pixel, the position of each pixel of the subsequent projected image is obtained, so that the matching feature of the subsequent projected image is aligned with the reference feature of the reference projected image, and the subsequent projected image is coincident with the same image region of the reference projected image. .

Coordinated position

Step S33, performing pixel fusion on the overlapping area of the subsequent projection image satisfying the preset projection condition and the reference projection image, and directly splicing the non-overlapping area to form a new reference projection image until the acquired subsequent projection image is traversed;

For each pixel on the input frame image (ie, the subsequent projected image) that currently meets the preset projection condition, it is first determined whether the pixel overlaps with the pixel of the previous panoramic image (ie, the reference projected image). The judging method is that the color of each pixel position is initially 0 when the panorama is initialized. Each pixel of the new input frame image is judged at the composite position if the color of the original pixel is all 0s, and does not overlap; If the colors are not all 0, they overlap.

After the subsequent projection image satisfying the preset projection condition is aligned with the reference projection image, the overlapping region of the subsequent projection image and the reference projection image is pixel-fused, that is, based on the image fusion technology, according to the color value of the pixel of the subsequent projection image of the overlapping region and Refer to the color value of the pixel of the projected image for pixel fusion to obtain the fused pixel color value; and directly splicing the subsequent projection image and the reference projection image without overlapping regions, that is, if the reference projection image has a subsequent projection image It indicates that the mobile terminal acquires a new image by moving, and directly splicing the newly acquired subsequent projection image with the corresponding position of the reference projection image; thus, the image formed by the pixel fusion and the image mosaic is used as a new reference projection image. Similarly, the same processing is performed on the other subsequent projected images by using the new reference projection image, and the loop is repeated until the acquired subsequent projected images are traversed or processed. In addition, the image after the subsequent projection image is registered and registered with the reference projection image may be synchronously displayed in the display area of the mobile terminal for the user to view.

In step S34, the finally formed reference projection image is taken as an output projection image.

After the mobile terminal viewfinder stops acquiring the image, and all the acquired subsequent projection images are registered and fused with the reference projection image, the finally formed new reference projection image is used as the output projection image.

In this embodiment, by comparing the reference features in the reference projection image with the matching features in the subsequent projection images, the positional relationship between the reference projection image and the subsequent projection image is obtained, that is, the registration of the subsequent projection image with respect to the reference projection image is obtained. Parameter, and then aligning the subsequent projection image with the reference projection image according to the positional relationship between the reference projection image and the subsequent projection image, and then pixel merging the overlapping regions of the subsequent projection image with the reference projection image, directly splicing the non-overlapping regions to form a new one Referring to the projected image until the acquired subsequent projected image is traversed; finally, the finally formed reference projected image is used as an output projected image, thereby registering and merging each subsequent projected image with the updated reference projected image to form a long exposure. The panoramic output projection image corresponding to the captured image can be combined into a panoramic projection output image of a large viewing angle image, and the mobile terminal can realize the long exposure shooting function when the mobile terminal is shaken at a certain amplitude, and the mobile terminal is expanded. Long exposure shooting Wai.

Further, based on the third embodiment of the long exposure panoramic image capturing method of the present application, a fourth embodiment of the long exposure panoramic image capturing method is proposed. In the fourth embodiment, the long exposure panoramic view is provided. Photo shooting methods also include:

Step S35: Acquire a motion parameter of the mobile terminal when acquiring an image captured by the mobile terminal viewfinder, and the motion parameter includes a moving direction and a tilt angle;

When the motion parameter starts to acquire the image acquired by the mobile terminal viewfinder, the accelerometer sense and the gyroscope installed inside the mobile terminal sense the user's movement direction and the tilt angle of the mobile terminal; or use the image registration method to the current image. The features are extracted and the alignment position is found to determine the direction and tilt angle at which the user holds the mobile terminal.

Step S32 includes:

Step S321, determining a combination direction of the subsequent projection image that meets the preset projection condition and the reference projection image according to the moving direction of the mobile terminal;

According to the motion parameter of the mobile terminal, the combination direction of the continuous projection image and the reference projection image satisfying the preset projection condition can be roughly determined. For example, if the mobile terminal moves horizontally to the right, the subsequent projection image that satisfies the preset projection condition should be compared with the reference projection image. The right side is combined, that is, the combined direction of the subsequent projected image and the reference projected image is right-handed.

Step S322, combining the subsequent projection image with the reference projection image according to the combination direction and the positional relationship between the matching feature and the reference feature.

After determining the combination direction of the subsequent projection image and the reference projection image, according to the positional relationship between the matching feature and the reference feature, the subsequent projection image can be quickly aligned with the reference projection image and the registration is combined.

Further, based on the third embodiment of the long exposure panoramic image capturing method of the present application, a fifth embodiment of the long exposure panoramic image capturing method is proposed. In the fifth embodiment, the overlapping of the subsequent projected image and the reference projected image is performed. The steps of performing pixel fusion in the area include:

Step S331, acquiring an input color value, a reference color value, and a fusion number of each pixel position in an overlapping area of the subsequent projection image and the reference projection image that meet the preset projection condition;

Step S332, adjusting color values of each pixel in the overlapping area according to the input color value, the reference color value, and the number of fusion times to complete pixel fusion, wherein the input color value is a color value of a pixel position of the subsequent projected image in the overlapping area. The reference color value is a color value of the pixel position of the reference projection image in the overlapping area, and the number of fusion times is the number of times the pixel of the reference projection image is pixel-fused in the overlapping area.

Obtaining an input color value of each pixel point in an overlapping area of the subsequent projection image that meets the preset projection condition, and acquiring a reference color value and a pixel of each pixel point in the overlapping area of the reference projection image and the subsequent projection image The number of fusions of the fusion; adjusting the color value of each pixel in the overlapping area according to the input color value, the reference color value and the number of fusion times of each pixel position in the overlapping area to complete the pixel fusion. For example, according to the weighted average method, the color value of each pixel in the overlapping region is calculated, and the number of times of fusion of a pixel in the overlapping region is F, the reference color value is C _F , and the input color value is C _P , then The color values are:

It is to be understood that the term "comprises", "comprising", or any other variants thereof, is intended to encompass a non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes those elements. It also includes other elements that are not explicitly listed, or elements that are inherent to such a process, method, article, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device that comprises the element.

The serial numbers of the embodiments of the present invention are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former is better. Implementation. Based on such understanding, the technical solution of the present application, which is essential or contributes to the related art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, CD-ROM). The instructions include a number of instructions for causing a terminal device (which may be a cell phone, computer, server, air conditioner, or network device, etc.) to perform the methods described in various embodiments of the present application.

The above is only a preferred embodiment of the present application, and is not intended to limit the scope of the patent application, and the equivalent structure or equivalent process transformations made by the specification and the drawings of the present application, or directly or indirectly applied to other related technical fields. The same is included in the scope of patent protection of this application.

Industrial applicability

The embodiment of the present application provides a long exposure panoramic image capturing device and method, which enhances the adaptability of the mobile terminal to jitter in a long exposure shooting mode, so that the user does not need to carry a fixing device when performing long exposure shooting outdoors, so that the mobile terminal The long exposure shooting function is more convenient to use.

Claims

A long exposure panoramic image capturing device comprising:

The image acquisition module is configured to acquire an image acquired by the mobile terminal viewfinder at a time when receiving the long exposure shooting instruction;

The image projection module is configured to project the acquired image to a preset projection space to form a corresponding projection image, wherein the acquired projection image corresponding to the first frame image is used as a reference projection image, and the acquired projection image corresponding to the subsequent image is obtained. As a subsequent projected image;

The registration fusion module is configured to perform position registration and pixel fusion of the subsequent projection images of the plurality of viewing angles satisfying the preset projection conditions one by one with the reference projection image to form a new reference projection image until the traversal is obtained. Subsequently projecting an image to obtain an output projected image;

A back projection module is configured to backproject the output projected image to a coordinate space of the image acquired by the mobile terminal viewfinder to generate a long exposure panoramic image.
The long exposure panoramic image capturing apparatus according to claim 1, wherein the image projection module comprises:

a parameter obtaining unit, configured to acquire a motion parameter of the mobile terminal, where the motion parameter includes a motion direction and a tilt angle;

a space determining unit configured to determine a projection space for image projection according to the acquired motion parameter;

And a projection unit configured to project the acquired image to the determined projection space to form a corresponding projection image.
The long exposure panoramic image capturing apparatus according to claim 1 or 2, wherein the registration fusion module comprises:

a feature extraction unit configured to extract a reference feature from the reference projection image, and extract matching features corresponding to the reference feature from the subsequent projection images of the plurality of perspectives satisfying the preset projection condition one by one;

The registration unit is configured to combine the subsequent projection image that meets the preset projection condition with the reference projection image according to the positional relationship between the matching feature and the reference feature;

The merging unit is configured to perform pixel merging and overlapping of the overlapping area of the reference projection image satisfying the preset projection condition with the non-overlapping area to form a new reference projection image until the acquired subsequent projection image is traversed And the resulting reference projection image is taken as an output projection image.
The long exposure panoramic image capturing apparatus according to claim 3, wherein

The image acquisition module is further configured to acquire a motion parameter of the mobile terminal when the image acquired by the mobile terminal viewfinder is acquired, and the motion parameter includes a moving direction and a tilt angle;

The registration unit is further configured to:

Determining, according to a motion parameter of the mobile terminal, a combination direction of the subsequent projected image that satisfies a preset projection condition and the reference projected image;

And matching the subsequent projection image satisfying the preset projection condition with the reference projection image according to the bonding direction and the positional relationship between the matching feature and the reference feature.
The long exposure panoramic image capturing apparatus according to claim 3, wherein

The fusion unit is further configured to:

Obtaining an input color value, a reference color value, and a number of fusion times of each pixel position in an overlapping area of the subsequent projection image and the reference projection image that satisfy a preset projection condition;

And adjusting a color value of each pixel in the overlapping area to complete pixel fusion according to the input color value, the reference color value, and the number of fusions, wherein the input color value is a color of a pixel position of the subsequent projected image in the overlapping area. The value and the reference color value are color values of the pixel position of the reference projection image in the overlapping area, and the number of times of fusion is the number of times the pixel of the reference projection image is pixel-fused in the overlapping area.
The long exposure panoramic image capturing apparatus according to claim 2, wherein said parameter acquisition unit is configured to acquire a motion parameter of the mobile terminal by:

The parameter acquisition unit controls an accelerometer and a gyroscope installed inside the mobile terminal to sense a movement direction and a tilt angle of the user holding the mobile terminal, or use an image registration method to extract features from the current image, find an alignment position, and determine a user. Hold the moving direction and tilt angle of the mobile terminal.
The long exposure panoramic image capturing apparatus according to claim 2, wherein said projection is empty The space includes a cylindrical projection space, a spherical projection space, and a cube projection space.
The long exposure panoramic image capturing apparatus according to claim 3, wherein the feature extraction unit is configured to extract features by one of the following methods: feature point matching, optical flow method, and cross correlation.
The long exposure panoramic image capturing apparatus according to claim 3, wherein the registration unit is configured to obtain a registration parameter according to a positional relationship between the matching feature and the reference feature, and the registration parameter is satisfied according to the registration parameter. The subsequent projection image of the projection condition is combined with the reference projection image registration.
The long exposure panoramic image capturing apparatus according to claim 5, wherein said merging unit is configured to calculate a color value of each pixel point in the overlap region according to a weighted averaging method.
A long exposure panoramic image capturing method, comprising:

Obtaining an image acquired by the mobile terminal viewfinder at a time when receiving a long exposure shooting instruction;

Projecting the acquired image to a preset projection space to form a corresponding projection image, wherein the acquired projection image corresponding to the first frame image is used as a reference projection image, and the acquired projection image corresponding to the subsequent image is used as a subsequent projection image;

The subsequent projection images of the plurality of viewing angles satisfying the preset projection condition are position-aligned and pixel-fused with the reference projection image one by one to form a new reference projection image, until the acquired subsequent projection image is traversed to obtain an output projection image;

The output projection image is back projected to a coordinate space of the mobile terminal viewfinder acquisition image to generate a long exposure panoramic image.
The method of claim 11 , wherein the step of projecting the acquired image to a preset projection space to form a corresponding projection image comprises:

Obtaining a motion parameter of the mobile terminal, where the motion parameter includes a motion direction and a tilt angle;

Determining a projection space for image projection based on the acquired motion parameters;

The acquired image is projected onto the determined projection space to form a corresponding projected image.
The long exposure panoramic image capturing method according to claim 11 or 12, wherein said subsequent projection images of a plurality of viewing angles satisfying a predetermined projection condition are subjected to position registration and pixel fusion one by one with said reference projection image Forming a new reference projection image, and traversing the acquired subsequent projection image to obtain an output projection image includes:

Extracting reference features from the reference projection image, and extracting matching features corresponding to the reference features from the subsequent projection images of the plurality of perspectives satisfying the preset projection conditions one by one;

And matching the subsequent projection image that meets the preset projection condition with the reference projection image according to the positional relationship between the matching feature and the reference feature;

Performing pixel fusion on the overlapping area of the predetermined projection image and the reference projection image, and directly splicing the non-overlapping area to form a new reference projection image until the acquired subsequent projection image is traversed;

The resulting reference projection image is taken as an output projection image.
The long exposure panoramic image capturing method according to claim 13, wherein the long exposure panoramic image capturing method further comprises: acquiring a motion parameter of the mobile terminal when acquiring an image acquired by the timing of the mobile terminal viewfinder, wherein the motion parameter comprises moving Direction and inclination angle;

The step of combining the subsequent projection image satisfying the preset projection condition with the reference projection image according to the positional relationship between the matching feature and the reference feature comprises:

Determining, according to a motion parameter of the mobile terminal, a combination direction of the subsequent projected image that satisfies a preset projection condition and the reference projected image;

And matching the subsequent projection image satisfying the preset projection condition with the reference projection image according to the bonding direction and the positional relationship between the matching feature and the reference feature.
The long exposure panoramic image capturing method according to claim 13, wherein the step of pixel-blending the overlapping region of the predetermined projection image that satisfies the preset projection condition with the reference projection image comprises:

Obtaining an input color value, a reference color value, and a number of fusion times of each pixel position in an overlapping area of the subsequent projection image and the reference projection image that satisfy a preset projection condition;

And adjusting a color value of each pixel in the overlapping area to complete pixel fusion according to the input color value, the reference color value, and the number of fusions, wherein the input color value is a color of a pixel position of the subsequent projected image in the overlapping area. The value and the reference color value are color values of the pixel position of the reference projection image in the overlapping area, and the number of times of fusion is the number of times the pixel of the reference projection image is pixel-fused in the overlapping area.
The long exposure panoramic image capturing method according to claim 12, wherein said obtaining Motion parameters of the mobile terminal, including:

The accelerometer and the gyroscope installed inside the mobile terminal are controlled to sense the moving direction and the tilting angle of the user holding the mobile terminal, or the image registration method is used to extract the feature from the current image, and the alignment position is found to determine the user holding the mobile terminal. Direction of movement and angle of inclination.
The long exposure panoramic image capturing method according to claim 12, wherein the projection space comprises a cylindrical projection space, a spherical projection space, and a cube projection space.
The long exposure panoramic image capturing method according to claim 13, wherein said extracting reference features from said reference projection images and extracting one by one from said subsequent projection images satisfying a plurality of viewing angles of predetermined projection conditions The matching feature corresponding to the reference feature includes: performing feature extraction by one of the following methods: feature point matching, optical flow method, and cross-correlation.
The long exposure panoramic image capturing method according to claim 13, wherein the registering the subsequent projection image satisfying the preset projection condition with the reference projection image according to the positional relationship between the matching feature and the reference feature The combining includes: obtaining a registration parameter according to the positional relationship between the matching feature and the reference feature, and combining the subsequent projection image that meets the preset projection condition with the reference projection image according to the registration parameter.
The long exposure panoramic image capturing method according to claim 15, wherein said adjusting a color value of each pixel in said overlapping region according to said input color value, a reference color value, and a number of fusion times to complete pixel fusion, Including: calculating the color value of each pixel in the overlapping area according to the weighted average method.