WO2015108243A1

WO2015108243A1 - System and method for providing contents using object recognition, and application thereof

Info

Publication number: WO2015108243A1
Application number: PCT/KR2014/004018
Authority: WO
Inventors: 양승호; 한상준
Original assignee: 주식회사 웰시스템 (영업소)
Priority date: 2014-01-20
Filing date: 2014-05-07
Publication date: 2015-07-23

Abstract

The present invention relates to a system and method for providing contents using object recognition, which can always stably perform an object recognition operation by actively adjusting the amount of transmitted/received data in consideration of a load of a server and a network. The method can comprise the steps of: acquiring an image, and then generating a plurality of descriptors according to each size of the image, by a mobile terminal; requesting provision of contents from a server through a descriptor corresponding to an original size of the server, by a mobile terminal; searching for contents corresponding to the descriptor, providing the searched contents to the mobile terminal when the searching of the contents succeeds, and providing server load information to the mobile terminal when the searching of the contents fails, by the server; and performing a contents providing operation by the mobile terminal having received the contents, wherein the mobile terminal having received the server load information changes the types of descriptor on the basis of the server load information and requests provision of contents again.

Description

Content providing system, method and application thereof using object recognition

The present invention relates to an object recognition technology, and in particular, when performing an object recognition operation based on a web, a content providing system using object recognition to monitor a network environment and to actively adjust a load required for data transmission and reception, and It is about a method.

Object recognition technology is a technology that processes and analyzes an image to find out what object is in the image. Conventional object recognition technology has built a database of images and features of an object in a single device, and recognizes the target object in the currently input image using the information of the built database, but in this case, the types of objects that can be recognized are limited. Has the disadvantage of being.

Recently, a technology for expanding the type of objects that can be recognized by constructing a database related to object recognition on the web that can register and provide infinite information has been proposed and widely used.

However, since the mobile terminal needs to search for and acquire necessary information after accessing a server that builds and operates a database, the object recognition speed and performance are greatly affected by the load of the server and the network. That is, when there are many mobile terminals connected to the server or when the size of the requested content is large or when the network load is excessive, there is a disadvantage in that a smooth content providing operation cannot be performed.

Accordingly, an object of the present invention is to provide a system and method for providing content using object recognition that can stably perform an object recognition operation by actively adjusting the size of data transmission and reception in consideration of load of a server and a network.

The object of the present invention is not limited to the above-mentioned object, and other objects which are not mentioned will be clearly understood by those skilled in the art from the following description.

As a means for solving the above problem, according to the first embodiment of the present invention, a plurality of descriptors are generated for each image size, and the content is provided based on one of the plurality of descriptors, but the descriptor is based on the content provision result. A mobile terminal of varying types; And after resizing the content size in multiple stages, generating and storing a descriptor corresponding to each of the content source and the recycled content, and when the descriptor is transmitted from the mobile terminal, the content corresponding to the transmitted descriptor is stored. Provided is a content providing system using object recognition including a server for acquiring and providing.

The server searches for the content corresponding to the transmitted descriptor, and if the content search is successful, provides the content and the server load information to the mobile terminal, but if the content search fails, providing only the server load information to the mobile terminal. It is characterized by.

When the mobile terminal receives the content and the server load information together from the server, the mobile terminal checks the server load information and the response time of the server, and then the threshold time is set in spite of the small server load. It is characterized in that it further includes a function for varying the descriptor type to be used in the content providing request only when larger.

In addition, when the mobile terminal receives the content and server load information together from the server, the mobile terminal checks the server load information and the response time of the server, and then the response time is low despite the small server load. The method may further include a function of varying a content streaming method only when it is larger than a set threshold.

In addition, when the mobile terminal receives only server load information from the server, the mobile terminal further includes a function of changing a descriptor type to be used for a content providing request based on the server load information, and then re-requesting to provide content. It is done.

As a means for solving the above problem, according to a second embodiment of the present invention, after the mobile terminal acquires an image, generating a plurality of descriptors for each image size; Requesting, by the mobile terminal, the server to provide content through a descriptor corresponding to the original size of the image; After the server searches for the content corresponding to the descriptor, if the content search is successful, providing the successful content to the mobile terminal, and if the content search fails, providing server load information to the mobile terminal; And the mobile terminal receiving the content performs a content providing operation, and the mobile terminal receiving the server load information varies a descriptor type based on the server load information, and then re-requests to provide the content. Provided is a content providing method using object recognition.

The providing to the mobile terminal may be provided by mapping the server load information to the successful content when the content search is successful.

The method may further include varying a descriptor type to be used in the content providing request based on the response time if the server provides the content and the server load is less than the preset value. It characterized in that it further comprises.

In addition, the method, if the server provides the content and the server load is less than the predetermined value, if the response time is greater than the predetermined value, based on the response time variable content streaming method to be used in the content providing request Characterized in that it further comprises the step.

As a means for solving the above problem, according to a third embodiment of the present invention, after the mobile terminal acquires an image, generating a plurality of descriptors for each image size; Requesting the server to provide content through a descriptor corresponding to the original size of the image; And if the server fails to provide the content, changing a descriptor type to be used for the content providing request based on the load of the server, and then re-requesting the content providing.

The method may further include varying a descriptor type to be used in the content providing request or changing a content streaming method based on the response time if the server provides the content but the response time is larger than a preset value. It is done.

As a means for solving the above problem, according to the fourth embodiment of the present invention, after resizing the content size in multiple steps, generating and storing a descriptor corresponding to each of the content original and the recycled content ; Searching for content corresponding to the descriptor when the mobile terminal requests to provide content through the descriptor; And if the content search is successful, mapping the server load information to the searched successful content and providing the server load information to the mobile terminal. If the content search fails, providing only the server load information to the mobile terminal. To provide.

As described above, the present invention enables the object recognition operation to be performed while considering the load of the server and the network, thereby providing a system of maximum performance.

1 is a diagram illustrating a content providing system using object recognition according to an embodiment of the present invention.

2 is a view for explaining an image and content sizing method according to an embodiment of the present invention.

3 is a view for explaining a database construction method of a content providing system using object recognition according to an embodiment of the present invention.

4 is a view illustrating an object recognition operation method of a content providing system using object recognition according to an embodiment of the present invention.

5 is a view illustrating an object recognition operation method of a content providing system using object recognition according to another embodiment of the present invention.

6 is a flowchart illustrating an object information providing method according to an embodiment of the present invention.

7 to 10 are diagrams illustrating a method of providing object information according to an embodiment of the present invention.

11 is a block diagram of an object information providing apparatus according to an embodiment of the present invention.

The objects and effects of the present invention and the technical configurations for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. In describing the present invention, when it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

Terms to be described later are terms defined in consideration of functions in the present invention, and may be changed according to intentions or customs of users or operators.

However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various forms. The present embodiments are merely provided to complete the disclosure of the present invention and to fully inform the scope of the invention to those skilled in the art, and the present invention is defined by the scope of the claims. It will be. Therefore, the definition should be made based on the contents throughout the specification.

As shown in FIG. 1, the content providing system 100 of the present invention may include a content providing server 110, at least one mobile terminal 120, a database 130, and the like.

The object recognition system 100 allows various operations such as image search, video playback, product purchase site access, navigation execution, etc. to be linked based on the image recognized by the mobile terminal 120. In addition, the terminal recognition image may be composed of a trademark, a logo, a movie poster, an album jacket, a photograph, and the like, and may enable the user to immediately inform and associate with the operation or information.

However, in order to perform the associative operation, the mobile terminal 120 provides an image recognized by the mobile terminal to the server 110, and the server 110 needs to exchange content corresponding to the mobile terminal 120. In the case of providing the image having a relatively large amount of data directly to the server 110, since a heavy load is placed on the server, the present invention allows a descriptor having a relatively small amount of data to be exchanged.

Descriptors represent feature points that are inherent in the image, which has features that enable efficient description and management of the image. In other words, if a descriptor is extracted from an image and made into a database, various contents can be managed and controlled only by the database composed of the descriptor.

Accordingly, in the present invention, the server 110 may search and provide content corresponding to the image currently recognized by the mobile terminal 120 using the descriptor.

In addition, the present invention allows the mobile terminal 120 to receive various contents corresponding to the image recognized by the server 110 through the descriptor, but the content size (depending on the load of the server 110 and the network 130) Or capacity) can be actively varied, thereby minimizing the effects of the server 110 and the network 130.

To this end, the mobile terminal 120 of the present invention resizes the size (or capacity) of the image acquired by itself in various steps as shown in FIG. 2, and then generates and stores a descriptor corresponding to each content size. Do it. Then, the mobile terminal 120 receives the necessary content from the server 110 based on the descriptor corresponding to the original size of the image, and determines the load of the server 110 and the network 130 based on the content providing state. .

In addition, by reducing the size of an image to be used for content query in proportion to the load of the server 110 and the network 130, and providing a descriptor corresponding thereto, the server 110 may also provide content of a reduced size. .

Hereinafter, the function of each component will be described in more detail.

The mobile terminal 120 may be carried by a user, such as a smart phone or a tablet PC, and may be implemented as various communication devices capable of performing various video / audio call operations, image acquisition operations, and Internet connection operations. The mobile terminal 120 obtains an image required for object recognition by using a camera embedded therein, and resizes the size of the image in various steps. And using descriptor generation algorithms such as Scale Invariant Feature Transform (SIFT), Maximum Stable Extremal Regions (MSER), and Speeded Up Robust Feature (SURF) to generate descriptors for each of the original image and the recycled image. do.

The server 110 requests and receives the necessary content based on one of a plurality of descriptors, and the load of the server 110 and the network 130 is determined based on the content providing state, and accordingly, the content query is applied to the content query. By varying the descriptor type to be used, the server 110 can also provide content of reduced size.

In this case, the content provided by the server 110 may be composed of operation information including information on an object recognition operation and additional information required to perform the operation. Examples of the operation information may include information for displaying an image, playing a video, purchasing a product, and executing a navigation. The additional information may be configured to include URLs of nodes in which images are stored, URLs of merchandise sales sites for product purchases, and latitude and longitude of a map for navigation. Of course, at this time, it is natural that each operation information and additional information may be variously expanded by an enterprise customer, a content manager, or a system development operator.

The server 110 stores and manages various contents for supporting the object recognition operation. However, in the present invention, the content size provided to the mobile terminal 120 can be actively changed according to the network environment, and after the content size is resized in several steps at the time of content registration, the content source and the sized Create and store descriptors corresponding to each content.

The server 110 of the present invention performs the content search and provision operation based on the descriptor transmitted from the mobile terminal 120, by providing additional load information when providing the content, the mobile terminal 120 loads its own Make sure you know how much it is right now.

The database 130 stores, for each content, a plurality of content copies of which the content original is resized in various sizes in addition to the content original. In this case, a descriptor having different information is mapped to each of the content original and the plurality of content copies.

First, when the content requested for registration occurs, the server 110 resizes the corresponding content in various steps (S10). In step S10, the recycling step can be adjusted according to the type and size of the content. For example, if the content size is large, the resizing step may be set to three levels, otherwise the resizing step may be set to two steps.

After generating a descriptor corresponding to each of the content original and the recycled content, the descriptor is mapped to each of the content original and the recycled content (S11).

In this case, the descriptor may be automatically generated based on an image or a video included in the content, but if the descriptor is not automatically generated, the descriptor may be manually input by a content provider or a system administrator.

In addition, as long as the content includes a video (S12), a streaming method to be used for each network environment may be defined and stored in advance (S13). That is, the codec used for video streaming includes AVI (Audio Video Interleaved), ASF (Advanced Streaming Format), MPEG (Moving Picture Expert Group), MOV, RM, etc., and loads of the server 110 and the network 130. If small, a codec that supports a high-quality large capacity among them, and if the load of the server 110 and the network 130 is large, it is possible to use a codec that supports a low-quality low capacity.

First, when the mobile terminal 120 acquires a terminal image using a camera embedded therein (S20), the mobile terminal resizes the size of the terminal image in several steps, and then corresponds to each size of the terminal image. Generate a plurality of descriptors (S21).

In the present invention, the descriptor corresponding to the original size of the terminal image is named as a general descriptor, and the descriptor corresponding to each of the recycled images is named as an adaptive descriptor.

The mobile terminal 120 transmits a general descriptor corresponding to the original size of the terminal image to the server 110 and requests the server 110 to provide content corresponding to the terminal image (S22).

Then, the server 110 searches the database 130 based on the general descriptor provided by the mobile terminal 120 to obtain content corresponding to the general descriptor (S23).

If the content search is successful (S24), the server 110 determines the current load (i.e., how much data is in the data reception queue) to generate load information, and then the mobile terminal together with the searched content. Provided to 120 (S25).

On the other hand, if the content search is not successful to pass a certain time (S24), the server 110 to provide only its load information to the mobile terminal 120 (S26).

Then, the mobile terminal 120 succeeds in receiving the content and is provided with the content and the server load information, or fails to receive the content and receives only the server load information (S27 and S30).

The mobile terminal 120, which has successfully received the content, checks the server load information and at the same time checks the time from the content providing request to receiving the content, that is, the response time, and although the server load is small, the response time is higher than the preset threshold. If large, it is determined that a lot of load is currently applied to the network 130 (S28).

In addition, the descriptor type to be used for the content providing request is changed so that the content providing request operation can be performed with the minimum load. That is, the mobile terminal 120 selects an adaptive descriptor corresponding to the image downsizing the original image, thereby allowing the next content providing request operation to be performed. At this time, the downsizing ratio of the original image will be desirable to increase in proportion to the response time (S29).

On the other hand, the mobile terminal 120 that failed to receive the content checks the current load of the server 110 based on the server load information, and after determining the downsizing ratio of the original image to be proportional to the current load of the server 110, Select the corresponding adaptive descriptor. Then, the selected adaptive descriptor is retransmitted to the server 110 to request content provision once more (S30 and S31).

Then, the server 110 performs the content search operation once again based on the adaptive descriptor, and transmits the content and server load information together when the content search succeeds, and transmits only the server load information when the content search fails. (S23-S26).

That is, in the present invention, when the content search fails, the content providing request operation may be repeatedly performed while sequentially adjusting the image size. However, the repeatable number of times for providing a content providing request is set so that an infinite repetition of the content providing request is prevented in advance.

In addition, in the above embodiment, the type of descriptor is varied according to the network load, but in some cases, the streaming method may be adjusted as shown in FIG. 5.

The present invention also provides a method for providing an object using the content and a system therefor by applying the above-described content providing method.

Referring to FIG. 6, an image of an object is first transmitted from a user terminal to an object information providing server (S110). In one embodiment of the present invention, the user terminal is provided with an imaging device such as a camera, and any electronic device capable of network connection, for example, a mobile phone, a tablet PC, etc. may be the user terminal. In addition, as an object in the present specification and any object that can be photographed by a camera, for example, not only a normal product but also a specific place, a restaurant, a sign, and the like can be the object.

Thereafter, the object is identified in the object information providing server from the characteristic of the transmitted image (S120). The identification may be performed based on a character or an image, and the object may be identified by various conventional techniques, and thus a detailed description thereof will be omitted below.

Thereafter, object information corresponding to the identified object is transmitted from the object information providing server to the user terminal (S130). In the present invention, the object information is stored in the object information providing server through a separate terminal (information providing terminal) provided by the information provider for providing information about the object, the present inventors in particular in accordance with the amount proposed by the information provider It provides a method for differentially setting the amount and type of information stored and provided to the user. Furthermore, by providing feedback to the information provider on the number and location information of the information provided to the user in response to a user request, the information provider provides a technology for securing big data.

Referring to FIG. 7, first, three restaurants A, B, and C are disclosed.

Referring to FIG. 8, images (eg, signs, business cards, building designs) of the restaurants A, B, and C are captured by the user terminal 110 and transmitted to the object information providing server 120.

Referring to FIG. 9, the object information providing server 120 identifies the restaurants A, B, and C from images (eg, letters) of the restaurants A, B, and C.

Referring to FIG. 10, restaurant A, B, and C information stored in the object information providing server 120 are transmitted to the user terminal 110. At this time, the amount of information provided depends on the amount of suggestions proposed by the information providers of restaurants A, B, and C through the information providing terminal (for example, a computer). For example, Restaurant A's information provider offers 1 million won, Restaurant B's information provider offers 500,000 won, and Restaurant C's information provider offers 100,000 won for the proposed amount. Evaluation of the restaurant blogger, associated surrounding area information, etc. may all be provided to the user terminal. However, the restaurant C information provider who provided the lowest proposed amount provides only the location or contact information as the restaurant information.

As described above, the inventor of the present invention provides various information to the information consumer based on the object-based computing, and allows the information provider to determine the amount, number, and the like of the information provision by itself, thereby enabling effective information provision. Therefore, in an embodiment of the present invention, the object information may be set differently for each object by its information provider in the range, quantity, number of provisions, and the like.

Furthermore, the present invention may further include the step of transmitting the number of transmission of the object information for the identified object, as described above, in the object information providing server, in this case, the object stored in the object information providing server The number of times of providing information is transmitted to the user terminal. Therefore, through such a method, other users can effectively grasp object information such as restaurant information that the user has been provided the most in a specific area, and use it as a kind of big data.

According to another embodiment of the present invention, a configuration in which the number of objects or the number of user terminals provided with the object information is provided corresponding to the proposed amount of money is provided.

Furthermore, the present invention provides an object information providing service apparatus for performing the object information providing method described above.

Referring to FIG. 11, the object information providing service apparatus may include an image receiver 210 which receives a video image of an object from a user terminal; An object identifier 220 to identify the object from the transmitted image; A suggestion amount information storage unit 230 for storing offer amount information corresponding to the identified object; And an object information transmitter 240 for transmitting object information on the identified object to the user terminal in response to the stored proposal amount information. In this case, the amount of object information or the number of transmissions, etc. may be determined in proportion to the proposal amount stored in the proposal amount information storage unit 230, which is omitted as described above.

In addition, the object information display method as described above may be implemented as program instructions for implementing the above, and a recording medium that can be read by a computer that records the program instructions, for example, ROM, RAM, CD-ROM Magnetic tape, floppy disk, optical media storage, and the like.

The computer-readable recording medium may also be distributed over network coupled computer devices so that the computer-readable code is stored and executed in a distributed fashion. In this case, any one or more of the plurality of distributed computers may execute some of the functions presented above, and transmit the results to one or more of the other distributed computers, and receive the results. The computer may also execute some of the functions presented above, and provide the results to other distributed computers as well.

It can be implemented as a program instruction for implementing this as described above, the recording medium that can be read by the computer that recorded the program instructions, for example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical Media storage devices.

In addition, the computer-readable recording medium having recorded the program as described above is distributed to computer devices connected through a network, so that the computer-readable code can be stored and executed in a distributed manner. In this case, any one or more of the plurality of distributed computers may execute some of the functions presented above, and transmit the results to one or more of the other distributed computers, and receive the results. The computer may also execute some of the functions presented above, and provide the results to other distributed computers as well.

A computer capable of reading a recording medium recording an application, which is a program for driving a content providing system and method using object recognition according to each embodiment of the present invention, is a mobile terminal as well as a general PC such as a general desktop or a laptop. It may include a mobile terminal, such as a tablet PC, personal digital assistants (PDAs), and mobile communication terminals, as well as to be interpreted as any computing device.

The present invention relates to an object recognition technology, and in particular, when performing an object recognition operation based on a web, a content providing system using object recognition to monitor a network environment and to actively adjust a load required for data transmission and reception, and Regarding the method, industrial applicability is recognized.

Claims

A mobile terminal for generating a plurality of descriptors for each image size and requesting content provision based on one of the plurality of descriptors, and varying the descriptor type based on the content provision result; And

After resizing the content size in multiple stages, a descriptor corresponding to each of the content source and the recycled content is generated and stored, and when the descriptor is transmitted from the mobile terminal, the content corresponding to the transmitted descriptor is obtained. And Content providing system using the object recognition including a providing server.
The method of claim 1, wherein the server

Search for content corresponding to the transmitted descriptor, and if content search is successful, provide content and server load information to the mobile terminal, and if content search fails, provide only the server load information to the mobile terminal. Content providing system using object recognition.
The method of claim 2, wherein the mobile terminal

When the content and server load information are transmitted together from the server, when the server load information is checked and the response time of the server is checked, when the response time is larger than the preset threshold despite the small server load. Content providing system using the object recognition, characterized in that it further comprises a function for varying the descriptor type to be used for the content providing request.
The method of claim 2, wherein the mobile terminal

When the content and server load information are transmitted together from the server, when the server load information is checked and the response time of the server is checked, when the response time is larger than the preset threshold despite the small server load. Content providing system using object recognition, characterized in that it further comprises a function to vary the content streaming method only.
After the mobile terminal acquires an image, generating a plurality of descriptors for each image size;

Requesting, by the mobile terminal, the server to provide content through a descriptor corresponding to the original size of the image;

After the server searches for the content corresponding to the descriptor, if the content search is successful, providing the successful content to the mobile terminal, and if the content search fails, providing server load information to the mobile terminal; And

The mobile terminal receiving the content performs a content providing operation, and the mobile terminal receiving the server load information varies a descriptor type based on the server load information, and then re-requests to provide the content. Content providing method using object recognition.
The method of claim 5, wherein the providing to the mobile terminal

When content search is successful, the content providing method using object recognition, characterized in that for providing the server load information to the searched successful content.
The method of claim 6,

If the server provides the content and the server load is smaller than the preset value, but the response time is greater than the preset value, the method further comprises: varying the descriptor type to be used in the content providing request based on the response time. Content providing method using the object recognition, characterized in that.
The method of claim 7, wherein

If the server provides the content and the server load is less than the preset value, but the response time is greater than the preset value, the method further comprises: varying a content streaming method to be used in the content providing request based on the response time; Content providing method using the object recognition, characterized in that.
An object information providing method using a content providing method according to any one of claims 1 to 8,

Transmitting a video image of the object from the user terminal to the object information providing server;

Identifying the object from a feature of the transmitted video image;

And transmitting object information corresponding to the identified object from the object information providing server to the user terminal.
The method of claim 9,

The object information is set differently according to an object, and the object information is transmitted from an information providing terminal providing the object information to the object information providing server and stored.
The method of claim 10,

The object information providing method is characterized in that the amount of the object information depends on the offer amount transmitted from the information providing terminal to the object information providing server.
The method of claim 11,

And the amount of the object information increases in proportion to the offer amount.
The method of claim 12, wherein the object information providing method comprises:

The number of the transmission of the object information for the identified object is further stored in the object information providing server, wherein the number of object information stored in the object information providing server and the identification information of the object, the user terminal Object information providing method characterized in that the transmission to the user terminal according to the request signal.