CN101292258B - System and methods for creation and use of a mixed media environment - Google Patents
System and methods for creation and use of a mixed media environment Download PDFInfo
- Publication number
- CN101292258B CN101292258B CN2006800393767A CN200680039376A CN101292258B CN 101292258 B CN101292258 B CN 101292258B CN 2006800393767 A CN2006800393767 A CN 2006800393767A CN 200680039376 A CN200680039376 A CN 200680039376A CN 101292258 B CN101292258 B CN 101292258B
- Authority
- CN
- China
- Prior art keywords
- document
- expression
- mmr
- image
- text
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system of the present invention provides mechanisms for forming a mixed media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system of the present invention facilitates business methods that take advantage of the combination of a portable electronic device, such as a cellular camera phone, and a paper document.
Description
Technical field
The present invention relates to be used to produce the technology of the mixed media document that forms from two media type at least, and more specifically, relate to blending agent reality border (MMR) system that uses the print media that combines with electronic media to produce mixed media document.
Background technology
Document is printed and reproduction technology was used in many environment many years.For example, in private and commercial office, have in the home environment of personal computer and document print and the publication service environment in, all use printer and duplicating machine.Yet; Before do not thought and printed and reproduction technology has been the means that are connected static dump medium (that is paper document) and comprise the function served as bridge of the gap between mutual " virtual world " of digital communication system, network, information supply, advertisement, amusement and ecommerce and so on.
Print media is as the communication information, for example news and advertising message, main source last several centuries.Several years in the past; Through making with electronically readable and the form utilization that can search for, and through introducing interactive multimedia performance, personal computer and personal electronic device; For example PDA(Personal Digital Assistant) device and cellular phone (as; Camera cell phone) appearance and growing popularity have enlarged the notion of print media, and for traditional print media, it is beyond example.
Unfortunately, there is gap between the addressable virtual physical world of electricity based on the multimedia world and print media.For example, although everyone every day of all addressable print media and electronic information of developed world almost, the user of print media and personal electronic device does not have formation (that is, being used for convenient mixed media document) necessary instrument and the technology that be connected between the two.
In addition, traditional print media provides special favourable attribute, sense of touch for example, and not required power and lasting tissue and storage, it does not provide in virtual or digital media.Likewise, traditional digital media also provides special favourable attribute, for example portable (as, in the storage of mobile phone or portable computer, carry) and be easy to transmit (as, through Email).
Because these reasons, there are needs about the technology that makes it possible to develop the benefit related with medium both that print and virtual.
Summary of the invention
At least one aspect of one or more embodiment of the present invention provides the real border (MMR) of blending agent system and the method that is associated.MMR of the present invention system is provided for forming and comprises such as as the print paper of first medium with as the mechanism with the mixed media document of at least two types such medium of the text of electronic form or data, digital picture, digital photo, digital movie, digital audio file or network linking of second medium.In addition, it is convenient that MMR of the present invention system makes the business method of such portable electron device of utilization such as camera cell phone and paper document.MMR of the present invention system comprises MMR processor, acquisition equipment, communication mechanism and comprises the storer of MMR software.Can also the MMR processor be connected to storer or source, input media and the output unit of media type.MMR software comprises can be by the routine of MMR processor execution, and it is used to visit the MMR document of other digital content, creates or change MMR document, and uses document to carry out such as other such operations such as business transaction, data query and reports.
The present invention also comprises the method for many novelties, and it comprises: be used to create the method for the real border of blending agent document, be used to use the method for the real border of blending agent document and the method that is used to change or delete the real border of blending agent document.
Another aspect at least of one or more embodiment of the present invention with the machine readable media of order number (for example provides; One or more compact disks, disk, server, memory stick or hard disk drive, ROM, RAM or be suitable for the medium of any kind of store electrons instruction); When being carried out by one or more processors, it impels processor to carry out to be used for the process of the information of visiting mixed media document system.For example, this process can with similar or its variant of described method here.
Do not comprise allly in these described characteristics and advantage, and especially, consider to draw and describe, for those of ordinary skills, many other characteristics and advantage will be clearly.In addition, should be noted that, be mainly legibility and selected employed language in the instructions, and be not limited to the scope of inventive concept with the purpose of directiveness.
Description of drawings
Through with similar parts in same reference number indication, the accompanying drawing, illustrate the present invention as an example, and not as restriction.
Figure 1A illustrates according to one embodiment of present invention and the functional block diagram of the real border (MMR) of the blending agent that disposes system;
Figure 1B illustrates according to another embodiment of the invention and the functional block diagram of the MMR system that disposes;
Fig. 2 A, 2B, 2C and 2D illustrate acquisition equipment according to an embodiment of the invention;
Fig. 2 E illustrates according to one embodiment of present invention and the functional block diagram of the acquisition equipment that disposes;
Fig. 3 illustrates according to one embodiment of present invention and the functional block diagram of the MMR computing machine that disposes;
Fig. 4 illustrates according to one embodiment of present invention and one group of included software part in the MMR software suite that disposes;
Fig. 5 illustrates expression according to one embodiment of present invention and the diagram of embodiment of the MMR document of configuration;
Fig. 6 illustrates document finger print matching method according to an embodiment of the invention;
Fig. 7 illustrates according to one embodiment of present invention and the document finger print matching system that disposes;
Fig. 8 illustrates the flow process that text according to an embodiment of the invention/non-text is distinguished;
Fig. 9 illustrates the example that text according to an embodiment of the invention/non-text is distinguished;
Figure 10 illustrates the flow process of the point size of the text that is used for estimating image fragment according to an embodiment of the invention;
Figure 11 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 12 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 13 illustrates the example that interactive image according to an embodiment of the invention is analyzed;
Figure 14 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 15 illustrates the example that literal bounding box according to an embodiment of the invention is surveyed;
Figure 16 illustrates Feature Extraction Technology according to an embodiment of the invention;
Figure 17 illustrates Feature Extraction Technology according to another embodiment of the invention;
Figure 18 illustrates Feature Extraction Technology according to another embodiment of the invention;
Figure 19 illustrates Feature Extraction Technology according to another embodiment of the invention;
Figure 20 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 21 illustrates the multi-categorizer feature extraction of document fingerprint matching according to an embodiment of the invention;
Figure 22 and 23 illustrates the example of document fingerprint matching technology according to an embodiment of the invention;
Figure 24 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 25 illustrates the flow process of the feedback of database-driven according to an embodiment of the invention;
Figure 26 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 27 illustrates the flow process of the classification of database-driven according to an embodiment of the invention;
Figure 28 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 29 illustrates the flow process of the multiple classifition of database-driven according to an embodiment of the invention;
Figure 30 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 31 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 32 illustrates document fingerprint matching technology according to another embodiment of the invention;
Figure 33 illustrates the flow process of multilayer level identification according to an embodiment of the invention;
Figure 34 A illustrates according to one embodiment of present invention and the functional block diagram of the MMR Database Systems that dispose;
Figure 34 B illustrates the example of the MMR feature extraction of the technology based on OCR according to an embodiment of the invention;
Figure 34 C illustrates example index table tissue according to an embodiment of the invention;
Figure 35 illustrates the method that is used to produce a MMR concordance list according to an embodiment of the invention;
Figure 36 illustrates the method about graduate a group of document, the page and hypothesis on location of destination document that is used to calculate according to an embodiment of the invention;
Figure 37 A illustrates according to another embodiment of the invention and the MMR functions of components block diagram that disposes;
Figure 37 B illustrates one group of included in the MMR print software according to an embodiment of the invention software part;
Figure 38 illustrates the process flow diagram that in document, embeds the method for focus according to an embodiment of the invention;
Figure 39 A illustrates the example of html file according to an embodiment of the invention;
Figure 39 B illustrates the example of marked version of the html file of Figure 39 A;
Figure 40 A illustrates the example of the html file of Figure 39 A that is shown in the browser according to an embodiment of the invention;
Figure 40 B illustrates the example of printing edition of the html file of Figure 40 A according to an embodiment of the invention;
Figure 41 illustrates symbol focus according to an embodiment of the invention and describes;
Figure 42 A and 42B illustrate the exemplary page_desc.xml file of the html file of Figure 39 A according to an embodiment of the invention;
Figure 43 illustrates according to an embodiment of the invention, corresponding to the hotspot.xml file of Figure 41,42A and 42B;
Figure 44 illustrates the process flow diagram of the employed process of forwarding DLL according to an embodiment of the invention;
Figure 45 illustrates the process flow diagram of conversion according to an embodiment of the invention corresponding to the method for the character of the focus in the document;
Figure 46 illustrates the example of the electronic edition of document according to an embodiment of the invention;
Figure 47 illustrates the example that document is revised in printing according to an embodiment of the invention;
Figure 48 illustrates the process flow diagram of the method for shared document note according to an embodiment of the invention;
Figure 49 A illustrates the sample source webpage in the browser according to an embodiment of the invention;
The sample that Figure 49 B illustrates in the browser according to an embodiment of the invention is revised webpage;
Figure 49 C illustrates sample printing webpage according to an embodiment of the invention;
Figure 50 A illustrates the process flow diagram of interpolation focus according to an embodiment of the invention to the method for image conversion document;
Figure 50 B illustrates the process flow diagram of method that definition according to an embodiment of the invention is used to be added into the focus of image conversion document;
Figure 51 A illustrates the example of the user interface of the part that the newsprint page that scans according to an embodiment is shown;
Figure 51 B illustrates and is used for definition of data or mutual, with selected focus related user interface;
Figure 51 C illustrates the user interface that comprises Figure 51 B that assigns frame according to an embodiment of the invention;
Figure 51 D illustrates the user interface that is used for the focus in the display document according to an embodiment of the invention;
Figure 52 illustrates the process flow diagram of the method for use MMR document according to an embodiment of the invention and MMR system;
Figure 53 illustrates the block diagram of one group of exemplary commercial entity of according to an embodiment of the invention and MMR system relationship;
Figure 54 illustrates according to an embodiment of the invention, as the general business method easily through using the MMR system, the process flow diagram of method.
Figure 55 illustrates according to an embodiment of the invention, is used to visit the process flow diagram of the method for mixed media document.
Figure 56 A, 56B and 56C illustrate according to various embodiment of the present invention, are used to visit the method for mixed media document and the various examples of use.
Figure 57 illustrates according to an embodiment of the invention, is used to visit the process flow diagram of the method for mixed media document.
Figure 58 illustrates according to an embodiment of the invention, is used to visit the user interface of mixed media document.
Figure 59 illustrates according to an embodiment of the invention, is used for to the tagged method of medium file.
Figure 60 illustrates according to an embodiment of the invention, gives the tagged user interface of medium file.
Figure 61 A, 61B, 61C and 61D illustrate the process flow diagram of use according to the method for MMR document of the present invention and MMR system.
Figure 62 illustrates the process flow diagram of use according to the method for MMR document of the present invention and MMR system.
Figure 63 illustrates the functional block diagram that comprises the MMR system implementation example that does not have the MMR of optical acquisition massaging device.
Figure 64 is illustrated in the user interface of the menu of the MMR document on the MMR massaging device of MMR system of Figure 63.
Figure 65 is illustrated in the MMR document that shown on the MMR massaging device and the user interface of hot information.
Figure 66 illustrates the process flow diagram of operation of the MMR massaging device of Figure 63.
Figure 67 illustrates the functional block diagram of the MMR system implementation example of the MMR massaging device that comprises acquisition equipment and do not have optical acquisition.
Figure 68 illustrates the process flow diagram of operation of the MMR massaging device of Figure 67.
Figure 69 illustrates the block diagram of the MMR system that comprises the voice-mail server of handling hot information.
Figure 70 is illustrated in the process flow diagram that upgrades the method for hot information in the MMR system of Figure 69.
Figure 71 is illustrated in the process flow diagram of the method for retrieve hotspot information in the MMR system of Figure 69.
Figure 72 illustrates the block diagram of the MMR system that comprises the email message server of handling hot information.
Figure 73 is illustrated in the process flow diagram of the method for retrieve hotspot information in the MMR system of Figure 72.
Figure 74 illustrates the block diagram of the MMR system that comprises document database.
Figure 75 illustrates the block diagram that comprises the MMR system that transports database.
Figure 76 be illustrated in use the MMR individualized document come in order goods or the service use, according to the functional block diagram of an embodiment of MMR of the present invention system.
Figure 77 illustrates and is used to use the MMR individualized document to come the process flow diagram of order goods or service method.
Figure 78 illustrates according to another embodiment of the invention, the block diagram of the architecture of the MMR system of the action of execution MMR document.
Figure 79 illustrates according to an embodiment of the invention, relates to the block diagram of a part in the properties data storehouse of action.
Figure 80 illustrates according to an embodiment of the invention, is used to carry out the process flow diagram of method of the action of MMR document.
Figure 81 illustrates according to an embodiment of the invention, is used to check the process flow diagram of method of the historical information of relevant action.
Figure 82 A and 82B illustrate according to an embodiment of the invention, are used to carry out and move the process flow diagram of method of the order that is associated.
Figure 83 illustrates the block diagram of the architecture of distributed MMR according to an embodiment of the invention system.
Figure 84 illustrates according to an embodiment of the invention, is used for the MMR system is used as the process flow diagram of method of input control and the output unit of third party system.
Figure 85 illustrates according to an embodiment of the invention, is used for using in working environment the process flow diagram of the method for MMR system.
Embodiment
The real border (MMR) of blending agent system and the method that is associated are described.The MMR system is provided for forming and comprises at least two types medium; Like print paper as first kind of medium; And digital photograph, digital movie, digital audio file, digital text file or network linking be as second kind of medium, the mechanism of mixed media document.MMR system and/or technology can be further used for convenient various utilize portable electron device (as, PDA or camera cell phone) with the combining of paper document, so that the business prototype of mixed media document to be provided.
In a particular embodiment, the MMR system comprises the content-based retrieval database, the two-dimensional geometry relation between its target of representing to be extracted from document printing with the mode that allows use text based index search.The evidence technology of adding up combines the frequency that characteristic occurs with the possibility of its position in 2 dimensional region.In such embodiment, the MMR Database Systems comprise the concordance list of the description that reception is calculated by the MMR feature extraction algorithm.X-y position in those pages of concordance list identification document, the page and each characteristic appearance place.Provide the data from concordance list, the evidence accumulation algorithm is calculated graduate one group of document, the page and hypothesis on location.As desired, can use relational database (or other storage facility that is fit to) to store other characteristic about each document, the page and position.
The MMR Database Systems also can comprise other parts, like MMR processor, acquisition equipment, communication mechanism with comprise the storer of MMR software.Also can the MMR processor be connected to storer or source, input media and the output unit of media type.In such configuration; MMR software comprises the executable routine of MMR processor; Be used to visit MMR document, establishment or modification MMR document and use document to carry out other operation with other digital content, like business transaction, data query, report, or the like.
The MMR system survey
With reference now to Figure 1A,, real border (MMR) the system 100a of blending agent according to an embodiment of the invention is shown.The 100a of MMR system comprises MMR processor 102, communication mechanism 104, has the acquisition equipment 106 of portable input media 168 and portable output unit 170, comprises the storer 108 of MMR software, basic medium memory device 160, MMR medium memory device 162, output unit 164 and input media 166.Through provide use from the information of existing document printing (first media type) as second media type, like the mode of the index of audio frequency, video, text, updated information and service, the 100a of MMR system creates mixed media environment.
Basic medium memory device 160 is what to be used for its original stored in form second media type, and MMR medium memory device 162 is to be used for store M MR document, database and other as described in this, with the information of creating the MMR environment.Although illustrate respectively, in another embodiment, basic medium memory device 160 can be the part of same storage with MMR medium memory device 162, or integrated.Data-carrier store 160,162 is further stored data or the instruction about MMR processor 102; And comprise one or more devices; It comprises; For example, hard disk drive, floppy disk, CD-ROM device, DVD-ROM device, DVD-RAM device, DVD-RW device, flash memory device or any other suitable mass storage device.
Figure 1B illustrates according to another embodiment of the invention and the functional block diagram of the 100b of MMR system that disposes.In this embodiment, the 100b of MMR system comprises the printer 116 of MMR computing machine 112 (by user's 110 operations), network medium server 114 and generation document printing 118.The 100b of MMR system further comprises office's inlet 120, ISP's server 122, the electronic console 124 that is electrically connected to STB 126 and document scanner 127.Through network 128 communication linkage between MMR computing machine 112, network medium server 114, printer 116, office's inlet 120, ISP's server 122, STB 126 and the document scanner 127 is provided; Network 128 can be LAN (for example; Office or home network), WAN (for example, the Internet or company's network), the combination of LAN/WAN or any other data routing that can communicate by letter through its a plurality of calculation elements.
The 100b of MMR system further comprises can pass through cellular infrastructure 132, Wireless Fidelity (Wi-Fi) technology 134, Bluetooth technology 136 and/or infrared ray (IR) technology 138, with the acquisition equipment 106 of one or more computing machine 112, network medium server 114, user's printer 116, office's inlet 120, ISP's server 122, electronic console 124, STB 126 and document scanner 127 radio communications.Alternately; Perhaps in addition; Acquisition equipment 106 can pass through cable technology 140, communicates by letter with document scanner 127 with MMR computing machine 112, network medium server 114, user's printer 116, office's inlet 120, ISP's server 122, electronic console 124, STB 126 with wired mode.Although Wi-Fi technology 134, Bluetooth technology 136, IR technology 138 and cable technology 140 are being shown as the element that separates among Figure 1B; Such technology also can be integrated into processing environment (as; MMR computing machine 112, network medium server 114, acquisition equipment 106, or the like).In addition, the 100b of MMR system further comprises the geographic position mechanism 142 of or wire communication wireless with ISP's server 122 or network 128.This also can be integrated among the acquisition equipment 106.
The incident that office inlet 120 occurs for the environment that is used for catching MMR user 110, the incident that for example occurs in MMR user 110 the office, selectable mechanism.Office's inlet 120 is for example, to be located away from the computing machine of MMR computing machine 112.In this situation, office inlet 120 is connected directly to MMR computing machine 112 or is connected to MMR computing machine 112 through network 128.Alternately, office's inlet 120 is presented in the MMR computing machine 112.For example, office inlet 120 makes up from traditional personal computer (PC), and enlarges with the suitable hardware of supporting any acquisition equipment that is associated 106 then and enrich.Office's inlet 120 can comprise acquisition equipment, for example video camera and audio sound-recording machine.Alternately, the data from MMR computing machine 112 can caught and store to office's inlet 120.For example, office's inlet 120 can receive and monitor function and the incident that occurs on the MMR computing machine 112.As a result, office inlet 120 can write down all Voice & Videos in MMR user 110 the physical environment, and all incidents that occur on the record MMR computing machine 112.In a particular embodiment, the incident that office's inlet 120 is caught from MMR computing machine 112, the video screen during like positive Edit Document is caught.When doing like this, office's inlet 120 is caught when creating given document, the website of being browsed and other document of being consulted.Can utilize for MMR user 110 through his/her MMR computing machine 112 or acquisition equipment 106 after a while.In addition, office's inlet 120 can be added into the multimedia server of the montage of its document as the user.In addition, office inlet 120 can be caught other office incident, for example when the talk of paper document appearance on the table the time (as, phone or office), the discussion on the phone and the little meeting in the office.Through the use of the identical content-based retrieval technology that develops for acquisition equipment 106, the video camera (not shown) on office's inlet 120 can be discerned the paper document on MMR user 110 the physics desktop.
ISP's server 122 can be through the information of network 128 visits or any commerce server of application program for the MMR user 110 who keeps the MMR 100b of system.Especially, ISP's server 122 is any ISP's related with the 100b of MMR system representative.ISP's server 122 is that for example, wired TV supplier's commerce server is like Comcast company; Cellular telephone services supplier is like Verizon Wireless; ISP is like the inferior communication of Ah's Delphi; The Online Music ISP is like Sony; And the like, but be not limited thereto.
Wi-Fi technology 134, Bluetooth technology 136 and IR technology 138 are the representative of the technology of the radio communication between the convenient electronic installation.As known, Wi-Fi technology 134 be with based on the related technology of the wireless lan (wlan) product of 802.11 standards.As known, Bluetooth technology 136 is to describe the use that connects through short-distance radio, and cellular phone, computing machine and PDA be interconnected telecommunication industrial specification how.IR technology 138 allows electronic installation through short range radio signals communication.For example, IR technology 138 is that TV remote controller, laptop computer, PDAs and other install employed sight line wireless communication medium.IR technology 138 is worked to the frequency spectrum below the visible light at microwave therefrom.In addition, in one or more other embodiment, can use IEEE 802.15 (UWB) and/or 802.16 (WiMAX) standard support of wireless communication.
In operation, acquisition equipment 106 is used as the client computer that MMR user 110 is had.Exist on it to make the content-based retrieval operation to carry out, and acquisition equipment 106 is connected to the software application of the infrastructure of the 100b of MMR system through cellular infrastructure 132, Wi-Fi technology 134, Bluetooth technology 136, IR technology 138 and/or cable technology 140.In addition; On MMR computing machine 112, exist to carry out picture print catch operation, event capturing operate (as; The edit history of preservation document), server operation (as; The data and the incident that are used for being supplied to other object after a while and on MMR computing machine 112, are preserved) or the printer management operation (as, can printer 116 be installed as the needed data queue with the MMR as document layout and multimedia clips) such, but the software application of the several operations that are not limited thereto.Network medium server 114 provides and is attached to document printing, the document printing of printing like the MMR computing machine through belonging to MMR user 110 112 118, the visit of data.When doing like this, second medium like video or audio frequency, with first medium, associates like paper document.Hereinafter with reference Fig. 2 E, 3,4 and 5 describes related software application and/or the machine-processed more details that are used to form second medium to the first medium.
Acquisition equipment
Fig. 2 A, 2B, 2C and 2D illustrate exemplary acquisition device 106 according to an embodiment of the invention.More clearly, Fig. 2 A is depicted as the acquisition equipment 106a of camera cell phone.Fig. 2 B is depicted as the acquisition equipment 106b of PDA device.Fig. 2 C is depicted as the acquisition equipment 106c of computer peripheral devices.A web camera that example is any standard of computer peripheral devices.Fig. 2 D is depicted as it is presented to the acquisition equipment 106d in the calculation element (like, MMR computing machine 112).For example, acquisition equipment 106d is a computer graphics card.Can find the exemplary details of acquisition equipment 106 with reference to figure 2E.
In the situation of acquisition equipment 106a and 106b, acquisition equipment 106 can for MMR user 110 all, and can follow the tracks of its physical location through geographic position mechanism 142 or through each cell tower in the cellular infrastructure 132 ID number.
With reference now to Fig. 2 E,, the functional block diagram according to an embodiment of acquisition equipment 106 of the present invention is shown.Acquisition equipment 106 comprises at least one of processor 210, display 212, keypad 214, memory storage 216, wireless communication link 218, wire communication link 220, MMR software suite 222, acquisition equipment user interface (UI) 224, document fingerprint matching module 226, third party software module 228 and multiple catch mechanism 230.Exemplary acquisition mechanism 230 comprises the high bright device of video camera 232, digital camera 234, phonographic recorder 236, electronics 238, laser instrument 240, GPS device 242 and RFID reader 244, but is not limited thereto.
Processor 210 is CPU (CPU), like the Pentium microprocessor of Intel company's manufacturing, but is not limited thereto.Display 212 is the video display mechanism of any standard, as in the graspable electronic installation employed those.More clearly, for example, display 212 is any digital indicator, like LCD (LCD) or Organic Light Emitting Diode (OLED) display.Keypad 214 is the alphanumeric symbol input mechanism of any standard, employed keypad in criterion calculation device and the graspable electronic installation as the honeycomb fashion mobile phone.Memory storage 216 is any volatibility or Nonvolatile memory devices, for example, as know, hard disk drive or random access memory (RAM) device.
Acquisition equipment user interface (UI) 224 is for being used to operate the user interface of acquisition equipment 106.Through using acquisition equipment UI 224,, various menus are presented to MMR user 110 for the selection of above that function.More clearly; The menu of acquisition equipment UI 224 allows MMR user's 110 management roles; As mutual with paper document, from existing document sense data, with data write existing document, check and with those documents related augmented reality mutual and check and mutual with the augmented reality of the document associations that on his/her MMR computing machine 112, is shown, but be not limited thereto.
Document fingerprint matching module 226 is for being used for extracting from least one text image of catching of the catch mechanism 230 through acquisition equipment 106 software module of characteristic.Document fingerprint matching module 226 also can be carried out the pattern match between the database of the image of being caught and document.In the most basic level, and according to an embodiment, document fingerprint matching module 226 is confirmed the position of the image fragment in the bigger page-images, and wherein that page-images is from very big document sets, to select.Document fingerprint matching module 226 comprises data that reception catches, from the expression of the extracting data image of being caught, carry out the fragment identification in the document and move the routine and the program of a row x-y position of analyzing, carrying out the decisive combination and the output page that input picture was positioned at.For example, in order to discern document and the chapters and sections in the document that wherein extracts it, document fingerprint matching module 226 can be the level that combines from the image of the fragment of text, to be extracted and the algorithm of vertical features.In case extracted characteristic, for the distinguished symbol document, just inquiry for example, is positioned at the document printing index (not shown) on MMR computing machine 112 or the network medium server 114.Under the control of acquisition equipment UI 224, document fingerprint matching module 226 addressable document printing index.MMR computing machine 112 with reference to figure 3 is described the document printing index in further detail.Notice that in an alternative embodiment document fingerprint matching module 226 possibly be the part of MMR computing machine 112, is not positioned at acquisition equipment 106.In such embodiment, acquisition equipment 106 is sent to MMR computing machine 112 with the original data of catching, so that image extraction, pattern match and document and location recognition.In another embodiment, document fingerprint matching module 226 is only carried out feature extraction, and the characteristic of being extracted is sent to MMR computing machine 112, so that pattern match and identification.
Third party software module 228 is the representative of any third party software module of being used to strengthen any operation that possibly on acquisition equipment 106, occurred.Exemplary third party software comprises fail-safe software, image aware software, image processing software and MMR database software.
As mentioned above, acquisition equipment 106 can comprise any amount of catch mechanism 230, will describe its example now.
Digital camera 234 be can capture digital image any standard digital camera apparatus.
The high bright device 238 of electronics is for providing the high bright device of electronics of scanning, storage and transmission print text, bar code and little image to the ability of PC, laptop computer or PDA device.For example, the fast link hand held scanner that the high bright device 238 of electronics is a WizCom Technologies company, its permission information is stored on the pen or through serial port, infrared communication or USB adapter, directly is passed to computer applied algorithm.
As know, laser instrument 240 is for producing relevant, approaching monochromatic light source through stimulated emission.For example, laser instrument 240 is the laser diode of standard, and it launches the semiconductor device of coherent light for when applying forward bias.Related with laser instrument 240 and be included in wherein be to measure the detector of total amount that laser instrument 240 is guided in this light that image reflected.
In a particular embodiment, acquisition equipment 106 comprises at least one of processor 210, display 212, keyboard 214, memory storage 216, wireless communication link 218, wire communication link 220, MMR software suite 222, acquisition equipment UI 224, document fingerprint matching module 226, third party software module 228 and catch mechanism 230.When doing like this, acquisition equipment 106 is a global function device.Alternately, acquisition equipment 106 can have less function, and thereby can comprise one group of limited functional part.For example; MMR software suite 222 can be remotely located at document fingerprint matching module 226; For example, the MMR computing machine 112 of the 100b of MMR system or network medium server 114 places, and by acquisition equipment 106 through wireless communication link 218 or wire communication link 220 visits.
The MMR computing machine
With reference now to Fig. 3,, the MMR computing machine 112 that disposes according to embodiments of the invention is shown.As can see that MMR computing machine 112 is connected to the network medium server 114 that comprises one or more multimedias (MM) file 336, produces user's printer 116, the document scanner 127 of document printing 118 and comprise acquisition equipment UI 224 and the acquisition equipment 106 of first instance of document fingerprint matching module 226.Communication linkage between these parts can directly link or pass through network.In addition, document scanner 127 comprises second instance of document fingerprint matching module 226 '.
The MMR computing machine 112 of this exemplary embodiment comprises one or more source files 310; First source document (SD) browser 312; The 2nd SD browser 314; Printer driver 316; Document printing (PD) trapping module 318; The document event database 320 of storage PD index 322; Event capturing module 324; Document analysis device module 326; Multimedia (MM) montage browser/editor module 328; The printer driver 330 of MM; Document-video paper (DVP) print system 332; With video paper document 334.
The one SD browser 312 and the 2nd SD browser 314 are independent PC application program or the plug-in unit about the existing PC application program of the visit that the data that have been associated with source file 310 are provided.The first and second SD browsers 312,314 can be used for retrieving original html file or MM montage, on MMR computing machine 112, to show.
As know, printer driver 316 is the printer driver software of the communication linkage between controlling application program and PDL or any special printer employed printer control language.Especially, no matter when print a document, like document printing 118, printer driver 316 all will have the data of correct control command, those of the printing equipment that is used for them that provided of company of Ricoh for example, the printer 116 of feeding.In one embodiment, printer driver 316 is different from traditional print driver, because it catches the expression of x-y coordinate, font and the point size of each character on each printer page automatically.In other words, it catches the information of the content of relevant each document of being printed, and those data are fed back to PD trapping module 318.
According to one embodiment of present invention, document event database 320 is any standard database of changing for the relation between storage print document and the incident.(with reference to figure 34A, hereinafter further is described as the MMR database with document event database 320).For example, document event database 320 storage from source file 310 (as, Word, HTML, pdf document) to the bi-directional chaining of the incident that is associated with document printing 118.Exemplary event comprises has printed the note of catching, with the client applications of acquisition equipment 106 multimedia be added into document or multimedia clips that promptly is engraved in multimedia clips on the acquisition equipment 106 after the Word document.In addition, can be stored in the document event database 320, comprise with source file 310 other related incidents and when opening, closing or removing given source file 310, charge to daily record; , given source file 310 charges to daily record when being in the applications active on the desktop of MMR computing machine 112; Daily record is charged in document " duplicates " and " moving " operates time and destination; And the edit history of given source file 310 charged to daily record.Such incident is caught and is stored in the document event database 320 by event capturing module 324.Connect document event database 320 and come the output of reception sources file 310, event capturing module 324, PD trapping module 318 and scanner 127, and also be connected to acquisition equipment 106, inquire about and data to receive, and output is provided.
Document event database 320 is also stored PD index 322.PD index 322 is the software application on the sign format that will be from the Feature Mapping that image extracted of document printing to them (like the image of scanning to Word).In one embodiment, PD trapping module 318 is given x-y position that PD index 322 provides each character on the printer page and font, point size, or the like.When printing given document, make up PD index 322.Yet, catch all print datas and can it being kept in the PD index 322 in the mode of after a while time inquiry.For example, if document printing 118 comprises the word " garden " that is physically located at delegation on the word on the page " rose ", then such inquiry (that is, word " garden " is on word " rose ") supported in PD index 322.PD index 322 comprises the record that word " garden " above that appears at which position in which document, which page and those pages on the word " rose ".Thereby, organize PD index 322, to support based on characteristic or text based inquiry.Through using PD trapping module 318 during the printing and/or, produce content as the PD index 322 of the electronic representation of document printing through during scan operation, using the document fingerprint matching module 226 of document scanner 127.Hereinafter will be with reference to the other architecture and the function of figure 34A-C, 35 and 36 descriptive data bases 320 and PD index 322.
The exemplary functions of event capturing module 324 comprises: 1) tracking-tracking activity file and application program; 2) thump catch-thump catches related with applications active; 3) frame buffer catch with index demarcate-index for each frame buffer image with optical character identification (OCR) result of frame buffer data, the time that consequently can chapters and sections and its of document printing be shown on screen is complementary.Graphical display interface (GDI) the shade d11 of the text drawing command of the PC desktop of alternately, can enough seizure being issued by PC operating system catches text.MMR user 110 can point to document with acquisition equipment 106, and determines when that it is movable on the desktop of MMR computing machine 112; And 4) read history to catch-in order how long to follow the tracks of; And which part of special document is visible for MMR user 110, and frame buffer is caught with the data of index proving operation and is connected with the analysis of document time of activity on the desktop of his/her MMR computing machine 112.When doing like this, whether reading document in order to infer MMR user 110, related with other incident possibly appear, move like button or mouse.
Being combined on the MMR computing machine 112 of document event database 320, PD index 322 and event capturing module 324 realizes partly, perhaps alternately, realizes as the database of sharing.If realize partly, and realize comparing with the mode of sharing, then need less security.
Document analysis device module 326 is a software application; It analyzes the source file 310 relevant with each document printing 118; To be positioned at useful object wherein; Like URL (URL), address, title, author, time or locative phrase, as, Hallidie Building.When doing like this, confirm the position of those objects in the printing edition of source file 310.Receiving trap can use the output of document analysis device module 326 then, with statement with other information amplification document 118, and the accuracy of raising pattern match.In addition, for example in the situation of URL, receiving trap also can take to move the webpage that the use location retrieval is associated with URL.Connect document analysis device module 326 with reception sources file 310, and this module offers document fingerprint matching module 226 with its output.Although only as being connected to the document fingerprint matching module 226 of acquisition equipment and illustrating, the output of document analysis device module 326 can be connected to all or any amount of document fingerprint matching module 226, and no matter where they are positioned at.In addition, the output of document analysis device module 326 also can be stored in the document event database 320, so that use after a while.
MM montage browser/editor module 328 is for providing the software application of creation function.MM montage browser/editor module 328 be independent software application or, alternately,, be the plug-in unit (by representing) that moves on the document viewer to the dotted line of the 2nd SD browser 314.MM montage browser/editor module 328 is shown to the user with multimedia file, and is connected to the network medium server, to receive multimedia file 336.In addition, when MMR user 110 just creating document (as, multimedia clips is attached to paper document) time, MM montage browser/editor module 328 is the support facility of this function.MM montage browser/editor module 328 is for illustrating metadata, the information of analyzing like the document of printing from the moment of approaching when catching multimedia, application program.
The printer driver 330 of MM provides creation MMR the ability of document.For example, add bright text among the UI that MMR user 110 can be produced at the printer driver 330 by MM, and will comprise retrieving multimedia data or be added into the text in the action of carrying out some other processes on the network 128 or on MMR computing machine 112.The printer driver 330 of MM and DVP print system 332 combine to provide alternative output format of using bar code.This form must not need the content-based retrieval technology.The printer driver 330 of MM is for being used to support video paper technology, that is, video paper 334, printer driver.The printer driver 330 of MM is created and is comprised that the papery of bar code representes, as the multimedia mode of visit.Comparatively speaking, printer driver 316 is created the papery that comprises the MMR technology and is represented, as the multimedia mode of visit.The origination techniques that is embodied in the combination of MM montage browser/editing machine 328 and SD browser 314 can be created the output format identical with SD browser 312, thereby makes it possible to carry out the establishment of the MMR document prepared for content-based retrieval.Any data in the document event database 320 that 332 execution of DVP print system are associated with document are printed the attended operation of expression to it, perhaps imply bar code with clear and definite perhaps using.The pattern of the text feature that implicit bar code refers to use as bar code.
The MM file 336 of network medium server 114 is the representative of any set of multiple file type and file layout.For example, MM file 336 is text source file, webpage, audio file, video file, audio/video file and image file (like, digital photograph).
As described in Figure 1B, document scanner 127 is used for the conversion of existing document printing to MMR-preparation document.Yet, continuing with reference to figure 3, each page of the document that is applied to through the feature extraction operation with document fingerprint matching module 226 ' to be scanned is used for the existing document that MMR-enables with document scanner 127.Subsequently, increase PD index 322 with the scanning and the result of feature extraction operation, and thereby, the electronic representation of the document that is scanned is stored in the document event database 320.Can the information in the PD index 322 be used to create the MMR document then.
Continuation is noticed the software function of MMR computing machine 112 and not only is confined to MMR computing machine 112 with reference to figure 3.Alternately, can the software function shown in Fig. 3 be distributed in the configuration of the Any user definition between MMR computing machine 112, network medium server 114, ISP's server 122 and the acquisition equipment 106 of the 100b of MMR system.For example; Printer driver 330 and the DVP print system 332 of source file 310, SD browser 312, SD browser 314, printer driver 316, PD trapping module 318, document event database 320, PD index 322, event capturing module 324, document analysis device module 326, MM montage browser/editor module 328, MM can be positioned within the acquisition equipment 106 fully; Thereby and, the function of enhancing is provided for acquisition equipment 106.
The MMR software suite
Fig. 4 illustrates one group of included in the MMR software suite 222 according to an embodiment of the invention software part.Should be appreciated that, can comprise in MMR computing machine 112, acquisition equipment 106, network medium server 114 and other server in the MMR software suite 222 all or some.In addition, other embodiment of MMR software suite 222 possibly have from their one to all any amount of parts of illustrating.The MMR software suite 222 of this example comprises: multimedia is explained software 410, and it comprises searching part 412 based on content of text, based on the searching part 414 and the secret writing change parts 416 of picture material; Paper reads history log 418; The online history log 420 that reads; Collaborative document is consulted parts 422, real-time informing parts 424, multimedia retrieval parts 426; Desktop video reminder feature 428; Webpage reminder feature 430, physics history log 432; Complete form is consulted device parts 434; Time transfer unit 436, position inform that parts 438, PC create parts 440; Document production parts 442; Acquisition equipment creation parts 444; Unconscious upload component 446; Documentation release searching part 448; PC document metadata parts 450; Acquisition equipment UI parts 452; With specific area parts 454.
According to a specific embodiment, multimedia is explained the basic fundamental that software 410 combines the tissue formation MMR 100b of system of document event database 320.More clearly, to explain software 410 be that the multimedia that is used to manage paper document is explained to multimedia.For example, MMR user 110 points to any chapters and sections of paper documents with acquisition equipment 106, and uses at least one of catch mechanism 230 of acquisition equipment 106 to come to add to those chapters and sections then and explain.In a particular example, the lawyer gives an oral account the record (establishment audio file) of the chapters and sections of relevant contract.Multi-medium data (audio file) is attached to automatically the original electron version of document.Text printout subsequently comprises the indication of the existence of those notes alternatively.
Searching part 412 based on content of text is a software application of from text, retrieving content-based information.For example, through using the searching part 412 based on content of text, retrieval of content from the text fragment is discerned original document and chapters and sections in the document, and perhaps identification is connected to the out of Memory of that fragment.Searching part 412 based on content of text can utilize the technology based on OCR.Alternately, be used for carrying out from the two-dimensional arrangement that does not comprise the word length of text fragment of the operation of the content-based retrieval of text based on the technology of OCR.Based on an example of the searching part 412 of content of text is the level from the image of text fragments, extracted of combination and the algorithm of vertical features, with identification document and the chapters and sections in the document that wherein extracts it.Can serial ground, concurrently or side by side usage level and vertical features.Use like this, so that realization of High Speed and robustness are provided when noise occurring not based on the feature set of OCR.
Searching part 414 based on picture material is a software application of from image, retrieving content-based information.Carry out the data and the image between the image in the database 320 of being caught based on the searching part 414 of picture material and compare, to produce possible images match of row and the confidence level that is associated.In addition, each images match can have the data that are associated or respond user's input and the action of execution.In an example; Through with image transitions for being used for inquiring about vector representation about the image data base of image with identical feature placement, can retrieve based on the searching part 414 of picture material, for example; Content based on raster image (for example, map).Alternate embodiments is used the color content of image or the geometric arrangement of the object in the image, in database, to search matching image.
Secret writing change parts 416 are for carrying out the software application of secret writing change before printing.For the MMR application program is operated better, before print text and image, numerical information is added into said text and image.In alternative embodiment, secret writing change parts 416 produce and store M MR document, and the document comprises: 1) the original substance as text, audio frequency or video information; 2) with any picture text, audio frequency, video, the Applets of Java, hypertext link, or the like the other content that exists of such form.The secret writing change can be included in embed watermark in colour or the gray level image, the printing of the dot pattern on the document background, the perhaps trickle change of the profile of printable character to encoded digital information.
Paper reads the read history log of history log 418 for paper document.Paper reads history log 418 and is positioned at, for example, and in the document event database 320.Paper read history log 418 be based on by Creative Company of Ricoh exploitation from the document recognition technology of video, it is used to produce the history of the document that MMR user 110 read.For example, for reminding reading and/or any incident that is associated of MMR user's 110 documents, it is of great use that paper reads historical diary 418.
The online history log that reads that reads history log 420 for online document.Onlinely read the analysis that history log 420 is based on OS Events, and be positioned at, for example, in the document event database 320.The online history log 420 that reads is MMR user 110 online document that is read and which record partly that reads document.Can the online clauses and subclauses that read history log 420 be printed in any printout subsequently with many modes, for example notes are provided through bottom at each page, perhaps bright based on the text that reads every section time quantum that is spent through adding with various colors.In addition, multimedia note software 410 can enroll this data in the PD index 322.Alternatively, can be through being equipped with the online history log 420 that reads of MMR computing machine 112 assistance of the device as the face detection system of monitoring MMR computing machine 112.
Collaborative document is consulted parts 422 for through his/her acquisition equipment 106 being pointed to any chapters and sections of documents, allows more than reader of the different editions of identical paper document to consult the software application of the applied note of other reader.For example, can be shown as the overlayer on the document sketch map note on the acquisition equipment 106.Collaborative document is consulted parts 422 and can be used the existing cooperation software of any kind to realize, or with the existing cooperation software cooperation of any kind of.
Real-time informing parts 424 are the software application of carrying out the real-time informing of the document that just is being read.For example, when MMR user 110 read document, his/her read trace and is posted up on blog or the online bulletin board.As a result, to identical topic interested other people can visit and talk about the document.
Desktop video reminder feature 428 is the software application of the incident of reminding MMR user 110 and on MMR computing machine 112, occurring.For example, through acquisition equipment 106 being pointed to chapters and sections of paper document, MMR user 110 can see the video clipping of the variation of the desktop that the MMR computing machine 112 that when those chapters and sections are visible, takes place is shown.In addition, desktop video reminder feature 428 can be used to retrieve other multimedia that MMR computing machine 112 is write down, for example the audio frequency that is appeared on every side of MMR computing machine 112.
The webpage of webpage reminder feature 430 for reminding MMR user 110 on his/her MMR computing machine 112, to be checked.For example, through on paper document, waving the camera lens of acquisition equipment 106, MMR user 110 can see the trace of the webpage of when the corresponding chapters and sections of document are shown on the desktop of MMR computing machine 112, being checked.Can perhaps on the display 212 of acquisition equipment 106, webpage be shown in SD browser 312, browser 314.Alternately, webpage is presented on as original URL on the display 212 of acquisition equipment 106 or on the MMR computing machine 112.
Physics history log 432 is present in, for example, and in the document event database 320.Physics history log 432 is the physics history log of paper document.For example, MMR user 110 points to paper documents with his/her acquisition equipment 106, and through using institute's canned data in the physics history log 432, can confirm and other adjacent document of interested documents sometime in the past.For example, the similar tracker of RFID can convenient this operation.In this situation, acquisition equipment 106 comprises RFID reader 244.
Complete form consults that device parts 434 obtain for retrieval is previous is used to improve the software application of the information of form.For example, MMR user 110 points to blank form (for example, the medical claim form of printing from the website) with his/her acquisition equipment 106, and provides the history of the information of before being imported.Subsequently, consult this previous information of importing of device parts 434 usefulness by this complete form and fill this form automatically.
The software application of parts 438 for the known paper document in management position informed in the position.The management of the paper document of for example, the similar tracker of RFID is convenient location aware.For example, acquisition equipment is caught the trace in MMR user 110 geographic position 106 all day, and scans the RFID label that is attached to document or comprises the file of document.The RFID scan operation is carried out by the RFID reader 244 of acquisition equipment 106, to survey any RFID label in its scope.Can pass through the identifier of each cell tower in the cellular infrastructure 132, perhaps alternately, via the GPS device 242 of the acquisition equipment 106 that combines with geographic position mechanism 142, the geographic position of following the tracks of MMR user 110.Alternately, can accomplish document recognition with the video camera 232 of " video of always opening " or acquisition equipment 106.Position data provides " geo-location reference " document, and it enables to illustrate all day document and is positioned at the interface based on map where.An application possibly be the lawyer who carries file visit remote client.In alternative embodiment, document 118 comprise when moving document and carry out some preliminary face detection operations can perception perception mechanism attached to it.Perceptional function is via the one group of gyrostat that is attached to paper document or similarly installs.Position-based information, the 100b of MMR system indication is " calling " possessory cellular phone when, to tell him document is just mobile.Cellular phone can be added into its virtual briefcase with that document.In addition, this is the notion of " invisible " bar code, and it is that the video camera 232 or the digital camera 234 of acquisition equipment 106 is visible, but is sightless or very faint machine readable mark for the people.Can consider can be decoded on acquisition equipment 106 various black mark and secret writing or, the print image digital watermark is confirmed the position.
The software application that document production parts 442 are operated for the creation of carrying out existing document.Can, for example, perhaps, perhaps realize document production parts 442 as enterprise version as individual's version.In individual version, MMR user's 110 scanned documents also are added into MMR document database (for example, the document event database 320) with them.In enterprise version, publisher (perhaps third party) is (perhaps soft-proof originally) establishment MMR document from the original electron source.This function can be embedded into high-end issue bag (for example, AdobeReader) and the backstage that is provided with another entity service be connected.
Acquisition equipment creation parts 444 are the software application of directly on acquisition equipment 106, carrying out the creation operation.Use acquisition equipment creation parts 444, MMR user 110 extracts key phrase from his paper document at hand, and this key phrase and the other content of dynamically catching are together stored, to create interim MMR document.In addition; Through using acquisition equipment creation parts 444; MMR user 110 can be back to his/her MMR computing machine 112; And with interim MMR profile download that he created to existing document application program, like PowerPoint, then its editor is become other type of document of last version or the Another application program of MMR document.When doing like this, can image and text automatically be inserted in the page of existing document, in the page that inserts the PowerPoint document.
(automatically, no user intervene ground) is uploaded to document printing the software application of acquisition equipment 106 to unconscious upload component 446 for unconsciously.Because at most of time acquisition equipment 106 all is that MMR user 110 owns; Comprise when MMR user 110 in 112 last times of his/her MMR computing machine; Except document being sent to the printer 116; In conjunction with Wi-Fi technology 134 or Bluetooth technology 136, via the wireless communication link 218 of acquisition equipment 106, if perhaps acquisition equipment 106 is connected/docks with MMR computing machine 112; Then through wired connection, printer driver 316 also can the document that those are identical be pressed into the memory storage 216 of acquisition equipment 106.By this way, after document printing, MMR user 110 never can forget and picks the document, because it automatically is uploaded to acquisition equipment 106.
Documentation release searching part 448 is the past of the given source file 310 of retrieval and the software application of version in future.For example, MMR user 110 points to document printing with acquisition equipment 106, and then the current source file 310 (for example, Word file) in documentation release searching part 448 location and source file 310 other in the past with the version in future.In a particular embodiment, this manipulates the Windows file tracking software that this position is duplicated source file 310 and be displaced in tracking.Equally here also can use other such file to follow the tracks of software.For example, the word can be enough selected from source file 310 of Google WDS or Microsoft Windows search assistant and the current version of file is found in the inquiry formed.
PC document metadata parts 450 are the software application of the metadata of search file.For example, MMR user 110 points to document printing with acquisition equipment 106, and whom PC document metadata parts 450 confirm and printed the document, the document of when printing, and document is printed wherein, and at the file path of printing given source file 310 constantly.
Acquisition equipment UI parts 452 are the software application of the operation of the UI of management acquisition equipment 106, and it allows MMR user 110 and paper document mutual.Acquisition equipment UI parts 452 combine permission MMR user 110 from existing document reading of data with acquisition equipment UI's 224; And data are write existing document; Check and with those documents related augmented reality mutual (promptly; Through acquisition equipment 106, MMR user 110 can check when creating document or what take place during at Edit Document), and check and with the document that on his/her acquisition equipment 106, shows related augmented reality mutual.
Continuation notices that with reference to figure 3 and 4 software part of MMR software suite 222 can completely or partially be present on one or more MMR computing machines 112, network medium server 114, ISP's server 122 and the acquisition equipment 106 of the 100b of MMR system.In other words; Can be with the operation of the 100b of MMR system; The for example performed any operation of MMR software suite 222 is distributed in the configuration of the Any user definition between MMR computing machine 112, network medium server 114, ISP's server 122 and the acquisition equipment 106 (perhaps among the 100b of system included other such processing environment).
According to this open invention, will be clearly, can carry out the basic function of the 100a/100b of MMR system with some combination of the software part of MMR software suite 222.For example, the basic function of the embodiment of the 100a/100b of MMR system comprises:
● creating perhaps increases the MMR document that comprises first medium part and second medium part;
● use first medium part (for example, paper document) of MMR document to visit the information in second medium part;
● use first medium part (for example, the paper document) triggering of MMR document or start the process in the electronic applications;
● use first medium part (for example, paper document) of MMR document to create or increase by the second medium part;
● use second medium of MMR document partly to create or increase by the first medium part;
● use second medium of MMR document partly to trigger or start in the electronic applications or with the relevant process of the first medium part;
The MMR document
Fig. 5 illustrates the diagram of MMR document 500 according to an embodiment of the invention.More clearly, Fig. 5 illustrates expression 502, action or second medium 504, index or the focus 506 of a part that comprises document printing 118 and whole MMR documents 500 of the electronic representation 508 of documents 118.Although typically MMR document 500 is stored in document event database 320, also can with its be stored in acquisition equipment or be connected to network 128 any other the device in.In one embodiment, a plurality of MMR documents can be corresponding to document printing.In another embodiment, the structure shown in the reconstructed chart 5 is to create a plurality of focuses 506 in single document printing.In a particular embodiment, MMR document 500 comprises the expression 502 and focus 506 with the position in the page and the page; Second medium 504 and electronic representation 508 are optional and as by dotted line, drawing.Notice,, can add second medium 504 and electronic representation 508 after a while creating the MMR document if need in this way.This basic embodiment can be used for locating document or corresponding to the specific position in the document of expression.
The expression 502 of the part of document printing 118 can be to exist with any form (image, vector, pixel, text, code, or the like) that is applicable to pattern match and discerns at least one position in the document.The position that expression 502 is preferably discerned in the document printing uniquely.In one embodiment, expression 502 is text fingerprints as shown in Figure 5.During printing, catch text fingerprints 502 automatically via PD trapping module 318, and it is stored in the PD index 322.Alternately, during scan operation, catch text fingerprints 502 automatically via the document fingerprint matching module 226 ' of document scanner 127, and it is stored in the PD index 322., represent that then 502 alternately can be whole documents, text fragment, word if but it is the unique instance in the document, the some of image, unique attribute or any other expression of document compatible portion.
The action or second medium 504 are preferably the data structure of digital document or any kind.Second medium 504 among the most basic embodiment can be one or more orders that the text that will appear maybe will be carried out.Second media type 504 more typically be with by the expression 502 documents identified relevant text, audio file or the video file of a part.Second media type 504 possibly be note or data structure or the file that comprises a plurality of files of a plurality of different medium types and same type.For example, second medium 504 can be text, order, image, pdf document, video file, audio file, application file (as, spreadsheet or word processing file), or the like.
Index or focus 506 for expression 502 with move or second medium 504 between link.Focus 506 makes expression 502 related with second medium 504.In one embodiment, index or focus 506 comprise as x in the document and the positional information the y coordinate.Focus 506 possibly be point, zone or even whole document.In one embodiment, focus is the pointer with expression 502, the pointer of second medium 504 and the data structure of the position in the document.Should be understood that MMR document 500 possibly have a plurality of focuses 506, and in such situation, data structure is created the link between a plurality of positions in a plurality of expressions, a plurality of second medium file and the document printing 118.
In alternative embodiment, MMR document 500 comprises the electronic representation 508 of whole documents 118.This electronic representation can be used to confirm the position of focus 506, and also can be used for display document on acquisition equipment 106 or MMR computing machine 112 through user interface.
The exemplary use of MMR document 500 is following.Through analyzing text fingerprints or representing 502, discern the text fragments of being caught via the document fingerprint matching module 226 of acquisition equipment 106.For example, MMR user 110 points to document printing 118 with the video camera 232 or the digital camera 234 of his/her acquisition equipment 106, and catches image.Subsequently, document fingerprint matching module 226 is carried out it and is analyzed on the image of being caught, to confirm whether there are the clauses and subclauses that are associated in the PD index 322.If find occurrence, on the display 212 of his/her acquisition equipment 106, be the existence of MMR user's 110 high bright demonstration focuses 506.As shown in Figure 5, high bright demonstration word or expression.Each focus 506 in the document printing 118 all are used as to other user-defined or predetermined data, as are present in of MM file 336 on the network medium server 114, link.Institute's stored text fingerprint or represent that 502 visit allows electronic data is added into any focus 506 in any MMR document 500 or the document in the PD index 322.As described with reference to figure 4, comprise at least one focus 506 (as, the link) paper document be called as MMR document 500.
Until 2D, 3,4 and 5, the exemplary operation of the 100b of MMR system is following with reference to Figure 1B, 2A in continuation.MMR user 110 or any other entity, for example publishing house opens given source file 310 and starts printing, to produce paper document, like document printing 118.During printing, automatically perform some action, as: (1) via PD trapping module 318, is caught print format automatically, and it is passed to acquisition equipment 106 printing constantly.Be positioned at through use, for example, the PD trapping module 318 of output place of SD browser 312 is being printed the electronic representation 508 of catching document constantly automatically.For example, the content that MMR user 110 prints from SD browser 312, and this content filters PD trapping module 318.As discussed previously, when showing document, can confirm the two-dimensional arrangement of the text on the page for printing; (2) printing constantly,, catch given source file 310 automatically via PD trapping module 318; And (3) maybe can increase other interesting information that multimedia on the acquisition equipment 106 is explained interface in order to locate " entity of appointment ", via document analysis device module 326, analyzes print format and/or source file 310.The entity of appointment does, for example, is used for adding after a while multimedia " anchor ", that is, and and the focus 506 that generates automatically.Document analysis device module 326 receives the incoming source document 310 relevant with given document printing 118.Document analysis device module 326 is to discern the application program of the expression of using with focus 506 in the document 118 502, for example, and title, author, time or position, and thereby, the information that prompting will receive on acquisition equipment 106; (4) automatically give print format and/or source file 310 for content-based retrieval and index, that is, set up PD index 322; (5) in document event database 320, make clauses and subclauses about document and the incident that is associated with source file 310, for example, edit history and current location; And (6) carry out interactive session in printer driver 316, and it allows MMR user 110 before printing focus 506, said focus 506 to be added into document, and thereby form MMR document 500.With the data storage that is associated on MMR computing machine 112 or be uploaded to network medium server 114.
Exemplary alternate embodiments
MMR system 100 (100a or 100b) is not limited to the configuration shown in Figure 1A-1B, 2A-2D and the 3-5.MMR software can be allocated between acquisition equipment 106 and the MMR computing machine 112 whole or in part, and need be far fewer than preceding text with reference to figure 3 and 4 described all modules.A plurality of configurations all are possible, comprise as follows:
The first alternative embodiment of MMR system 100 comprises acquisition equipment 106 and acquisition equipment software.Acquisition equipment software be acquisition equipment UI 224 with document fingerprint matching module 226 (as, shown in Fig. 3).On acquisition equipment 106, perhaps alternately, on the external server as network medium server 114 or ISP's server 122 of addressable acquisition equipment 106, carry out acquisition equipment software.In this embodiment, the network service that the data that are connected to publication are provided capable of using.Can use graduate identifying schemes, at first discern publication therein, and discern the page and chapters and sections in the publication then.
The second alternative embodiment of MMR system 100 comprises that acquisition equipment 106, acquisition equipment software and document use software.As with reference to figure 4 shown with describe, the second alternative embodiment comprises and catches and index to document printing, and connects basic document incident, like the edit history of document, software.This allows MMR user 110 that his/her acquisition equipment 106 is pointed to any document printing, and confirms the name and the position of the source file 310 of generation the document, and definite when and where of printing.
The 3rd alternative embodiment of MMR system 100 comprises that acquisition equipment 106, acquisition equipment software, document use software and event capturing module 324.Event capturing module 324 is added into MMR computing machine 112; The incident that this computer capture is associated with document, the for example time when they are visible on the desktop of MMR computing machine 112 (confirming), the URL that when document is opened, is visited, the character of perhaps when document is opened, on keyboard, keying in through monitoring GDI character line generator.
The 4th alternative embodiment of MMR system 100 comprises acquisition equipment 106, acquisition equipment software and printer 116.In this 4th alternative embodiment, the similar communication linkage that printer 116 is equipped with bluetooth transceiver or communicates by letter with near any MMR user 110 the acquisition equipment 106 being in it.No matter when any MMR user 110 picks up document from printer 116, and printer 116 is pressed into MMR data (document layout and multimedia clips) that user's acquisition equipment 106.For the multi-medium data that obtains to be associated with specific document, user's printer 116 comprises keypad, passes through its user's login and input code.The document can be included in the printing of the code of its footer to be represented, it can insert through printer driver 316.
The 5th alternative embodiment of MMR system 100 comprises acquisition equipment 106, acquisition equipment software and office's inlet 120.Office's inlet device is preferably the personalized version of office's inlet 120.The incident that office's inlet 120 is caught in the office is like session, talks/call and meeting.120 identifications of office's inlet are also followed the tracks of the specific paper document on the physics desktop.Office's inlet 120 is carried out document recognition software (that is, document fingerprint matching module 226 and main frame document event database 320) in addition.This 5th alternative embodiment can be used for from MMR computing machine 112 computation of unloading working loads; And the facilitated method that the 100b of MMR system packing is become consumer devices (for example, the 100b of MMR system being sold as hardware and the software product on the mini computing machine of the Mac of Apple Computer, carried out) is provided.
The 6th alternative embodiment of MMR system 100 comprises acquisition equipment 106, acquisition equipment software and network medium server 114.In this embodiment, multi-medium data is present in network medium server 114, for example the Comcast ordering server.When MMR user 110 passes through to use his/her acquisition equipment 106 scanned document text fragment; The STB 126 that consequent look-up command or transfer to is associated with MMR user 110 CATV (cable TV) is (through the Internet; Wirelessly; Perhaps through beeper top box 126 on the phone), perhaps transfer to the Comcast server.In two kinds of situations, multimedia all from the Comcast server flows to STB 126.System 100 knows and whither sends data, because MMR user 110 had before registered his/her phone.Thereby, can acquisition equipment 106 be used for the visit and the control of STB 126.
The 7th alternative embodiment of MMR system 100 comprises acquisition equipment 106, acquisition equipment software, network medium server 114 and location-based service.In this embodiment, the location aware service is distinguished between a plurality of destinations from the output of Comcast system (or other appropriate communication system).Perhaps through distinguishing cell phone towers ID automatically, perhaps, carry out this function through allowing MMR user 110 to select the keypad interface of position that will video data.Thereby when another position of visit, as long as that other position has wired access, the user just can visit their program that cable television operators provided and other wired TV characteristic.
Document fingerprint matching (" based on the fragment identification of image ")
As described earlier, the document fingerprint matching relates to a part or " fragment " of discerning the MMR document uniquely.With reference to figure 6, document fingerprint matching module/system 610 receives the image 612 of being caught.Document finger print matching system 610 is inquired about the page set in the document database 3400 (for example, hereinafter with reference Figure 34 A further describes) then, and returns a row page and the document that comprises them, comprises the image 612 of being caught in it.Each result is the x-y position of the input picture 612 that occurs being caught.Those skilled in the art will notice that outside that database 3400 can be in document fingerprint matching module 610 (for example; As shown in Figure 6); But the inside that also can be in document fingerprint matching module 610 (for example; As Fig. 7,11,12,14,20,24,26,28 and 30-32 shown in, wherein document fingerprint matching module 610 comprises database 3400).
Fig. 7 illustrates the block diagram of document finger print matching system 610 according to an embodiment of the invention.Acquisition equipment 106 is caught image.The image of being caught is sent to quality assessment modules 712, and it is based on the needs and the ability of downstream, carries out the preliminary judgement about the content of the image of being caught effectively.For example, if the image of being caught is quality so, so that in downstream document finger print matching system 610, can not handle it, then quality assessment modules 712 impels acquisition equipment 106 to catch image again with higher resolution.In addition, for example, quality assessment modules 712 can be surveyed many other relevant characteristics of the image of being caught, the sharpness of the text that is comprised in the image of for example being caught, and it is the whether indication of " focusing " of the image of being caught.In addition, quality assessment modules 712 can be confirmed whether the image of being caught comprises and possibly be the something of the part of document.For example, the image fragment that comprises non-file and picture (for example, desk, outdoor scene) indicates the user just the visual field of acquisition equipment 106 to be converted to new document.
In addition, in one or more embodiments, quality assessment modules 712 can be distinguished by execution contexts/non-text, consequently only through comprising the image of discernible text.Fig. 8 illustrates the flow process that text/non-text is distinguished according to one or more embodiment.Extract many row pixels in step 810 from the input picture fragment.Typically, input picture is a gray-scale map, and each value in the row all is the integer (for 8 pixels) from zero to 255.In step 812, survey the local peaking in every row.This can carry out with the method for common " moving window " be familiar with, and the window of regular length (for example, the N pixel) slides along row in the method, each M pixel, wherein M＜N.In each step, confirm the existence of peak value through the marked difference (for example, greater than 40) of seeking grey level's value.If peak value is positioned at a position of window, then no matter when moving window and this position crossover all suppress the detection of other peak value.Also can survey the gap between the continuous peak value in step 812.Step 812 is applicable to the many row (C) in the image fragment, and in step 814 with the histogram gap width that adds up.
Other histogram of deriving in the training data with gap histogram and the known classification (in step 816) of from have database 818, being stored is compared, and exports the measurement of degree of confidence of decision and that decision of the classification (perhaps text or non-text) of relevant fragment together.The histogrammic typical outward appearance that the histogram classification consideration of step 816 is derived from the image of text; And it comprises two peaks closely; Center places the distance between the row last, and is wherein maybe integral multiple high away from other one or two littler peak at those peaks in histogram.With the measurement of statistical variance, this classification can be confirmed histogrammic shape, and perhaps it can use range observation, and for example, Hamming or Euclidean distance are compared histogram one by one with the prototype of being stored.
Equally with reference to figure 9, it illustrates the example that text/non-text is distinguished now.Handle input picture 910, with many row of sampling, with its subclass of dotted line indication.Grey level's histogram of typical row 912 shown in 914.The Y value is the grey level in 910, and the X value is the row in 910.The gap of being detected between the peak value shown in 916 in the histogram.The histogram of the gap width that is listed as from all samplings shown in 918.This example is illustrated the histogrammic shape that derives from the fragment that comprises text.
The flow process of point size that is used for the text of estimated image fragment shown in Figure 10.This flow process utilizes the blur level of image to be inversely proportional to the fact of acquisition equipment from the distance of the page.Through the ambiguous estimation amount, can estimated distance, and that distance can be used for, with respect to known " standardized " highly, with the scaled of the object in the image.This behavior can be used for estimating the point size of the text in the new image.
In the training stage 1010, in step 1012, the image capture apparatus that is used in known distance obtains to have the image (being called " calibration " image) of fragment of the text of known font and point size.Measure height in step 1014 with the text character in that image of many pixels expression.For example, this can manually carry out with the imagery annotation instrument as Microsoft's photo editor.Estimate the blur level in the calibration image in step 1016.For example, this can be undertaken by measuring with the frequency spectrum of known two-dimensional fast fourier transform.This also can unit formal representation be many pixels 1020.
When presenting " new " image in step 1024, as in time of running MMR recognition system, handle image, to cut apart and Character segmentation method localization of text with the row of knowing usually that around each character, produces bounding box in step 1026.Can express the height of those square frames with pixel.In step 1028, with step 1016 similarly mode estimate the blur level of new images.Combine these measurements in step 1030, with first estimation 1032 of the point size that produces each character (perhaps being equal to ground, every row).This can carry out through calculating equation: (calibration image is blured the fuzzy size of size/new images) * (new images text height/calibration image text height) * (calibration image font weight).This confirms the point size of the text in the calibration image in proportion, to produce the point size of the text in the input picture fragment of estimating.Can identical scale function be applied to the height of the bounding box of each character.This produces the decision about each character in the fragment.For example, if fragment comprises 50 characters, then this process will produce 50 votings about the point size of the font in the fragment.Can derive single estimation with the intermediate value of this voting then about point size.
In addition, more clearly return, in one or more embodiments, can the feedback of quality assessment modules 712 to acquisition equipment 106 be conducted to the user interface (UI) of acquisition equipment 106 with reference to figure 7.For example, feedback possibly comprise the indication that exists with sound or vibration mode, and it is indicated the image of being caught to comprise and looks like text but ambiguous something, and the indication user should make acquisition equipment 106 firm.Feedback possibly also comprise the parameter of the optical devices that change acquisition equipment 106, with the order of the quality that improves the image caught.For example, can focus, F f-stop and/or time shutter, so that improve the quality of the image of being caught.
In addition, through the needs of employed special feature extraction algorithm, can make the feedback specialization of quality assessment modules 712 to acquisition equipment 106.As described further below, feature extraction becomes symbolic representation with image transitions.In the recognition system of the length of calculating word, making caught image blurring possibly be very desirable for the optical devices of acquisition equipment 106.Although those skilled in the art will notice that such adjusting possibly produce the mankind or optical character identification (OCR) process is perhaps unrecognizable, be suitable for the image of Feature Extraction Technology well.Through instruction being fed back to acquisition equipment 106, impel acquisition equipment 106 that its camera lens is defocused, thereby and produce fuzzy image, quality assessment modules 712 can realize this point.
By control structure 714 change feedback procedures.Generally speaking, control structure 714 other parts from document finger print matching system 610 receive data and symbolic information.The execution sequence of the various steps in the control structure 714 decision document finger print matching systems 610, and can make the computational load optimization.The x-y position of the image fragment that control structure 714 identifications are received.More specifically, the information of the parameter of the needs of control structure 714 reception features relevant leaching process, the result of quality assessment modules 712 and acquisition equipment 106, and can suitably change them.This can dynamically carry out on a frame connects the basis of a frame.Among joining in the system that uses a plurality of feature extracting methods, the blurred picture that possibly need the big fragment of text, and another possibly need the high resolving power sharp focus figure of paper texture.In such situation, control structure 714 can send a command to quality assessment modules 712, indicates it to work as when having text in its visual field, produces the appropriate image quality.Quality assessment modules 712 is mutual with acquisition equipment 106, to produce correct image (for example, N blurred picture of big fragment, M image of sharp focus paper texture (high resolving power) thereafter).The progress that control structure 714 is followed the tracks of through those images of handling pipeline is to guarantee to have used corresponding characteristic extraction and classification.
Based on the needs of recognition system, the quality of image processing module 716 change input pictures.The example of the type of image change comprises sharpening, offset correction and binarization.Such algorithm comprises as mask size, the rotation of expectation and the many adjustable parameter the threshold value.
As shown in Figure 7, the feedback that document finger print matching system 610 uses from feature extraction and sort module 718,720 (hereinafter is described) is dynamically to change the parameter of image processing module 716.Feasible like this, because the user typically can point to the same position in the document with their acquisition equipment 106 continuous several seconds ground.For example, suppose that acquisition equipment 106 per seconds handle 30 frames, the frame of then how handling after a while to be caught with the possibility of result influence of the initial several frames of any sequential processes.
Characteristic extracting module 718 becomes symbolic representation with the image transitions of being caught.In an example, characteristic extracting module 718 location words, and calculate their bounding box.In another example, characteristic extracting module 718 location join parts together, and calculate the descriptor of their shape.In addition, in one or more embodiments, document finger print matching system 610 is shared the result's of features relevant extraction metadata with control structure 714, and uses that metadata to regulate the parameter of other system unit.Those skilled in the art will notice that this possibly reduce computation requirement significantly through suppressing the identification of difference qualitative data, and improve accuracy.For example, the characteristic extracting module 718 of identified word bounding box can be told the quantity of control structure 714 its row that find and " word ".If the quantity of word too high (for example, the indication input picture is a segment), then control structure 714 can indicate quality assessment modules 712 to produce fuzzyyer image.Quality assessment modules 712 can be sent to acquisition equipment 106 with appropriate signals then.Alternately, control structure 714 can order image processing module 716 to use smoothing filter.
In addition, as shown in Figure 7, between sort module 720 and database 3400, can there be feedback.In addition, those skilled in the art will remember that database 3400 can be in the outside of module 610 as shown in Figure 6.Can use the conforming decision of relevant fragment, about having other fragment of similar outward appearance, and Query Database 3400.This will compare the perfect view data of the fragment of being stored in the database 3400 with other image in the database 3400, rather than the input picture fragment is compared with database 3400.This can provide the other affirmation level about the decision of sort module 720, and can allow some pre-service of matched data.
Also can be at fragment, and be not only the symbolic representation on the view data, carry out database relatively.For example, best decision maybe the indicating image fragment comprises No. 12 Arial fonts of double pitch.The database comparison can be located the fragment in other document with similar font spacing, and only uses text meta-data, rather than image is relatively located the word layout.
In addition, database 3400, sort module 720, and control structure 714 between can have feedback.Enough from the information of eigenvector recognizing site except storing, database 3400 can be stored the original image that comprises document, with and the relevant information of the symbolic representation of graphics part.This allows control structure 714 dynamically to change the behavior of other system unit.For example, if exist two kinds to seem possible decision about given image fragment, then database 3400 possibly indicated the existence about image, through dwindle and check on the right of the zone, can eliminate their ambiguity.Control structure 714 can be sent suitable message to acquisition equipment 106, indicates it to dwindle.Characteristic extracting module 718 can be about the right of images printed check image on the document with sort module 720.
In addition, notice that the hypothesis fragment is arranged in document exactly, then database 3400 storage is about the details around the data of image fragment.This can be used for further triggering unexpected hardware and software image analysis step in the prior art.In a situation, that detailed information is provided by the printing capture systems of the detailed denotational description of preserving document.In one or more other embodiment,, can obtain similar information through scanned document.
Still with reference to figure 7,724 receptions of Position Tracking module are from the conforming information about the image fragment of control structure 714.Position Tracking module 724 uses it from the copy of the whole document file pages of database 3400 retrievals or the data structure of description document.Reference position is the anchor that the Position Tracking process begins.When quality assessment modules 712 determines that the image of being caught is fit to follow the tracks of, the view data that Position Tracking module 724 receives from acquisition equipment 106.Position Tracking module 724 also has the information about the time that since successfully having discerned previous frame, has passed.Position Tracking module 724 is used optic flow technique, and it allows it to estimate the distance that between continuous frame, has moved at acquisition equipment on the document 106.The sampling rate of given acquisition equipment 106 even the data that it is seen possibly be unrecognizable, also can be estimated its target.Through of the comparison of its view data, can confirm the estimated position of acquisition equipment 106 with the respective image data that from database document, is derived.Simple example is calculated the image of being caught and the crossing dependency of the desired image in the database 3400.
Thereby Position Tracking module 724 provides the mutual use of database images, with the process of guide position track algorithm.This permission is attached to non-text object with electronic reciprocal, like figure and image.In addition, in one or more other embodiment, can under the situation that does not have the described image comparison of preceding text/affirmation step, realize such depending on.In other words, move through the moment of estimating the acquisition equipment 106 on the page, can estimate be in the electronic link in the visual field that is independent of the image of being caught.
Figure 11 illustrates the technology of document fingerprint matching according to an embodiment of the invention." feedforward " technology shown in Figure 11 is handled each fragment independently.Extract characteristic its x-y images of positions fragment from those pages that are used to locate one or more pages and fragment and occur.For example, in one or more embodiments, the feature extraction of document fingerprint matching possibly depend on the level and the orthogonal sets characteristic (for example, word, character, piece) of the image of being caught.Can use the extraction characteristic of these groups to search the document (with the fragment in those documents) that comprises the characteristic of being extracted then.The horizontal word that can use the OCR function to discern in the image of being caught is right.The horizontal word that uses each identification then is used to confirm to comprise all right documents of horizontal word of being discerned to forming the search inquiry of database 3400, and the right x-y position of the word in those documents.For example, for horizontal word to " the, cat ", database 3400 possibly return (15, x, y), (20, x y), indicates horizontal word that " the, cat " appeared at x-y position indicated in document 15 and 20.Similarly, right for each vertical adjacent word, about the right x-y position of word in all documents of comprising the right instance of word and those documents, Query Database 3400.For example, for vertical adjacent word to " in, hat ", database 3400 possibly return (15, x, y), (7, x, y), the vertical adjacent word of indication appears at x-y position indicated in document 15 and 7 to " in, hat ".Then, document and the positional information of using database 3400 to be returned, can carry out as to which document from the various horizontal word that extracted the image of being caught to and vertical adjacent word between confirming maximum position crossovers appears.Response can be confirmed the existence of which focus and the medium that is linked, and this possibly cause discerning the document that comprises the image of being caught.
Figure 12 illustrates another document fingerprint matching technology according to an embodiment of the invention." interactive image analysis " technology shown in Figure 12 relate to Flame Image Process and the feature extraction that possibly occur before the recognition image fragment between mutual.For example, image processing module 716 possibly at first estimated the blur level in the input picture.Then, characteristic extracting module 718 is calculated from the distance of the page and the point size of image text.Then, image processing module 716 possibly use the characteristic of the font of that point size, on image, carries out the template matches step.Subsequently, characteristic extracting module 718 maybe thereby be extracted character or word feature from the result.In addition, those skilled in the art will recognize that font, point size and characteristic possibly be limited by the font in database 3400 documents.
The example of analyzing with reference to figure 12 described interactive images like preceding text shown in Figure 13.Handle the input picture fragment in step 1310, the font of the text in the estimated image fragment and point size and its distance from camera.Those skilled in the art will notice that can carry out font with known technology estimates (that is the identification of the candidate of the font of the text in the fragment).For example, can use with reference to the described flow process of Figure 10 and carry out point size and distance estimations.In addition, can use other technology, for example can easily be adapted to the known method of distance of the focal point of acquisition equipment.
Still with reference to Figure 13, use row partitioning algorithm, tectonic boundary frame around its line of text in fragment in step 1312.Using the known technology as the convergent-divergent in step 1314 is fixing size with the highly standardized of each row image.Will about the consistance of the font that detected in the image with and point size transmit 1324 to font prototype collection 1322, wherein use them to retrieve the image prototype of the character in the font of each appointment.
Be based on the function of the employed parameter of step 1314, in step 1320 with the character prototype size criteriaization in the selected font.Image classification in step 1316 can be compared the character after the size criteriaization of exporting in step 1320 with the output of step 1314, produce decision with each the x-y position in the image fragment.About each the character i that is detected in the image fragment, i=1 ... N can use the known method of image template coupling to produce picture (ci, xi; Yi, wi, the hi) output of that kind, wherein ci is the consistance of character; (xi yi) is the upper left corner of its bounding box, and hi, and wi is its width and height.
In step 1318, can be like the described execution geometric relationship of preceding text restricting data library lookup, but in a situation, can specific adaptation right in character, rather than word is right.In such situation: " a-b " possibility pointing character a and b are that level is adjacent; " a+b " possibly indicate them is vertical adjacent; " a/b " possibly indicate the southwest of a at b; And " b " possibly indicate the southeast of a at b.Can be from the xi of every pair of character, the yi value derives geometric relationship.Can organize MMR database 3400, thus its return comprise character to rather than the right row document file page of word.Step 1326 is output as and is expressed as n-tuple (documenti, pagei, xi, yi, actioni, the candidate list that input picture scorei) is complementary through the grading system ordering.
Figure 14 illustrates another document fingerprint matching technology according to an embodiment of the invention." producing and test " technology shown in Figure 14 is handled each fragment independently.It extracts characteristic from the image fragment, it is used to locate many page-images that possibly comprise given image fragment.In addition, in one or more embodiments, can carry out other extraction classifying step, the page classified with the possibility that comprises this image fragment through them.
Still with reference to preceding text with reference to Figure 14 described " produce and test " technology, can extract the characteristic of the image of being caught, and comprise the document fragment of characteristic of these extractions of maximum quantity in can identification database 3400.Further handle initial X document fragment (" candidate ") then with maximum matching characteristics.In this processing, the relative position of the characteristic in the relative position of the characteristic in the coupling document fragment candidate and the query image is compared.Relatively calculate scoring based on this.Then, identification is corresponding to the score of best coupling document fragment P.If the highest scoring is then found document fragment P, as the coupling to query image then greater than adapting to threshold value.Threshold value is adapted to many parameters, comprises, for example, the quantity of the characteristic of being extracted.In database 3400, what known document fragment P come from, and thereby, confirm that query image is from identical position.
Figure 15 illustrates the example of word boundary frame probe algorithm.Be illustrated in the Flame Image Process input picture fragment 1510 afterwards of making rotation correction.Usually be known as the slant correction algorithm, this type technology is rotated text image, so that it is arranged along transverse axis.In the bounding box probe algorithm next step is the calculating of horizontal projection profile diagram 1512.By this way, through the threshold value that known adaptation threshold value or sliding window algorithm select 1516 row to survey, consequently zone " on threshold value " is corresponding to line of text.1514 and 1518 extract and handle the zone in every row in a similar fashion, with the zone on threshold value of the word in the location indication row.The example of the bounding box that in a line of text, is detected shown in 1520.
In order to compare, can extract various characteristics with document fragment candidate.For example, can extract yardstick invariant features conversion (SIFT) characteristic, angle point characteristic, salient point, ascender, and descender, word boundary, and at interval, so that coupling.One of the characteristic that can from file and picture, extract reliably is word boundary.In case extracted word boundary, they just can form group as shown in Figure 16.In Figure 16; For example; All having the such mode of crossover word boundary forms orthogonal sets down with it above that with word boundary, and the total quantity of crossover word boundary is 3 (noticing that in one or more other embodiment the minimum number of crossover word boundary maybe be different) at least.For example, first unique point (second word square frame in second row, length is 6) has two word boundaries (length is 5 and 7) above that, and under it, has a word boundary (length is 5).Second unique point (the 4th word square frame in the third line, length is 5) has two word boundaries (length is 4 and 5) above that, and under it, has two word boundaries (length is 8 and 7).Thereby as shown in Figure 16, with the length of middle word boundary, the length of word boundary on thereafter its, and thereafter its following length of word boundary are then represented indicated characteristic.In addition, the length of noticing the word square frame can be based on any module.Thereby for some word square frames, it is possible having alternative length.In such situation, can extract and comprise all or some their characteristics of Res fungibiles.
In addition, in one or more embodiments, can extract characteristic, applying 0 expression at interval, and represent that with 1 word is regional.Example shown in Figure 17.Piece is on the right represented the word/interval region corresponding to the document fragment on the left side.
Can the characteristic of being extracted be compared with various range observations, comprise, for example, standard and Hamming distance.Alternately, in one or more embodiments, the document fragment that can use Hash table identification to have the characteristic identical with query image.In case discerned such fragment, calculating that just can be as shown in Figure 18 is from the angle of each unique point to further feature point.Alternately, the angle between can calculated characteristics point group.1802 illustrate the angle 1803,1804 and 1805 that calculates from ternary unique point.Can compare from the angle of each unique point to further feature point in the angle that is calculated and the query image then.If any angle of match point is similar, then can increase the similarity scoring then.Alternately, if the use angle group, and if similar on the angle group numeral between similar group the unique point in two images, then increase the similarity scoring then.In case between query image, calculated the scoring of each search file fragment, just select to cause the document fragment of score, and with its with adapt to threshold, whether satisfy some predetermined standards to confirm coupling.If satisfy standard, then coupling document path has been found in indication then.
In addition, in one or more embodiments, the characteristic of being extracted can be based on the length of word.Based on word height and width, each word is divided into estimated letter.When scanning on given word with under widow the time, according to it on and the interval information in the row under it, the binary value branch is tasked each of estimated letter.Represent binary code with the integer number then.For example, with reference to Figure 19, it illustrates each layout of all representing the word square frame of a word that detects in the image of being caught.Word 1910 is divided into estimated letter.With the length of (i) word 1910, (ii) the text of the row on the word 1910 arrange and (iii) the text of the row under the word 1910 arrange, this characteristic is described.Length with the takeoff word 1910 of estimated letter.On current estimated letter or under the binary coding of interval information extract the text placement information.In word 1910, only last estimated letter is on the interval; The second and the 3rd estimated letter is under the interval.In addition, be (6,100111,111110) with the feature coding of word 1910, wherein 0 expression at interval, and 1 expression is separated continuously.Rewrite with integer form, word 1910 is encoded to (6,39,62).
Figure 20 illustrates another document fingerprint matching technology according to an embodiment of the invention.Through they are classified independently, and the result combined the complementary information that " a plurality of sorter " the techniques make use different characteristic shown in Figure 20 is described.The example that is applied to this model of text fragments matching is extraction level and the vertical adjacent right length of word, and distinguishes the ranking compositor of fragment in the computational data storehouse.More specifically, for example, in one or more embodiments, by the position of " sorter " appurtenances through sort module 720 and definite characteristic.Use is used to confirm the combination of sorter of level and the vertical features of the image of being caught, to the image adding fingerprint of being caught.This consider the image of text comprise two independently information source carry out as its conforming observation, except the video sequence of word, also can use the vertical layout identification of word to extract the document of image from it.For example, as shown in Figure 21, through horizontal classification device 2112 and vertical classification device 2114, with the image of being caught 2110 classification.Except that the image that input is caught, each of sorter 2112,2114 all obtains information from database 3400, to export the ranking compositor of those document file pages that can use each classification successively.In other words, technology usage level of the multi-categorizer shown in Figure 21 and vertical features are independently with the image classification of being caught.Combine a graduate row document file page (hereinafter further describes example) according to combination algorithm 2118 then, it exports a graduate row document file page successively, and this tabulation is based on level and the vertical features of the image 2110 of being caught.Especially, in one or more embodiments, the information of using the characteristic that is detected in the relevant data storehouse 3400 how to work in coordination with generation combines other ranking compositor of branch from horizontal classification device 2112 and vertical classification device 2114.
Equally with reference to Figure 22, it illustrates about feature extraction now, the example how vertical layout combines with horizontal layout.In (a), illustrate and have the image of being caught 2200 that word is cut apart.From the image 2200 of being caught, confirm level and vertical " n-grams "." n-gram " all describes the sequence of n quantity of some characteristic quantities for each.For example, the quantity of the character in each word of three words of horizontal trigram specified level sequence.For example, for the image of being caught 2200, (b) horizontal trigram: 5-8-7 (being the quantity of the character in each of the word " upper " flatly arranged in first row of the image 2200 of being caught, " division " and " courses ") is shown; 7-3-5 (being the quantity of the character in each of the word " Project " flatly arranged in second row of the image 2200 of being caught, " has " and " begun "); 3-5-3 (being the quantity of the character in each of the word " has " flatly arranged in second row of the image 2200 of being caught, " begun " and " The "); 3-3-6 (being the quantity of the character in each of the word " 461 " flatly arranged in the third line of the image 2200 of being caught, " and " and " permit "); And 3-6-8 (being the quantity of the character in each of the word " and " flatly arranged in the third line of the image 2200 of being caught, " permit " and " projects ").
Vertical trigram specify on the given word with under the quantity of character in each word of word of homeotropic alignment.For example, for the image of being caught 2200, (c) vertical trigram: 5-7-3 (for the quantity of the character in each of the word " upper " vertically arranged, " Project " and " 461 ") is shown; 8-7-3 (being the quantity of the character in each of the word " division " vertically arranged, " Project " and " 461 "); 8-3-3 (being the quantity of the character in each of the word " division " vertically arranged, " has " and " and "); 8-3-6 (being the quantity of the character in each of the word " division " vertically arranged, " has " and " permit "); 8-5-6 (being the quantity of the character in each of the word " division " vertically arranged, " begun " and " permit "); 8-5-8 (being the quantity of the character in each of the word " division " vertically arranged, " begun " and " projects "); 7-5-6 (being the quantity of the character in each of the word " courses " vertically arranged, " begun " and " permit "); 7-5-8 (being the quantity of the character in each of the word " courses " vertically arranged, " begun " and " projects "); 7-3-8 (being the quantity of the character in each of the word " courses " vertically arranged, " The " and " projects "); 7-3-7 (being the quantity of the character in each of the word " Project " vertically arranged, " 461 " and " student "); And 3-3-7 (being the quantity of the character in each of the word " has " vertically arranged, " and " and " student ").
Based on determined level and vertical trigram from the image of being caught 2200 shown in Figure 22, produce each (d) and the lists of documents (e) of document that indication comprises level and vertical trigram.For example, in (d), horizontal trigram 7-3-5 appears in document 15,22 and 134.In addition, for example, in (e), vertical trigram 7-5-6 appears in document 15 and 17.Use (d) and lists of documents (e), the graduate tabulation of all relevant documents is shown respectively in (f) and (g).For example, in (f), five horizontal trigrams in (d) all relate to document 15, and (d) only a horizontal trigram relate to document 9.In addition, for example, in (g), 11 vertical trigrams in (e) all relate to document 15, and (e) only a vertical trigram relate to document 18.
Equally with reference to Figure 23, it illustrates the technology that is used for level of describing with reference to Figure 22 and the combination of vertical trigram information now.Use is about the information of the known physical position of the trigram on the original printer page, and the tabulation of the voting that this technology will be extracted from level and vertical features combines.For each M that exports each document that has among selecting by level and vertical classification device, compare with each vertical trigram of deciding by vote about that document in the position of each the horizontal trigram that will decide by vote about document.Document receives the many votings equal with the quantity of the horizontal trigram of any vertical trigram crossover, here when the bounding box crossover of two trigrams, and " crossover " appearance.In addition, with hereinafter with reference Figure 34 A 3406 and the version after suitable change of the evidence accumulation algorithm described calculates the x-y position at the center of crossover part.For example, as shown in Figure 23, (a) with (b) in tabulation (be respectively among Figure 22 (f) with (g)) intersect the page listings (c) of all being correlated with definite level and vertical trigram.Use cross tabulating (c), tabulation (d) and (e) (only illustrate by the trigram of being discerned relevant intersection document) and document printing database 3400, confirm the crossover part of document.For example, horizontal trigram 3-5-3 relates to document 6 with vertical trigram 8-3-6, and in the image of being caught 2200, this crossover on word " has " of those two trigrams; Thereby document 6 receives a voting about this crossover part.Shown in (f), for the special image of being caught 2200, document 15 receives the voting of maximum quantity, and thereby is identified as the document that comprises the image 2200 of being caught.(x1 is y1) as the position of the input picture in the document 15 in identification.Thereby; Review is with reference to Figure 22 and 23 described document fingerprint matching technology; The horizontal classification device uses the characteristic that derives from the horizontal arrangement of the word of text; And the vertical classification device uses the characteristic that derives from the vertical layout of those words, here based on the crossover part of those characteristics in the original document, and the result combined.Such feature extraction is provided for discerning uniquely the mechanism of document, because when the horizontal aspect of this feature extraction was limited by suitable grammer and language constraint, vertical aspect was not limited by such constraint.
In addition, although be particularly suitable for the use of trigram,, can use any n-gram for level and vertical features extraction/classification one or both with reference to the description of Figure 22 and 23.For example, in one or more embodiments,, can use vertical and horizontal n-gram, here n=4 for the multi-categorizer feature extraction.In one or more other embodiment, the horizontal classification device can extract characteristic based on n-gram, n=3 here, and the vertical classification device can extract characteristic based on n-gram, n=5 here.
In addition, in one or more embodiments, it not is strictly horizontal or vertical syntople that classification can be based on.For example, NW, SW, NW and SE syntople can be used for extraction/classification.
Figure 24 illustrates another document fingerprint matching technology according to an embodiment of the invention." feedback of database-driven " technology shown in Figure 24 consider through utilize can with the image of input document matching; With the subsequent step of the definite graphical analysis that will mate from the subimage and the input picture of original document therein, can improve the accuracy of file and picture matching system.This technology comprises the conversion of the noise that duplicates in the input picture to be appeared.After this masterplate The matching analysis can be arranged.
Figure 25 illustrates the flow process of the feedback of database-driven according to an embodiment of the invention.Described like preceding text; In step 2510,2512; At first pre-service and discern the input picture fragment (for example, use word OCR and word to search, character OCR and character to search, the configuration of word boundary frame), with many candidates of the identification of generation image fragment 2522.List (doci, pagei, xi, yi) under each candidate in this tabulation can comprise, doci is the identifier of document here, and pagei is the page in the document, and (xi yi) is the x-y coordinate at the center of the image fragment in that page.
Use from the range information of the page size criteriaization to optional fixed size at the original fragment searching algorithm of step 2514 whole input picture fragments, to guarantee to be converted into known spatial resolution, for example, 100dpi.The described font size algorithm for estimating of preceding text can be adapted to this task.Similarly, can use known from focus distance or from the degree of depth technology of focus.Equally, can be based on their height of word boundary frame, size criteriaization is the zoomed image fragment pari passu.
With the identifier of each document that receives about it and the page and MMR database together, original fragment searching algorithm queries MMR database 3400 with the center of the bounding box of the fragment that produces.The scope of the fragment that is produced depends on the size of standardized input fragment.By this way, can obtain the fragment of same spatial resolution and dimension.For example, when being normalized to 100dpi, the input fragment extends 50 pixels on each limit of the heart therein.In this situation, order MMR database generation center is placed the x-y value of appointment, the original fragment of the 100dpi of 100 pixel height and width.
Each the original image fragment that returns from MMR database 2524 can with under list (doci, pagei, xi, yi; Widthi, heihhti actioni) is associated; Here (doci, pagei, xi; Yi) be described like preceding text, widthi and heighti are the width and the height of the original fragment that calculates with pixel, and the optional action of actioni for being associated with the respective regions in the clauses and subclauses of doci in the database.Original fragment searching algorithm is exported this tabulation of 2518 image fragments and data 2518, and exports the output fragment of the size criteriaization of its structure together.
In addition, in one or more embodiments, fragments matching algorithm 2516 is compared the input fragment of size criteriaization with each original fragment, and assigns the scoring 2520 of measuring them and how to mate each other.Those skilled in the art will recognize that owing to the comparable size mechanism that is used to guarantee fragment, under many situations, just enough with the simple crossing dependency of Hamming distance.In addition, this process possibly comprise the introduction of the noise in the original fragment that imitates the picture noise that is detected in the input.More also possibly be complicated arbitrarily; And possibly comprise the comparison of any characteristic group; This characteristic group comprise two fragments OCR result and based on character, character to or the ranking compositor of the right quantity of word, wherein word is to being limited by like former geometric relationship.Yet in this situation, the total right quantity of geometry can be estimated as or be used as the ranking compositor module between input fragment and the original fragment.
In addition, output 2520 can be with the n-tuple (doci, pagei, xi, yi, actioni, form scorei), marking is here provided by the fragments matching algorithm, and tolerance input fragment and doci, the degree that the respective regions of pagei is complementary.
Figure 26 illustrates another document fingerprint matching technology according to an embodiment of the invention." sorter of database-driven " technology shown in Figure 26 is used initial classification, and generation possibly comprise one group of hypothesis of input picture.In database 3400, search those hypothesis, and automatically add classification policy for those hypothesis design features extract.An example is that identification input fragment is perhaps to comprise the Times font, perhaps comprises the Arial font.In this situation, control structure 714 is called serif/sans serif and is distinguished special-purpose feature extractor and sorter.
Figure 27 illustrates the flow process of the classification of database-driven according to an embodiment of the invention.And then first feature extraction 2710, by the described any or identification methods of preceding text with input picture fragment classification 2712, to produce document, the ranking compositor of the x-y position in the page and those pages.Each candidate in this tabulation can comprise, for example, list down (doci, pagei, xi, yi), doci is the identifier of document here, pagei is the page in the document, and (xi yi) is the x-y coordinate at the center of the image fragment in that page.The fragmentation pattern picture that can use the original fragment searching algorithm described with reference to Figure 25 2714 to produce about each candidate.
Still with reference to Figure 27, second feature extraction is applied to original fragment 2716.This possibly be different from first feature extraction, and possibly comprise, for example, and one or more font probe algorithms, character recognition technologies, bounding box and SIFT characteristic.The characteristic that is detected in each original fragment is inputed to automatic categorizer method for designing 2720, and this method comprises, for example, is the neural network that designs, SVMs and/or a nearest neighbor classifier of original fragment for the sample classification with the unknown.Can the second identical feature extraction be applied to 2718 input picture fragments, and what the characteristic that it detects was inputed to that this designs recently possibly be the special-purpose sorter of original fragment.
Output 2724 possibly be the n-tuple (doci, pagei, xi, yi, actioni, form scorei), mark here by 2720 automatically the sorting techniques 2722 of design provide.One of skill in the art will appreciate that scoring tolerance input fragment and doci, the degree that the respective regions of pagei is complementary.
Figure 28 illustrates another document fingerprint matching technology according to an embodiment of the invention." multi-categorizer of database-driven " shown in Figure 28 technology is through spreading all over a plurality of candidates of decision process, and reduces the chance of irrecoverable property mistake early stage in the identifying.Carry out several preliminary classification.Each all produces, and different character is extracted and the different brackets ordering of the input fragment that classification can be distinguished.For example, one in those groups possibly produced by horizontal n-grams, and the identification uniquely by distinguishing serif from sans serif.Another example possibly produced by Vertical n-grams, and the identification uniquely by the accurate Calculation of row separation.
Figure 29 illustrates the polytypic flow process of database-driven according to an embodiment of the invention.Shown in this flow process and Figure 27 that is similar, but it uses a plurality of different character extraction algorithms 2910 and 2912, to produce the independently ranking compositor of input picture fragment with sorter 2914 and 2916.The example of characteristic and sorting technique comprises described level of preceding text and vertical word length n-grams.Each sorter can produce following lising (doci, pagei, the xi that comprises at least about each candidate; Yi, the graduate tabulation of fragment identification scorei), doci is the identifier of document here; Pagei is the page in the document; (xi yi) is the x-y coordinate at the center of the image fragment in that page, and scorei tolerance is imported the degree that the relevant position in fragment and the database document is complementary.
The original fragment searching algorithm that can use preceding text to describe with reference to Figure 25 produces one group of original image fragment corresponding to the clauses and subclauses in the tabulation of the identification of the fragment in 2914 and 2916 the output.Can be as former third and fourth feature extraction 2918 and 2920 be applied to original fragment and the as above such automatic design described in the texts and pictures 27 and the sorter of application.
Still with reference to Figure 29, the ranking compositor that those sorters produced is combined, have clauses and subclauses (doci, pagei with generation; Xi, yi, actioni, single ranking compositor 2924 scorei); Here i=1 ..., the quantity of candidate, and the value in each clauses and subclauses is all described like preceding text.For example, can carry out ranking compositor and combine 2922 through measuring based on the known Borda counting method of its common location and scoring of assignment project in two ranking compositors.This can combine with the scoring of being assigned by independent sorter, to produce synthetic scoring.In addition, those skilled in the art will notice the method that can use other ranking compositor to combine.
Figure 30 illustrates another document fingerprint matching technology according to an embodiment of the invention." video sequence image adds up " shown in Figure 30 technology is through will be near the data integration of or adjacent frame, and the design of graphics picture.An example relates to " super-resolution ".N interim adjacent frame of its record, and use the knowledge execution of the some expanded function of lens to be essentially the operation that the sub-pixel edge strengthens.Effect is the spatial resolution that increases image.In addition, in one or more embodiments, can make the super-resolution method specialization, to stress the text special characteristic as hole, corner and point.Further expansion will be used the characteristic of candidate image fragment, like what confirm from database 3400, so that the super-resolution integrated functionality is specialized.
Figure 31 illustrates another document fingerprint matching technology according to an embodiment of the invention." video sequence characteristics adds up " technology shown in Figure 31 is before making a decision, and characteristic adds up on many interim adjacent frames.This utilizes the high sampling rate (for example, per second 30 frames) of acquisition equipment and user's intention, and it keeps acquisition equipment to point to several at least seconds of identical point on the document.On every frame, carry out feature extraction independently, and the result is combined, to produce single unified characteristic pattern.Cohesive process comprises the registration hiding step.In the inspection of the video clipping of text fragment, be quite obvious for the needs of this technology.Automatic focusing and contrast adjustment in typical capture device can produce visibly different result in adjacent frame of video.
Figure 32 illustrates another document fingerprint matching technology according to an embodiment of the invention." video sequence decision combine " technology shown in Figure 32 will combine from the decision of many interim adjacent frames.This utilizes the high sampling rate of typical acquisition equipment and user's intention, and it keeps acquisition equipment to point to several at least seconds of identical point on the document.Handle every frame independently, and produce the graduate row decision of itself.Those decisions are combined, to produce the single unified ranking compositor of input picture group.This technology comprises the registration hiding method of control decision cohesive process.
In one or more embodiments; Preceding text can be used for combining with one or more known matching techniques with reference to figure 6 to 32 described one or more various document fingerprint matching technology, and such combination is referred to herein as " multilayer level (or multifactor) identification ".Usually, in the identification of multilayer level, use first matching technique in document database, to locate one group of page, and discern fragment uniquely among using the page of second matching technique from this group then with specific criteria.
Figure 33 illustrates the example of the flow process of multilayer level identification according to an embodiment of the invention.At first, in step 3310, use acquisition equipment 106 on interested documents, to catch/scan " select " characteristic.This characteristic of selecting can be any characteristic, and it catches the selection that causes one group of document in the document database effectively.For example; The characteristic of selecting can be only for the numeral bar code (for example; Univeraal product code (UPC)), alphanumeric bar code (for example, code 39, code 93, code 128) or two-dimensional bar (for example, QR sign indicating number, PDF 417, Datamatrix, Maxicode).In addition, the characteristic of selecting can be, for example, and figure, image, trade mark, sign, special color or combination, key word or the phrase of color.In addition, in one or more embodiments, the characteristic of selecting can be confined to be suitable for the characteristic of acquisition equipment 106 identifications.
In step 3312, in case caught the characteristic of selecting in step 3310, related based on the characteristic of selecting of being caught selected in the document database one group of document and/or document file page.For example, if the sign of the company that is characterized as that selects that is caught is then selected to index in the database for comprising all documents of that sign.In another example, database can comprise the image of selecting of being caught and its trade mark storehouse relatively.When " hitting " arranged in this storehouse, all documents that selection is associated with the trade mark that is hit were to be used for like the described coupling subsequently of hereinafter.In addition, in one or more embodiments, can depend on the position of that characteristic of selecting on the characteristic of selecting of being caught and the document that is scanned in the selection of the document/page of step 3312.For example, the information that is associated with the characteristic of selecting of being caught can specify the image of selecting whether to be positioned at the upper right corner of document, rather than the lower left corner of opposed document.
In addition, those skilled in the art will notice, can make special the confirming of image that image comprises the characteristic of selecting of catching by acquisition equipment 106 or some other parts that receive original view data from acquisition equipment 106.For example, database self can be confirmed to comprise from the specific capture object that acquisition equipment 106 sends and selects characteristic, as its associated databases is selected and one group of document selecting feature association of catching.
In step 3314, after step 3312 had been selected the particular group document, acquisition equipment 106 continued scanning and catches the image of interested documents thus.Then, through using one or more with reference in the described different document fingerprint matching technology of figure 6-32, the image of this document of catching and document in step 3312 selection are mated.For example; Step 3310 will be indexed as the one group of document selecting characteristic that comprises the footwear figure based on catching of the footwear graph image on the interested documents after, can use the document coupling of catching image and a described group selection of foregoing a plurality of classifier technique with subsequently interested documents.
Thus, use the realization of handling with reference to the multilayer identification stream of the description of Figure 33, the quantity through the initial reduction page/document can reduce the fragment identification frequency, wherein with the said page/document and subsequently the images match of catching.In addition, through at first scanning the locational document of the characteristic of selecting that has image, bar code, figure or other type, the user can utilize so improved identification frequency.Through carrying out such action, the user can reduce the number with the document of subsequently the images match of catching apace.
The MMR Database Systems
Figure 34 A illustrates according to one embodiment of present invention and the functional block diagram of the MMR Database Systems 3400 that dispose.System 3400 is for content-based retrieval disposes, here so that can carry out the two-dimensional geometry relation between the mode indicated object of searching of text based index (or any index that other can be searched for).System 3400 adopts evidences to add up, and passing through, for example, the frequency that characteristic is taken place combines with the possibility of its position in the 2 dimensional region, and the raising search efficiency.In a particular embodiment; Database Systems 3400 are the detailed realization of document event database 320 (comprising PD index 322), and its content comprises the electronic representation of the document printing that is produced by trapping module 318 and/or the document fingerprint matching module of being discussed with reference to figure 3 like preceding text 226.According to this open invention, other application program of system 3400 will be clearly with configuration.
As can see, Database Systems 3400 comprise MMR concordance list module 3404, evidence accumulator module 3406 and the relational database 3408 (or any other suitable storage facility) of reception by the description of MMR characteristic extracting module 3402 calculating.The concordance list of the x-y position in document, the page and those pages that concordance list module 3404 each characteristic of inquiry identification take place.Can pass through, for example, MMR concordance list module 3404 or some other special-purpose modules produce concordance list.Evidence accumulator module 3406 programs are turned to or are configured to, given data from concordance list module 3404, and calculate graduate one group of document, the page and hypothesis on location 3410.Relational database 3408 can be used for storing the additional features 3412 of relevant each fragment.These comprise 504 among Fig. 5 and 508, but are not limited thereto.Through in deriving, using the two-dimensional arrangement of the text in the fragment, can quite big degree ground increase even the uniqueness of the fragment of very little text about the signature of fragment or fingerprint (that is, unique search terms).Other embodiment can utilize any two-dimensional arrangement of the object/characteristic in the fragment similarly in deriving about the signature of fragment and fingerprint, and about discerning fragment uniquely, embodiments of the invention are not intended to be limited to the two-dimensional arrangement of text.Other parts of the Database Systems of being illustrated among Figure 34 A 3400 and function comprise that signature search module 3418, the document of feedback guiding present application program module 3414 and subimage extraction module 3416.These parts and other system 3400 parts are mutual, with signature search and the dynamically original image generation that the feedback guiding is provided.In addition, system 3400 comprises the action processor 3413 that receives action.Action and its output that provides that action specified data storehouse system 3400 carries out.With explaining each of these other parts successively.
Utilize the example of MMR characteristic extracting module 3402 of the two-dimensional arrangement of the text in this fragment shown in Figure 34 B.In such embodiment, MMR characteristic extracting module 3402 programs are turned to or be configured to adopt based on the technology of OCR from the image fragment, extract characteristic (text or other target signature).In this specific embodiment, characteristic extracting module 3402 is extracted the x-y position of the word in the image of fragment of texts, and those location tables are shown its level that comprises or vertical adjacent word to group.If it is adjacent that they are levels, then convert the image fragment into connect word effectively to (for example, the-cat, in-the by "-"; The-hat, and is-back), and if their crossovers vertically then (are for example connected by "+"; The+in, cat+the, in+is, and the+back).This x-y position can be, for example, and based on some point of fixity in file and picture (from the upper left corner or the center of document), the pixel of in x and y in-plane, calculating.The level in this example noticed adjacent to can appearing at continually in many other text fragments, and vertical crossover to maybe be more rare in other text fragment.Can be similarly with other geometric relationship coding between the characteristics of image, the SW-NE that for example has "/" between the word in abutting connection with, have " " the NW-SE adjacency, or the like.Equally, can " characteristic " be generalized to the word boundary frame (perhaps further feature bounding box) of string encoding that can be arbitrarily enough but consistent.For example, can enough strings " 4rusl " expression have coarse last outline line but smooth lower whorl profile, with four times of high the same long bounding boxes.In addition, can geometric relationship be generalized to the arbitrarily angled and distance between the characteristic.For example, can use " 4rusl " expression NW-SE adjacent but by two words that " 4rusl " describes that have of the high separation of two words.According to this open invention, many encoding schemes will be clearly.In addition, notice and to use numeral, Boolean, geometric configuration and other such file characteristics, replace word right, discern fragment.
Figure 34 C illustrates example index table tissue according to an embodiment of the invention.As can see, the MMR concordance list comprises is inverted entry index table 3422 and document index table 3424.As will discuss successively; Each unique item or characteristic (for example, key point 3421) is all pointed to the position in the entry index table 3422, and this entry index table 3422 (for example keeps sensing one row record 3423; Rec# 1, Rec# 2; Or the like) the functional value (for example, key point x) of characteristic, and the candidate region on the page in the document all discerned in each record.In an example, the functional value (key point x) of key point and key point is identical.In another example, hash function is applied to key point, and this function is output as key point x.
A given row query term, each record that inspection is indexed through key point, and the identification zone the most consistent with all query terms.If this zone comprises sufficiently high coupling scoring (for example, based on predetermined matching threshold), then confirm hypothesis.Otherwise, announce the coupling failure, not return area.In this exemplary embodiment, as described earlier, key point be or the word that separates by "-" or by "+" to (as, " the-cat " or " cat+the ").This technology itself that geometric relationship is integrated with in the key point allows the use about the traditional text search technique of two-dimensional geometry inquiry.
Thereby the concordance list tissue becomes the Feature Conversion that is detected in the image fragment text items of representing characteristic itself and the geometric relationship between them.This allows the utilization of demarcation of traditional text index and searching method.For example, as will be clearly, by the vertical adjacent item " cat " and " the " of the symbol that can be called as " query term " " cat+the " expression according to this open invention.MMR technology described herein that the utilization of traditional text search data structure and method is convenient moving on internet text search system (for example, Google, Yahoo, Microsoft, or the like) connects.
In the inversion entry index table 3422 of this exemplary embodiment; Each record all uses six parameters: the width of document recognition (DocID), page number (PG), x/y side-play amount (being respectively X and Y) and rectangular area and height (being respectively W and H), the candidate region in the identification document on the page.DocID is for when document printing, based on time mark (or other metadata) and unique string of generation.But it can be any string of coupling apparatus ID and personnel ID.In any situation, document is all discerned by unique DocIDs, and has the record that is stored in the document index table.Page number is the page-number marker corresponding to paper output, and since 1.By the X-Y coordinate in the upper left corner, and the width of the bounding box in the standardized coordinate system with highly be the parametric representation matrix area.According to this open invention, many document interior location/coordinate schemes will be clearly, but the present invention is not intended to be limited to any special one.
According to one embodiment of present invention and the exemplary record structure of configuration is used 24-position DocID and 8-position page number, permission is up to 16,000,000 documents and 4,000,000,000 pages.About a no symbol-byte of each X of bounding box and Y side-play amount all provide the 30dpi level the spatial resolution vertical with 23dpi (suppose 8.5 " * 11 " the page, although can use other page size and/or spatial resolution).About the similar disposal of the width of bounding box and height (for example; A no symbol-byte about each W and H) allows that perhaps the point on " i " is the same little with fullstop; Perhaps with whole pages (for example, 8.5 " * 11 " or other) expression in equally big zone.Therefore, eight of each record bytes (1 byte of 1 byte of 3 bytes of DocID, 1 byte of PG, X, 1 byte of Y, W and 1 byte of H for 8 bytes) altogether can comprise a large amount of zones.
Document index table 3424 comprises the relevant information of relevant each document.In a particular embodiment, this information comprises the relevant field of document in the XML file, comprises print resolution, date printed, paper size, shadow file name, page-images position, or the like.Because when indexing to document, will print coordinate conversion becomes standardized coordinate system, calculates the search hypothesis and does not relate to this table.Thereby, only consult document index table 3424 about the candidate region that is complementary.Yet some losses of information in this decision hint index are because standardized coordinate is in the resolution lower than print resolution usually.If need in this way, when calculating the search hypothesis, alternative embodiment can use document index table 3424 (the perhaps higher resolution of standardized coordinate).
Thereby, concordance list module 3404 running, the image index that can carry out with the content-based retrieval that the x-y position that makes that object (for example, document file page) and those objects of given image querying nidus are interior is provided effectively.The characteristic that the combination of such image index and relational database 3408 allows to make image fragment and fragment (for example; Be attached to " action " of fragment, perhaps can scan bar code with the retrieval that impels other content relevant with fragment) position of matched object.Relational database 3408 also provides the method for " the opposite link " of the characteristic of other fragment in concordance list from fragment a to document.Opposite link provides to be found when its part from file and picture moves to another part, the mode of the characteristic that recognizer is seen expectation, and it can quite big degree ground improves as the performance of front end image analysis algorithm in the MMR system that this discusses.
The signature search of feedback guiding
Can the x-y coordinate (for example, the x-y coordinate at the center of image fragment) of image fragment and the identification of the document and the page be inputed to the signature search module 3418 that feedback leads equally.Signature search module 3418 search of feedback guiding are from the entry index table 3422 of giving the record 3423 that takes place in the set a distance at the center of image fragment.For example, can be through will being stored in about the record 3423 that each DocID-PG combines in the storage adjacent block with the series classification of X and Y value, and convenient this search.Through about the binary search of set-point (depending on when storage data X or the Y how to classify) with have given X certainly and the serial search of that position of all records of Y value, and execution is searched.Typically, this will comprise the x-y coordinate in the M inch ring of periphery of the wide and fragment that the H inch is high of in the given document of tolerance and page W inch.Locate the record that takes place in this ring, and through antitracking pointer location their key point or characteristic 3421.As Figure 34 A 3417 shown in, the tabulation of characteristic and their x-y position in the report ring.Can be based on the size of input picture, dynamically be arranged on the value of the W shown in 3415, H and M through recognition system, so that characteristic 3417 is in the outside of input picture fragment.
For example, for the ambiguity of eliminating a plurality of hypothesis, such characteristic of image database system 3400 is of great use.If the more than document of Database Systems 3400 reports possibly be complementary with the input picture fragment; The characteristic that then centers in the ring of fragment (for example will allow recognition system; Fingerprint matching module 226 or other recognition system that is fit to) move image capture apparatus through the guiding user slightly in the direction of the ambiguity that can eliminate decision, and the document that determines which document and user to hold matees most.For example (suppose to use the characteristic based on OCR, although this notion can extend to the characteristic group of indexing on any geometry), the image fragment among the document A possibly be positioned at word directly under " blue-xylophone ".Image fragment among the document B possibly be positioned at word directly under " blue-thunderbird ".Database Systems 3400 will be reported the position of the expectation of these characteristics, and recognition system possibly order user's (for example, through user interface) that camera is moved up by the indicated amount of difference at the top of the y coordinate of characteristic and fragment.Recognition system can be calculated the characteristic in that difference zone, and uses and confirm from the characteristic of document A and document B which matees most.For example, recognition system can be enough by (xylophone, " dictionary " aftertreatment of the characteristic of thunderbird) forming is from the OCR result in difference zone.The word that matees most with OCR result is corresponding to the document that matees most with input picture.The example of post-processing algorithm comprises usually known spelling correction technology (for example word processor and email application employed those).
Illustrate like this example, the design of Database Systems 3400 allows recognition system, describes through the mode matching characteristic with the needs of avoiding carrying out further database access, eliminates the ambiguity of a plurality of candidates with effective and efficient manner.Alternative solution will be for handling each image independently.
Dynamically original image generates
Equally can be (for example with the x-y coordinate of the position in the image fragment; The x-y coordinate at the center of image fragment) and the identification of the document and the page input to relational database 3408, can use them to retrieve the electronics original text of being stored of that document and the page therein.Then, can present application program module 3414 through document presents that document and becomes bitmap images.Equally, subimage extraction module 3416 uses other " square frame size " value that is provided by module 3414 to extract around the part of the bitmap at center.This bitmap is " original " expression of the desired outward appearance of image fragment, and it comprises the accurate expression of all characteristics that should appear in the input picture.Can return original fragment then as fragment characteristic 3412.This solution has overcome the desired excessive storage of prior art, and this prior art can be converted into the non-image expression of compression of data bitmap subsequently through storage when requiring, and the memory image bitmap.
Such storage scheme is useful because its make it possible to suppose-with the use of-check recognition strategy, the character representation retrieval of using from image therein to be extracted is through one group of candidate after the detailed signature analysis disambiguation.Usually, prediction will eliminate best arbitrarily that the characteristic of one group of candidate is impossible, but confirm that from the original image of those candidates this point is very desirable.For example, can in two data database documents, locate the image of word to " the cat ", one of them is printed with Times Roman font at first, and another is printed with the Helvetica font.Confirm simply whether input picture comprises the database document that will discern correct match of these fonts.Use template matches comparison measuring standard, the original fragment of those documents is compared with the input picture fragment, the candidate that identification is correct as the Euclidean distance.
Example comprises that (similarly method is suitable for other document format as the XML paper specificationXPS of postscript, PCL, pdf. or Microsoft for the relational database 3408 of store M icrosoft Word " .doc " file; Perhaps through the application program that appears as ghostscript; Or at XPS, have in the situation of Internet Explorer of Microsoft of the WinFX parts of being installed, can be converted into other such form of bitmap).Supposing document, the page, x-y position, square frame dimension and indicating preferred resolution is the identification of the systematic parameter of 600 dpi (dots per inch)s (dpi), then can call the Word application program, to produce bitmap images.This will provide the bitmap of 6600 row and 5100 row.Other parameter x=3 ", y=3 ", height=1 " and width=1 " referred database should return the center and place the fragment away from 600 pixel height and width of the point of the upper left corner x of the page and last 1800 pixels of y.
A plurality of databases
When using a plurality of Database Systems 3400, its each can comprise different document sets, can use original fragment to confirm whether two databases return identical document or which database and returned and import matched candidate person preferably.
If two databases return identical document, perhaps have different identifier 3410 (that is, and original document be identical be unconspicuous because their inputs respectively in different databases) and characteristic 3412, then original fragment will be almost completely identical.This can pass through, and for example, with the Hamming distance of the quantity of calculating pixels with different, original fragment is compared to each other and confirms.If it is identical that original document is a pixel to pixel, then Hamming distance will be zero.If fragment is difference a little, like what possibly caused by small font difference, then Hamming distance will be a little greater than zero.When the image difference in the calculating Hamming operator, this can cause " ring of light " effect around character edge.Different editions original presents operating system, different printer driver or the different font sets of different editions on the server of application program, runtime database, can both cause such a font difference.
Can on from the fragment of the more than x-y position in two documents, carry out original fragment comparison algorithm.What they were all should be identical, but such a sampling process will allow to overcome the redundancy that presents difference between the Database Systems.For example, when in two systems, being current, it is different up hill and dale that a kind of font possibly seem, but another kind of font maybe be identical.
If two or more databases return different documents as its optimum matching about input picture, then can original fragment be compared with input picture, to confirm that which is correct through comparison measuring standard based on pixel as the Hamming distance.
Being used for comparison is the content of the array that adds up of geometric distributions that compares and measures the characteristic of the document that each database reports from the result's of a more than database alternative strategy.Directly by database this totalizer being provided, with the needs of searching of the primitive character group of avoiding carrying out separation, is very desirable.Equally, this totalizer should be independent of the content of Database Systems 3400.In the embodiment shown in Figure 34 A, derivation activity array 3420.Can distribute through the inside of their value of measurement, relatively two movable arrays.
In more detail; If two or more databases return identical document; Perhaps has different identifier 3410 (promptly; Original document be identical be unconspicuous because their inputs respectively in different databases) and characteristic 3412, will be almost completely identical then from the movable array 3420 of each database.This can pass through, and for example, with the Hamming distance of the quantity of calculating pixels with different, array is compared to each other and confirms.If original document is identical, then Hamming distance will be zero.
If two or more databases return different documents as its optimum matching about input feature vector, then can compare their movable array 3420, to confirm which document and input picture " best " coupling.The movable array that correctly matees with the image fragment will comprise the crowd family that the center is similar to the high numerical value of the position that places fragment appearance place.The movable array that matees inadequately with the image fragment will comprise the numerical value of stochastic distribution.There are many strategies of knowing of randomness that are used to measure chromatic dispersion or image, for example entropy.Can be with such algorithm application in movable array 3420, to obtain the measurement that exists of indication group variety.For example, comprise the entropy that entropy corresponding to the movable array 3420 of the group variety of image fragment will considerably be different from the movable array 3420 that its numerical value distributes randomly.
In addition, notice a plurality of databases 3400 that independent client computer 106 maybe all addressable at any time its content must not conflicted each other.For example, enterprise possibly have each and all relates to the privately owned fragment of addressable fragment of disclosing of single document and enterprise.In such situation, client apparatus 106 will be kept a column data storehouse D1, D2, the D3 that consults in order ..., and will generate unified user's demonstration with identifier 3410 through the movable array 3420 that combines.Given client apparatus 106 possibly show from the available fragment of all databases, perhaps allows user-selected number according to storehouse subclass (for example, only D1, D3 and D7), and only shows the fragment from those databases.Can database be added into tabulation through subscribed services; Perhaps when client apparatus 106 is in certain position; Make database wirelessly to obtain; Perhaps because database is several one that has been loaded on the client apparatus 106, perhaps because current this device that using of verified certain user, perhaps even because this device just with certain pattern operation.For example, because the audio tweeter of special client apparatus opens or cuts out, perhaps because of the current client computer that is attached to of peripheral unit as the video frequency projector, some database just possibly be available.
Action
Further with reference to figure 34A, MMR database 3400 receives action and from a stack features of MMR characteristic extracting module 3402.Action specified command and parameter.In such embodiment, order and the definite fragment characteristic of being returned 3412 of its parameter.Can easily being become comprising of text by translation, for example, http, form receive action.
In such embodiment, order comprises: RETRIEVE, INSERT_TO < DATA >, RETRIEVE_TEXT < RADIUS >, TRANSFER < AMOUNT >, PURCHASE, PRISTINE_PATCH < RADIUS [DOCID PAGEID X Y DPI]>and ACCESS_DATABASE < DBID >.Now each will be discussed successively.
The RETRIEVE-retrieval is connected to the data of the x-y position in the given document file page.Action processor 3413 becomes the relation data library inquiry that retrieval possibly be stored near the data this x-y position with the RETRIEVE command conversion.This can require the issue of a more than data library inquiry, with the zone of search packet round the x-y position.Data retrieved is exported as fragment characteristic 3412.The exemplary application of RETRIEVE order is the multimedia viewer applications of retrieve video montage or multidate information object (for example, can retrieve the electronic address of current information).Data retrieved can comprise the menu of the step subsequently that appointment will be carried out on the MMR device.It also possibly be to go up the static data that shows, for example jpeg image or video clipping at phone (or other display device).Can parameter be offered the RETRIEVE order, it confirms the zone of search fragment characteristic.
INSERT_TO < DATA >-insert < DATA>in the x-y position of image fragment appointment.Action processor 3413 becomes the instruction about the x-y position that data is added into appointment of relational database with the INSERT_TO command conversion.Completing successfully of INSERT_TO order taken as for fragment characteristic 3412 really return.The exemplary application of INSERT_TO order is attached to data the software application on the MMR device of any x-y position in the paragraph of text for allowing the user.Data can be static multi-medium datas, and like jpeg image, video clipping or audio file, but it also can be appointment and the electronic data arbitrarily of given position associated action as the menu.
RETRIEVE_TEXT < RADIUS >-retrieval is by the text in < RADIUS>of the determined x-y of image fragment position.Can < RADIUS>be appointed as, for example, the many pixels in the image space perhaps can be appointed as the character that centers on by many words of evidence accumulator module 3406 determined x-y positions with it.< RADIUS>also can relate to the text object through analyzing.In this specific embodiment, action processor 3413 becomes the RETRIEVE_TEXT command conversion relation data library inquiry of the suitable text of retrieval.If < RADIUS>specifies through the text object of analyzing, then action processor only returns the text object through analyzing.If the text object through analyzing is not positioned near the x-y position of appointment, then action processor returns zero indication.In alternative embodiment, action processor calls the signature search module of feedback guiding, to retrieve the text that in the radius of given x-y position, occurs.Text string is returned as fragment characteristic 3412.The optional data that are associated with each word in the text string comprise x-y bounding box in the original document.The exemplary application of RETRIEVE_TEXT order is for to select text phrases, so that be included among another document from document printing.This possibly be used for, for example, and synthetic presentation document (for example, with the PowerPoint form) in the MMR system.
TRANSFER < AMOUNT >-whole documents of retrieval and can be loaded on some data that form on another database is connected to it.< AMOUNT>specifies the quantity and the type of institute's data retrieved.If < AMOUNT>is ALL, then action processor 3413 issue an orders are to database 3408, and it retrieves all data that are associated with document.The example of such order comprises DUMP or Unix TAR.If < AMOUNT>is SOURCE, the original source file of search file then.For example, this will retrieve the Word file of document printing.If < AMOUNT>is BITMAP, then retrieve the JPEG compressed version (or other common employed form) of the bitmap of document printing.If < AMOUNT>is PDF, then the PDF of search file representes.Rely on command name,, data retrieved is exported as fragment characteristic 3412 with the known form of invokes application.The exemplary application of TRANSFER order is represented the PDF of document to be passed to " document is seized device " of MMR device for allowing the user through the zonule imaging that makes text.
The PURCHASE-retrieval is connected to the description of product of the x-y position in the document.Action processor 3413 is at first carried out a series of one or more RETRIEVE orders, to obtain near the description of product the given x-y position.The description of product comprises, for example, and seller's name, the identification of product (for example, stock number) and seller's electronic address.Have precedence near other data type possibly being positioned at, and the retrieval description of product.For example, if jpeg is stored in the position by the determined x-y of image fragment, then alternatively retrieve the next immediate description of product.The description of product of retrieval is exported as fragment characteristic 3412.The exemplary application of PURCHASE order is associated with the advertisement in the document printing.Software application on the MMR device receives the description of product that is associated with advertisement; And before the seller of the appointment that sends it to specified electronic address place; Interpolation user's personally identifiable information (for example, name, Shipping Address, credit card number, or the like).
The electronic representation of the document that PRISTINE_PATCH < RADIUS [DOCID PAGEID X Y DPI] >-retrieval is specified, and extract and to have radius R ADIUS, the center places the image fragment of x-y.RADIUS can specify the radius of annular, but it also can specify rectangle fragment (for example, 2 inches high * 3 inch wide).It also can specify whole document file pages.(DocID, PG, x, y) information can provide as a part of moving expressly, and perhaps it can be derived from the image of text fragment.The original expression of action processor 3413 search file from relational database 3408.That expression can be a bitmap, but it also can be the electronic document that can appear.Original expression is passed to document presents application program 3414, it is for conversion into bitmap (having the resolution that is provided among the parameter DPI like dpi (dots per inch)), and then it is offered the subimage extraction 3416 of extracting desired fragment at it at this.The fragmentation pattern picture is returned as fragment characteristic 3412.
ACCESS_DATABASE < DBID >-database 3400 the is added into Database Lists of client computer 106.Except when preceding in tabulation outside any existing database, client computer can be consulted this database 300 now.DBID or specified file are perhaps specified the telecommunication network that relates to the data designated storehouse.
The index table generating method
Figure 35 illustrates the method 3500 that is used to produce the MMR concordance list according to an embodiment of the invention.Can, for example, implement this method through the Database Systems 3400 of Figure 34 A.In such embodiment, for example,, from scanning or document printing, produce the MMR concordance list through MMR concordance list module 3404 (or some other special-purpose modules).Can be with software, hardware (for example, gate-level logic), firmware (for example, disposing the microcontroller of the embedding routine that is used to implement this method), perhaps their some combinations are as other module described herein.
This method comprises reception 3510 paper documents.Paper document can be any document; For example have any amount of page informal letter (as; The mail that work is relevant, individual), Product labelling (as; Canned commodity, medicine, case dress electronic installation), the description of product (as; Snowblower, computer system, manufacturing system), product manual or show and colour (as; Automobile, ship, holiday resort), the service describing material (as, ISP, cleaning service), one or more pages, magazine or other such publication of book, the page of printing from the website, hand-written notes, the notes of catching and printing from blank, or from any disposal system (as, desktop PC or portable computer, camera, smart mobile phone, remote terminal) page printed.
This method continues to produce the electronic representation of 3512 paper documents, and this expression comprises the x-y position of the characteristic shown in the document.Target signature can be, for example, and the character in independent word, letter and/or the document.For example, if the scanning original document, then at first with its OCR and extract word (perhaps other target signature) and its x-y position (for example, the operation of the document fingerprint matching module 226 ' through scanner 127).If the printing original document, then the index calibration process receives the accurate expression (for example, the operation of the print driver 316 through printer 116) of XML form of font, point size and the x-y bounding box of each character (or other target signature).In this situation, concordance list generates and starts from step 3514, because receive electronic document (for example, from print driver 316) with the x-y feature locations of accurately identification.According to this open invention, the form except XML will be clearly.Through their " printings " to its output being directed to the print driver of file, consequently must not produce paper, can be with the electronic document input database as Microsoft Word, Adobe Acrobat and postscript.This triggers the generation of the XML file structure shown in the hereinafter.In all situations; XML and original document form (Word, Acrobat, postscript, or the like) all divide and send out identifier (being added into the doc i about i document of database), and to pass through that identifier; But also based on comprising the time of catching it, the date of printing, the application program that triggers printing, the title of output file; Or the like the characteristic of other " metadata " of document, the mode that makes it possible to carry out their retrievals after a while is stored in the relational database 3408.
The example of XML file structure is shown here:
$docID.xml：
<?xml?version＝“1.0”?>
<doclayout?ID＝″00001234″>
<setup>
<url>file?url/path?or?null?if?not?known</url>
<date>file?printed?date</date>
<app>application?that?triggered?print</app>
<text>$docID.txt</text>
<prfile>name?of?output?file</prfile>
<dpi>dpi?of?page?for?x，y?coordinates，eg.600</dpi>
<width>in?inch，like?8.5</width>
<height>in?inch，eg.11.0</height>
<imagescale>0.1?is?1/10th?scale?of?dpi</imagescale>
</setup>
<page?no＝″1>
<image>$docID_1.jpeg</image>
<sequence?box＝“x?y?w?h”>
<text>this?string?of?text</text>
<font>any?font?info</font>
<word?box＝″x?y?w?h″>
<text>word?text</text>
<char?box＝″x?y?w?h″>a</char>
<char?box＝″x?y?w?h″>b</char>
<char>1?entry?per?char，in?sequence</char>
</word>
</sequence>
</page>
</doclayout>
In a certain embodiments, word can comprise from any character of a-z, A-Z, 0-9 and any one of %$#; All other be separator.Can catch the original description of software (for example, on the server as database 320 servers, carrying out) establishment .xml file through the employed printing of index calibration process.Along with system obtains new document, actual format often develops, and comprises a plurality of elements.
The original series of the text that preservation print driver (for example, print driver 316) is received, and except that " _ %$# ", force logic word structure based on punctuation mark.Use the XML file as input, concordance list module 3404 is observed page boundary, and at first attempts through checking the quantity of two vertical crossovers between the continuous sequence sequence of packets to be become logical line.In a particular embodiment, if two sequence crossovers are less than their average height half the, then use row to interrupt the trial method that takes place.For typical text document (for example, the Microsoft Word document), such trial method quite works.For the html page, possibly need other geometric analysis with complex topology.Yet,, just must not extract perfect semantic file structure as long as can demarcate item as producing consistent index through query script.
Based on the structure of the electronic representation of paper document, this method continues 3514 and indexs for the position of each target signature on each page of papery document.In a particular embodiment, this step comprises to the every pair of level on each page of papery document and indexing with the position of vertical adjacent word.Explain that as previous the word that level is adjacent is that the adjacent words in the delegation is right.Vertical adjacent word is the word in the adjacent lines of vertically arranging.Can utilize other multidimensional aspect of the page similarly.
This method further comprises the fragment characteristic that storage 3516 is associated with each target signature.In a particular embodiment, the fragment characteristic comprises the action that is attached to fragment, and is stored in the relational database.Explain the position of the combination permission of such image index and storage facility and the characteristic matched object of image fragment and fragment as previous.Characteristic can be any data relevant with the path, for example metadata.Characteristic also can comprise, for example, with the action of implementing particular functionality, can be selected providing the linking and/or can or handle by scanning of the visit of other content relevant with fragment, with the bar code of the retrieval that impels other content relevant with fragment.
Generate about search terms, provide more precise definition, only observe the row structure here one section.For adjacent right of level,, form query term through connecting word with "-" separator.It is vertically right to use "+" to connect.If need in this way, can use word with its primitive form, capitalize (this creates more unique items, but same generation has the bigger index of other inquiry issue, to consider the thing as the case sensitivity) to preserve.It is right that index demarcation scheme allows that identical search strategy is applied to perhaps level or vertical word, perhaps both combinations.But the resolving ability of the reverse document frequency descriptive item of any situation.
The evidence accumulation method
Figure 36 illustrates and according to an embodiment of the inventionly is used to calculate graduate one group of document, the page and about the method 3600 of the hypothesis on location of destination document.Can, for example, implement this method through the Database Systems 3400 of Figure 34 A.In such embodiment, evidence accumulator module 3406 is used the data computation hypothesis from concordance list module 3404 as discussed previously.
This method with receive 3610 as the image fragment of bigger file and picture or all the destination document image the file and picture begin.This method continues one or more query terms that the two-dimentional relation between the object in the destination document image is caught in generation 3612.In a particular embodiment, through the generation level characteristic extraction procedure right, and produce query term with vertical word as before being discussed with reference to figure 34B.Yet, as will be clearly, can use any amount of characteristic extraction procedure as the described herein according to this open invention, produce query term, it catches the two-dimentional relation between the object in the target image.For example, can use to be used for the identical Feature Extraction Technology of index of construction method 3500, produce query term, for example refer step 3512 those (the producing electronic representations of paper document) of being discussed.In addition; Notice; The two dimension of query term shows that picture can (for example be applied to each query term individually; The level in the expression destination document and the single query item of perpendicular objects), perhaps be applied to last set item (for example, being right first query term, second query term right of horizontal word) with being vertical word.
This method continues to search each query term in the 3614 entry index tables 3422, to retrieve a column position that is associated with each query term.About each position, this method continues many zones that generation 3616 comprises the position.After handling all inquiries, this method further comprises the zone that identification 3618 is the most consistent with all query terms.In such embodiment, increase the scoring of each candidate region with weight (for example, based on each zone degree consistent) with all query terms.Whether the zone that this method continuation definite 3620 is discerned satisfies predetermined match-on criterion (for example, based on predetermined matching threshold).If like this, this method continue to confirm 3622 should the zone as the coupling of destination document image (for example, the most probable page that comprises said zone can or otherwise be used by visit).Otherwise this method continues refusal 3624 should the zone.
Word to the location storage of the coordinate space of " standardization " in entry index table 3422.This provides the consistance between different printer and the resolution of scanner.In a particular embodiment, 85 * 110 coordinate spaces are used for 8.5 " * 11 " the page.In such situation, right by its each word of location recognition in this 85 * 110 space.
In order to improve the efficient of search, can carry out two step processes.The first step comprises that the location most probable comprises the page of input picture fragment.Second step comprised calculates the interior x-y position of that page that most probable is the center of fragment.Such approach is introduced the real preferably possibility of coupling that possibly in the first step, miss.Yet, demarcate the space in sparse index, such possibility is rarely found.Thereby, depend on the size of index and desired performance, can use such efficient to develop skill.
In such embodiment, the right page of word that uses following algorithm to find most probable to comprise to be detected in the input picture fragment.
For?each?given?word-pair?wp
idf＝1/log(2+num_docs(wp))
For?each(doc，page)at?which?wp?occurred
Accum[doc，page]+＝idf；
end/*For?each(doc，page)*/
end?/*For?each?wp*/
(maxdoc，maxpage)＝max(Accum[doc，page])；
if(Accum[maxdoc，maxpage]＞thresh_page)
return(maxdoc，maxpage)；
This technology will be added into the totalizer that the page that is occurred by document and it is demarcated index above that about the right reverse document frequency (idf) of each word.Num_docs (wp) returns and comprises the quantity of word to the document of wp.Realize totalizer by evidence accumulator module 3406.If the maximal value in that totalizer surpasses threshold value, then its as be fragment optimum matching the page and export.Thereby this algorithm computing is with the page of word to mating most in identification and the inquiry.Alternately, can screen the Accum array, and conduct is reported a N page with " N best " page that the input document is complementary.
According to one embodiment of present invention, following evidence accumulation algorithm adds up about the evidence of the position of the input picture fragment in the single page.
For?each?given?word-pair?wp
idf＝1/log(2+num_docs(wp))
For?each(x，y)at?which?wp?occurred
(minx，maxx，miny，maxy)＝extent(x，y)；
maxdist＝maxdist(minx，maxx，，miny，maxy)；
For?i＝miny?to?maxy?do
For?j＝minx?to?maxx?do
norm_dist＝Norm_geometric_dist(i，j，x，y，maxdist)
Activity[i，j]+＝norm_dist；
weight＝idf*norm_dist；
Accum2[i，j]+＝weight；
end?/*for?j*/
end?/*for?I*/
end?/*For?each(y，y)*/
end?/*For?each*/
This algorithm computing is the unit in 85 * 110 spaces at center of input picture fragment with the location most probable.Among the embodiment here, through weight being added into the unit in the right FX of each word (be called ring district), this algorithm can be accomplished this point.To the given x of extent function, y is right, and its return about around fixed size zone (1.5 " high and 2 " wide be typical) minimum and maximal value.Extent function CONSIDERING BOUNDARY CONDITIONS, and guarantee its value of returning can not drop on outside the totalizer (that is, less than zero or x greater than 85 or y greater than 110).The maxdist function finds by bounding box coordinate (minx, maxx, miny, maxy) the maximum Euclidean distance between two points in the described bounding box.About each unit in the ring district, and calculate the determined weight of product by the standardized geometric distance between the center in the right reverse document frequency of word and unit and ring district.This makes the unit weight at the center of approaching be higher than unit at a distance.By each word of this algorithm process to after, search has a peaked unit in the Accum2 array.If that value has surpassed threshold value, then with its coordinate as the position of image fragment and report the coordinate of said unit.The norm_dist value that the activity array stores adds up.Because do not pass through idf with they convergent-divergents, they do not consider to comprise the quantity of the document in the right database of special word.Yet they provide with one group of given word really representes the two dimensional image of the x-y position of coupling.In addition, the document that is independent of in the database to be stored of the clauses and subclauses in the movable array.Can be with the inner this data structure derivation of using 3420 usually.
According to one embodiment of present invention, like the geometric distance of normalized shown in here.
Norm_geometric_dist(i，j，x，y，maxdist)
begin
d＝sqrt((i-x)
2+(j-y)
2)；
return(maxdist-d)；
end
Calculate the Euclidean distance between the center that the right position of word and ring distinguish, and return poor between this and the ultimate range that possibly calculate.
Handle through the evidence accumulation algorithm each word to after, search has peaked unit in the Accum2 array.If that value has surpassed predetermined threshold value, then with its coordinate as the position at the center of image fragment and report.
MMR type-script architecture
Figure 37 A illustrates MMR functions of components block diagram according to an embodiment of the invention.Basic MMR parts comprise having the computing machine 3705 that the printer 116 that is associated and/or shared document are explained (SDA) server 3755.
As known in the art, computing machine 3705 is desktop PC, laptop computer or the network computer of any standard.In one embodiment, computing machine is with reference to the described MMR computing machine 112 of Figure 1B.As the described herein, user's printer 116 is family, office or the business printer of any standard.User's printer 116 produces document printing 116, and it is the paper document of being made up of one or more printer pages.
SDA server 3755 is the network of the standard that has information, application program and/or the multiple file that is associated with the method for sharing note or the computing machine of centralization.For example, the shared note that is associated with webpage or other document is stored on the SDA server 3755.In this example, as the described herein, explain and be employed data among the MMR or mutual.SDA server 3755 is to connect addressable through the network according to an embodiment.In one embodiment, SDA server 3755 is with reference to the described network medium server 114 of Figure 1B.
Plug-in unit 3720 is for providing the software application of creation function.Plug-in unit 3720 is software application independently, perhaps alternately, is the plug-in unit of operation on the browser 3715.In one embodiment, plug-in unit 3720 is and the mutual computer program of the application program as browser 3715, so that particular functionality described herein to be provided.According to various embodiment, plug-in unit 3720 is carried out the various conversions and other change of the webpage that is shown in document or the browser 3715.For example; Plug-in unit 3720 with independent recognizable reference mark around the focus sign; To create focus; And html file that will " mark " version is back to browser 3715, the part of the document that transformation rule is applied to shown in the browser 3715, and retrieval and/or receive the sharing of documents that is shown in the browser 3715 and explain.In addition, plug-in unit 3720 can be carried out other function, for example creates through the document of change and creates symbol focus as the described herein and describe 3725.With reference to trapping module 3735, plug-in unit 3720 is convenient with reference to Figure 38,44,45,48 and the described method of 50A-B.
The symbol focus is described 3725 files for the focus in the identification document.The symbol focus is described 3725 hot period of identification and contents.In this example, the symbol focus is described 3725 and be stored in data-carrier store 3750.The example that the symbol focus is described is shown among Figure 41 in further detail.
Document and the webpage created for result through the file 3730 of change as the change of the source file 3710 through plug-in unit 3720 and conversion.For example, be a example like the mentioned html file of preceding text through the file 3730 of change through mark.As will be clearly,, be back to browser 3715 through the file 3730 of change, to be shown to the user in some situation according to this open invention.
Those skilled in the art will discern the output that trapping module 3735 is connected to browser 3715, so that data capture.Alternately, can in printer driver, directly realize the function of trapping module 3735.In one embodiment, trapping module 3735 is equal to as with reference to figure 3 described PD trapping modules 318.
Data-carrier store 3750 is used to store in order together to use any database of the file of being changed with method described herein for known in the art.For example, according to an embodiment, data-carrier store 3750 storage source files 3710, symbol focus describe 3725, page_desc.xml 3740, through the page layout that appears, share explain, document, focus definition and the character representation of image conversion.In one embodiment, data-carrier store 3750 is equal to as with reference to figure 3 described document event databases 320, and is equal to as with reference to the described Database Systems 3400 of figure 34A.
Figure 37 B illustrates one group of included in the MMR print software 3760 according to an embodiment of the invention software part.Should be understood that, in computing machine 112,905, acquisition equipment 106, network medium server 114 and other server as the described herein, can comprise all or some MMR print softwares 3760.Although will describe MMR print software 3760 now for comprising these various parts, those skilled in the art will discern, and MMR print software 3760 can have all any amount of these parts to them.MMR print software 3760 comprises conversion module 3765, merge module 3768, analysis module 3770, modular converter 3775, characteristic extracting module 3778, explains module 3780, focus module 3785, appears/display module 3790 and memory module 3795.
Merge module 3768 makes it possible to carry out the embedding corresponding to the mark of the sign of the focus in the electronic document, and is a kind of method of doing like this.In a particular embodiment, the starting point of the mark of embedding indication focus and the end point of focus.Alternately, can use the predetermined zone around the embodiment mark, discern the focus in the electronic document.Can use various such tagging schemes.
Feature extraction and coordinate that characteristic extracting module 3778 makes it possible to carry out representing corresponding to the printing of document and focus are caught, and are a kind of methods of doing like this.Coordinate is caught and is comprised that use forwarding dynamic link libraries branches to print command, and analysis is represented corresponding to the printing of the coordinate subclass of the character of focus or warp conversion.Characteristic extracting module 3778 makes it possible to realize the function according to the trapping module 3735 of an embodiment.
Appear/display module 3790 makes it possible to appear or the printing of display document or document is represented, and be a kind of method of doing like this.
The software module that software section 3765-3795 need not separate.Shown software arrangements only means as an example; As will be clearly according to this open invention, through with can expect other configuration within the scope of the invention.
In document, embed and hold a little
Figure 38 illustrates the flow process that in document, embeds the method for focus according to an embodiment of the invention.
According to this method, embedding 3810 is corresponding to the mark of the sign of the focus in the document in document.In one embodiment, receive the document that comprises the focus mark position,, for example, receive document from source file 3710 at browser 3715 in browser, to show.Focus comprises some texts or image pattern or other such document object of photo, and electronic data.Electronic data can comprise the multimedia as the audio or video, and perhaps it can be one group of step when the visit focus, will on acquisition equipment, carrying out.For example, if document is the HTML(Hypertext Markup Language) file, then browser 3715 can be InternetExplorer, and sign can be the URL (URL) in the html file.Figure 39 A illustrates the example of the such html file 3910 with URL 3920.Figure 40 A illustrates like browser 4010, for example, Internet Explorer, in the text of html file 3910 of Figure 39 A of being shown.
In order to embed 3810 marks, the plug-in unit 3720 of browser 3715 with independent recognizable reference mark around each focus mark position, to create focus.In one embodiment; The document that is shown in the plug-in unit 3720 change browsers 3715, for example, the HTML that is shown among the Internet Explorer of the example of continuation preceding text; And insert the focus mark position (for example, URL) is placed mark or label in the bracket.The terminal user who perhaps in browser 3715 or in the printing edition of document, checks document discovers less than mark, but in print command, can detect this mark.In this example, use the new font that is referred to herein as MMR Courier New, add beginning and finish reference mark.In MMRCourier New font, through representing about the exemplary glyph of character " b ", " e " or dot pattern is represented and numeral in the space.
Refer again to the exemplary html page shown in Figure 39 A and the 40A, plug-in unit 3720 inserts 3810 reference marks " b0 " in the beginning (" here ") of URL, and inserts 3810 reference marks " e0 " in the ending of URL, to indicate focus with identifier " 0 ".Because all as illustrating at interval, the user only can see the change that maybe can not see the outward appearance of document seldom for b, e and numerical character.In addition, as shown in Figure 41, plug-in unit 3720 is created the symbol focus of these marks of indication and is described 3725.It is 0 4120 that the symbol focus is described the hot period of 3725 identifications, and it is corresponding to 0 in " b0 " and " e0 " reference mark.In this example, the symbol focus is described 3725 and is stored in, for example, and data-carrier store 3750.
Shown in Figure 39 B, plug-in unit 3720 returns version to the browser 3715 of HTML3950 " through mark ".Through the HTML3950 of mark being that No. 1 the leap label 3960 of MMR Courier New is around reference mark i with Font Change.Because b, e and numerical character are as illustrate at interval, the user only can see the change that maybe can not see the outward appearance of document seldom.Through the HTML 3950 of mark is the example through the file 3730 of change.For the sake of simplicity, this example uses the single page model, yet the multipage surface model uses identical parameter.For example, if focus is crossed over page boundary, then it will have the reference mark corresponding to each page location, be identical about each focus identifier.
Next, the response print command catches 3820 corresponding to the coordinate of printing expression and focus.In one embodiment, trapping module 3735 " branches to " text and the drawing command in the print command.Trapping module 3735 is carried out all text and drawing command, and in addition, the x-y coordinate and the further feature of each character and/or image during intercepting and record are printed and represented.In this example; Trapping module 3735 relates to the device scene (DC) of printing expression; The handle of the structure that the text of its output for definition will depend on output format (that is, printer, window, file layout, memory buffer unit, or the like) and/or the printing of attributes of images are represented.In the process of the coordinate of catching 3820 printing expressions, use the reference mark that embeds among the HTML can discern focus at an easy rate.For example, when running into beginning label,, can find x-y position up to end mark if write down all characters.
According to an embodiment, trapping module 3735 is referred to herein as " DLL is caught in printing " for transmitting DLL, and it allows the interpolation or the change of the function of existing DLL.Transmit DLL In the view of the client fully as original DLL, yet, will call be forwarded to target (original) DLL before, other code (" branching to ") is added into some or all of functions.In this example, print and to catch DLL and be forwarding DLL about Windows Graphics Device Interface (Windows GDI) DLL gdi32.dll.Gdi32.dll has 600 output functions of surpassing, and what they were all need be forwarded.DLL is caught in printing, is referred to herein as gdi32_mmr.dll, allows the client to catch printout from any window application that uses DLL gdi32.dll to draw, and it only need carry out on local area computer, even be printed to remote server.
According to an embodiment, with gdi32_mmr.dll RNTO gdi32.dll, and be copied to Windows, impel its monitoring from the almost printing of each window application.According to another embodiment,, and be copied to master catalogue about the application program of its monitoring printing with gdi32_mmr.dll called after gdi32.dll.For example, be used on the monitoring Windows XP InternetExplorer Program Explorer.In this example, only this application program (for example, Internet Explorer) will automatically be called printing and catch the function among the DLL.
Figure 44 illustrates the process flow diagram of the employed process of forwarding DLL according to an embodiment of the invention.The function call that DLL gdi32_mmr.dll at first receives 4405 sensing gdi32.dll is caught in printing.In one embodiment, gdi32_mmr.dll receives all function calls of pointing to gdi32.dll.Approximate 200 of total function call that gdi32.dll monitoring is about 600, it is used for influencing with some mode the function of the outward appearance of printer page.Thereby, print catch DLL next confirm 4410 received whether call be monitored function call.If what received calls the function call that is not monitored, then this calls and walks around step 4415 until 4435, and transmits 4440 to gdi32.dll.
If it is monitored function call, then next this method confirms whether 4415 function calls specify the print apparatus scene (DC) of " newly ", that is, and and the printer DC that does not before also receive.This is through checking that with respect to the internal DC table printer DC confirms.As previous mentioned, the target that the DC encapsulation is used to draw (it possibly be printer, memory buffer unit, or the like), and picture font, color, or the like the same drawing setting.On DC, carry out all mapping operations (for example, LineTo (), DrawText (), or the like).If printer DC is not new, then there has been memory buffer unit corresponding to printer DC, and skips steps 4420.If printer DC is new, then create 4420 memory buffer unit DC corresponding to new printer DC.This memory buffer unit DC mirrors the outward appearance of printer page, and in this example, is equal to the printing that preceding text are correlated with and representes.Thereby, when printer DC is added into the internal DC table, create the memory buffer unit DC (and memory buffer unit) of identical dimensional, and make it with the internal DC table in printer DC be associated.
Whether next gdi32_mmr.dll confirms 4425 to call be the relevant function call of text.Approximate 12 of calling of 200 monitoring gdi32.dll are that text is relevant.If it is not that then skips steps 4430.If it is relevant that function call is a text, then that text is relevant output writes the 4430xml file, is referred to herein as page_desc.xml3740, shown in Figure 37 A.Page_desc.xml3740 is stored in, for example, and data-carrier store 3750.
Figure 42 A and 42B illustrate the exemplary page_desc.xml3740 about html file 3910 examples of being discussed with reference to figure 39A and 40A.Page_desc.xml3740 comprises x one by one, y, width and height, word 4210 (for example, character 4220 (for example, G) the coordinate information of all print texts Get) and one by one one by one.Coordinate is to exist with the form of putting, and it is the printing equivalent with respect to the pixel in the upper left corner of the page, except as otherwise noted.Page_desc.xml3740 also comprises the hot information that is in " sequence " form, for example beginning label 4230 and end mark 4240.For the focus (for example, page N is to page N+1's) of crossing over page boundary, it all illustrates on two pages (N and N+1); Focus identifier in two kinds of situation all is identical.In addition; Comprise the information that other is important among the page_desc.xml3740; For example the printer port title 4250, and it can be to following generation significant impact: .xml that is produced and .jpeg file, browser 3715 (or application program) title 4260 and the date of printing and time 4270 and counting (dpi) and resolution (res) about the per inch of the page 4280 and printable area 4290.
Refer again to Figure 44, and then call and be not relevant the confirming of text, perhaps and then that text is relevant output writes 4430 page_desc.xml3740, and gdi32_mmr.dll carries out 4435 about the function call on the memory buffer unit of DC.This step 4435 provides the output to printer, obtains the output of the memory buffer unit to the local area computer equally.Then, when increasing the page, the content of compression memory impact damper, and write out with the form of JPEG and PNG.Then function call is transmitted 4440 to gdi32.dll, it is as normally carrying out it.
Refer again to Figure 38, present 3830 and comprise the page layout that the printing that comprises focus is represented.In one embodiment, present 3830 and comprise document printing.Figure 40 B illustrates the example of printing edition 4011 of the html file 3910 of Figure 39 A and 40A.Notice that for the terminal user, reference mark is not obviously perceptible.The layout that appears is saved to, for example, and data-carrier store 3750.
According to an embodiment, printing is caught DLL the symbol focus is described data and page_desc.xml3740 in 3725, for example, shown in Figure 42 A-B, integrates with hotspot.xml 3745, as shown in Figure 43.In this example, when document printing, create hotspot.xml 3745.Example among Figure 43 illustrates focus 0 and appears at x=1303, y=350, and be that 190 pixels are wide and 71 pixels high.The content of focus is shown equally, that is, and http://www.ricoh.com.
Alternative embodiment according to trapping module 3820; Filtrator in Microsoft XPS (XML the prints explanation) print driver; Usually be known as " XPSDrv filtrator ", receive the text drawing command, and create like the described page_desc.xml file of preceding text.
Obvious perceptible focus
Figure 45 illustrates the process flow diagram of conversion according to an embodiment of the invention corresponding to the method for the character of the focus in the document.This method is changed document printing with indicating terminal user and the mode that presents the MMR identification software of focus.
At first, receive 4510 electronic documents that will print as character stream.For example, can receive 4510 documents at printer driver or in the software module that can filter character stream.In one embodiment, receive 4510 documents at browser 3715 from source file 3710.Figure 46 illustrates the example of the electronic edition of document 4610 according to an embodiment of the invention.Document 4610 in this example has two focuses, and one is associated with " listing hereinafter ", and one is associated with " possible prior art ".According to an embodiment, for the terminal user, focus is not obviously perceptible.Can perhaps, set up focus through with reference to the described coordinate catching method of Figure 38 according to other method described herein any one.
For beginning label is analyzed 4520 documents, the beginning of indication focus.Beginning label can be a reference mark as described earlier, or the mark of any other independent recognizable identification focus.In case find beginning label, just transformation rule is applied to the part of 4530 documents, that is, the character of beginning label and then is up to finding end mark.According to an embodiment, transformation rule impels the visible change corresponding to the part of the document of focus, for example through change character font or color.In this example, can be with original font, for example, Times New Roman is for conversion into different known fonts, for example, OCR-A.In another example, present text with different font colors, for example, blue#F86A.According to an embodiment, the process and the described process of preceding text of conversion font are similar.For example, if document 4610 is html files, then when in document 4510, running into reference mark, instead of fonts in html file just.
According to an embodiment, accomplish switch process through the plug-in unit 3720 of browser 3715, output is through the document 3730 of change.Figure 47 illustrates the example of the document 4710 of printing change according to an embodiment of the invention.Like what illustrated, from remaining text, focus 4720 and 4730 is visually recognizable.Especially, based on its different font, focus 4720 is visually recognizable, and based on its various colors and underscore, focus 4730 is visually recognizable.
Next, the document that will have the part of conversion presents 4540 becomes page layout, comprises the position of the focus in electronic document and the electronic document.In one embodiment, presenting document is document printing.In one embodiment, any according to the method for doing so described herein appears on the document that is included in the part with conversion and carries out feature extraction.In one embodiment, feature extraction comprises that according to an embodiment, the response print command is caught the page coordinates corresponding to electronic document.Be subclass analytical electron document then corresponding to the coordinate of the character of changing.According to an embodiment, the trapping module 3735 of Figure 37 A carries out feature extraction and/or coordinate is caught.
The MMR identification software uses identical each image of transformation rule pre-service.At first it seeks the text that follows the principles, and for example, it is OCR-A or blue#F86A, and it uses the recognizer of its standard then.
This aspect of the present invention is favourable, because it has reduced the computational load of MMR identification software fully, because it uses the very simple image pretreatment routine of eliminating a large amount of computing costs.In addition, for example,,, through eliminating a large amount of alternative solution that from select, to use, and improve the accuracy of feature extraction like the encirclement square frame on the part of document like what discussed with reference to figure 51A-D.In addition, visible which text of change indicating terminal user (or other document object) of text is the part of focus.
Shared document is explained
Figure 48 illustrates the process flow diagram of the method for shared document note according to an embodiment of the invention.This method makes the user in the environment of sharing, to append notes to document.Among the described hereinafter embodiment, shared environment is the webpage that various users are just consulting; Yet according to other embodiment, shared environment can be any environment of shared resource, for example working group therein.
According to this method, at browser, for example browser 3715, middle demonstration 4810 source documents.In one embodiment, from source file 3710 reception sources documents; In another embodiment, source document is for passing through network, and for example, the Internet connects, received webpage.Use the webpage example, Figure 49 A illustrates the sample source webpage 4910 in the browser according to an embodiment of the invention.In this example, webpage 4910 be about with popular child's the relevant recreation of books characteristic, the Jerry Butter Game, html file.
After the demonstration 4810 of source document, receive the sign of the part of the 4820 shared source documents of explaining and being associated with shared note, this is shared note and is associated with source document.Clear for describing, use single note in this example, yet a plurality of note is possible.In this example, explain and be data or as employed mutual in this MMR that discusses.According to an embodiment, explain and to be stored in shared document annotation server (SDA server), 3755 shown in Figure 37 A for example, and receive through the retrieval from this server.In one embodiment, connect addressable SDA server 3755 through network.This ability in this example that the plug-in unit of shared retrieval of explaining is convenient, for example, the plug-in unit 3720 shown in Figure 37 A.According to another embodiment, receive note and sign from the user.The user can create about the sharing of documents that does not have any note and explain, and perhaps can add or change existing the sharing of document and explain.For example, the user can add the part of bright source document, about indicating it with sharing explain related, is also provided via various methods described herein by the user.
Next, in browser, show 4830 documents through change.Document through change comprises the focus corresponding to the part of the source document of indicating in step 4820.Focus is specified and is shared the position of explaining.According to an embodiment,, and be back to browser 3715 through the part of file 3730 through change of document for being created of change by plug-in unit 3720.Figure 49 B illustrates the webpage 4920 of sample through changing in the browser according to an embodiment of the invention.The note 4940 that webpage 4920 illustrates the sign of focus 4930 and is associated, it is the video clipping in this example.Can visually distinguish sign 4930 from remaining webpage 4920 texts, for example, bright through adding.According to an embodiment,, explain 4940 and show when clicking sign 4930 or mouse being moved past tense.
The response print command catches 4840 corresponding to text coordinate and the focus represented through the printing of document of change.The details that coordinate is caught is any one according to about the method for that purpose described herein.
Then, present 4850 and comprise the page layout that the printing of focus is represented.According to an embodiment, present 4850 and be document printing.Figure 49 C illustrates sample printing webpage 4950 according to an embodiment of the invention.Printing network page layout 4950 comprises like the focus of being indicated 4930, yet the row in the printing layout 4950 interrupts being different from webpage 4920.In this example, the border of focus 4930 is sightless on printing layout 4950.
In optional last step, will share note and be stored in partly, for example, data-carrier store 3750, and use its with document printing 4950 in the related demarcation index of focus 4930.Also can preserve partly printing expression.In one embodiment, the printing behavior triggers the download and the establishment of local copies.
The focus of the document of image conversion
Figure 50 A illustrates the process flow diagram of interpolation focus according to an embodiment of the invention to the method for image conversion document.This method allows after scanning, focus to be added into paper document, perhaps after printing appears, focus is added into the symbol electronic document.
At first, be the document of image conversion with source document conversion 5010.According to an embodiment, at browser 3715 from source file 3710 reception sources documents.Conversion 5010 is through producing any method of the document that can carry out feature extraction above that, to produce character representation.According to an embodiment, paper document is scanned to become the document of image conversion.According to another embodiment, use appropriate application program to present the page sample of electronic document.For example, be the PostScript form if can present page sample, then use Ghostscript.Figure 51 A illustrates the example of the user interface 5105 of the part that the newsprint page 5110 that scans according to an embodiment is shown.Main window 5115 illustrates the part of the amplification of the newsprint page 5110, and sketch map 5120 illustrates which part of positive display page.
Next, feature extraction is used 5020 in the document of image conversion, to create character representation.For this purpose can be used any of various feature extracting methods described herein.According to an embodiment, through carrying out feature extraction with reference to the described trapping module 3735 of figure 37A.Then one or more focuses 5125 are added 5030 documents to image conversion.According to diversified embodiment, can define in advance maybe to need the definition focus.If defined focus, then define the coordinate position and the electronic data of the bounding box that comprises the focus on page number, the page or be attached to the mutual of focus.In one embodiment, like what illustrated among Figure 43, the form of hotspot.xml file is taked in the focus definition.
If also undefined focus, then the terminal user can define focus.Figure 50 B illustrates the process flow diagram of method that definition according to an embodiment of the invention is used to be added into the focus of image conversion document.At first, select 5032 candidate's focuses.For example, in Figure 51 A, the terminal user has used a part that bounding box 5125 selected document as focus.Next, about given database, confirm in optional step 5034 whether focus is unique.For example, n around " * n " should there be enough texts in the fragment, to discern focus uniquely.The example of the representative value of n is 2.If for database, focus is not sufficiently unique, then about how handling among the unclean embodiment, option is presented to the terminal user.For example, it is alternative that user interface can provide, and for example selects bigger zone, and it is ambiguous perhaps to accept, but its description is added into database.Other embodiment can use other method of definition focus.
In case select 5032 hotspot location, just define 5036 data or mutual, and it be attached to focus.Figure 51 B illustrates and is used for definition of data or mutual, with selected focus related user interface.For example, in case the user has selected bounding box 5125, just show edit box 5130.The button that use is associated, the user can cancel 5135 operations, only preserves 5140 bounding boxes 5125 simply, perhaps with data or assign 5145 alternately and give focus.If the user selects data or mutual the branch are tasked focus, then show and assign frame 5150, shown in Figure 51 C.Assign frame 5150 and allow the terminal user that image 5155, various other medium 5160 and network linking were tasked focus in 5165 minutes, it is by ID number 5170 identifications.The user can select to preserve the definition of 5175 focuses then.Although for the sake of simplicity, single focus has been described, a plurality of focuses are possible.Figure 51 D illustrates the user interface that is used for the focus 5125 in the display document.In one embodiment, the bounding box of different colours is corresponding to different data and type of interaction.
In optional step, the document of image conversion, focus definition and character representation are stored in 5040 together, for example, and data-carrier store 3750.
Figure 52 illustrates the method 5200 of use MMR document 500 according to an embodiment of the invention and the 100b of MMR system.
This method 5200 begins through the expression of obtaining 5,210 first documents or first document.The illustrative methods of obtaining first document comprises following: (1) obtains first document through the text layout of the document printing in the operating system of automatically catching MMR computing machine 112 via PD trapping module 318; (2) through the text layout of the document printing in the printer driver 316 of automatically catching MMR computing machine 112, obtain first document; (3) through via being connected to, for example, the standard document scanner device 127 scanning paper documents of MMR computing machine 112 obtain first document; And (4) through transmitting, upload or download automatically or manually, for the file of the expression of document printing to MMR computing machine 112, obtain first document.Although described obtaining step and be the great majority that obtain document printing or all, should be understood that, can be only carry out obtaining step 5210 about the part of the minimum of document printing.In addition,, can carry out this step, obtaining many documents, and create first document library although with regard to obtaining single document, described this method.
In case carried out obtaining step 5210, this method 5200 is just carried out 5212 index proving operations on first document.The index proving operation allows the respective electronic of document to represent and about the identification of second media type that is associated of the input that is complementary with first document that is obtained or its part.In an embodiment of this step, carry out the document index proving operation that produces PD index 322 through PD trapping module 318.Exemplary index proving operation comprises following: (1) indexs for the x-y position of the character of document printing; (2) index for the x-y position of the word of document printing; (3) index for the x-y position of the part of the image in image or the document printing; (4) carry out the operation of OCR image conversion, and index correspondingly for the x-y position of character and/or word; (4) carry out from the feature extraction that presents the image of the page, and index for the x-y position of characteristic; And the feature extraction on the symbol version of (5) simulation page, and index for the x-y position of characteristic.Index proving operation 5212 can comprise any one or group of the above-mentioned index proving operation that depends on application program of the present invention.
This method 5200 is also obtained 5,214 second documents.In this step 5214, second document that is obtained can be all document or the only part of second document (fragment).The illustrative methods of obtaining second document comprises following: (1) relies on one or more catch mechanisms 230 of acquisition equipment 106, scan text fragment; (2) one or more catch mechanisms 230 of dependence acquisition equipment 106, the scan text fragment, and subsequently, pretreatment image is to confirm correctly to extract the possibility of the feature description of being wanted.For example, if index is based on OCR's, then system possibly confirm whether image comprises line of text, and operates for the OCR of success, and whether image definition is enough.If this confirms failure, then scan another text fragment; (3) the machine-readable identification symbol (for example, International Standard Book Number (ISBN) or univeraal product code (UPC) code) of the document that is scanned is discerned in scanning; (4) data of the desired document of input identification or one group of document (for example, motion illustrated supplement magazine 2003 editions), and subsequently, through project (1) or (2) of using this method step, scan text fragment; (5) receive Email with second appended document; (6) receive second document through the document transmission; (7) with the part of one or more catch mechanisms 230 scan images of acquisition equipment 106; And (9) are with input media 166 inputs second document.
In case carried out step 5210 and 5214, this method is just carried out document or the pattern match between 5,216 first documents and second document.In one embodiment, this carries out through the document fingerprint matching of carrying out second document to the first document.Through inquiry PD index 322, on second media document, carry out document fingerprint matching operation.The example of document fingerprint matching with the synthetic descriptor of those characteristics, and is searched the document and the fragment of a part that comprises those descriptors for from the image of being caught, to extract characteristic in step 5214.Should be understood that, can repeatedly carry out this pattern match step, about each document once, wherein whether the many documents of database storing are complementary to confirm any document and second document in storehouse or the database.Alternately, index demarcating steps 5212 is added into the index of representing document sets with document 5210, and execution pattern coupling step once.
At last, this method 5200 carries out 5218 based on the result of step 5216 and alternatively based on the action of user's input.In one embodiment, this method 5200 search with, for example, be stored in the predetermined actions that the given document fragment in second medium 504 is associated, the focus 506 that finds as coupling in this second medium 504 and the step 5216 is associated.The example of predetermined actions comprises: (1) is from document event database 320, the Internet or other place, retrieving information; (2) information is write the position that the 100b of MMR system of the output of preparing receiving system is verified; (3) search information; (4) at client apparatus, for example on the acquisition equipment 106, display message, and guiding and user's interactive sessions; (5) inquiry determined action and data in method step 5216 are so that carry out (user's participation can be optional) after a while; And (6) are carried out determined action and data in method step 5216 immediately.The example results of this method step comprises information, through the execution of retrieval, some other actions of the document of change (for example; The purchase of stock or product), perhaps be sent to wired TV box; For example STB 126, the input of order, this STB (for example is connected to wired TV server; ISP's server 122), it is back to wired TV box with video.In case carry out step 5218, this method 5200 is accomplished and is finished.
Figure 53 illustrates the block diagram of the exemplary one group commercial entity 5300 related with the 100b of MMR system according to an embodiment of the invention.5300 groups of commercial entities comprise MMR ISP 5310, MMR consumer 5312, Guzman Dennis M. De 5314, printer user 5316, cellular telephone services supplier 5318, hardware manufacturer 5320, hardware retailer 5322, financial institution 5324, credit card processor 5326, document publisher 5328, document printer 5330, honour an agreement merchant 5332, wired TV supplier 5334, ISP 5336, software provider 5338, advertising company 5340 and commercial network 5370.
Guzman Dennis M. De 5314 is any supplier of digital multimedia product, for example Blockbuster Inc (Dallas, TX), it provides digital movie and video-game, and the U.S. (it provides digital music, film and TV to show for New York, Sony NY).
Cellular telephone services supplier 5318 is any cellular telephone services supplier; Verizon Wireless (Bedminster for example; NJ), Cingular Wireless (Atlanta, GA), T-Mobile USA (Bellevue, WA) and Sprint Nextel (Reston.VA).
Hardware manufacturer 5320 is the manufacturer of any hardware unit, for example the manufacturer of printer, cellular phone or PDA.Exemplary hardware manufacturer comprise Hewlett-Packard (Houston, TX), Motorola, Inc, (Schaumburg, IL) and the U.S. (New York, Sony Corporation NY).Hardware retailer 5322 is the retailer of any hardware unit, for example the retailer of printer, cellular phone or PDA.Exemplary hardware retailer comprise RadioShack Corporation (FortWorth, TX), Circuit City Stores, Inc. (Richmond; VA), Wal-Mart (Bentonville; AR) and Best Buy Co. (Richfield MN), but is not limited thereto.
As know, the merchant 5332 that honours an agreement is for being specially adapted for any third-party logistics warehouse of fulfiling of order.The exemplary merchant that honours an agreement comprise Corporate Disk Company (McHenry, IL), OrderMotion, Inc. (New York, NY) and Shipwire.com (Los Angeles CA), but is not limited thereto.
Wired TV supplier 5334 is any wired TV ISP, and for example, (Philadelphia, PA) (GreenwoodVillage CO), but is not limited thereto ComcastCorporation with Adelphia Communications.ISP 5336 is the representative of any entity of service that any kind of is provided.
Figure 54 illustrates method 5400 according to an embodiment of the invention, and it is for passing through to use the general easily business method of the MMR 100b of system.Method 5400 comprises step: opening relationships between at least two entities, confirm possible business transaction; Carry out at least one business transaction and pay product or service about this transaction.
At first, between at least two commercial entities 5300, set up 5410 relations.For example, can be at four categories widely, arrange commercial entity 5300 like (1) MMR founder, (2) MMR distribution person, (3) MMR user and (4) in other, and some commercial entity can belong to a more than category within it.According to this example, commercial entity 5300 is classified as follows:
● MMR founder-MMR ISP 5310, Guzman Dennis M. De 5314, document publisher 5328, document printer 5330, software provider 5338 and advertising company 5340;
● MMR distribution person-MMR ISP 5310, Guzman Dennis M. De 5314, cellular telephone services supplier 5318, hardware manufacturer 5320, hardware retailer 5322, document publisher 5328, document printer 5330, merchant 5332, wired TV supplier 5334, ISP 5336 and advertising company 5340 honour an agreement;
● MMR user-MMR consumer 5312, printer user 5316 and document printer 5330; And
● other-financial institution 5324 and credit card processor 5326.
For example in this method step; MMR ISP 5310 as MMR founder; With MMR consumer 5312 as MMR user, and as setting up commercial relations between MMR distribution person's cellular telephone services supplier 5318 and the hardware retailer 5322.In addition, hardware manufacturer 5320 has commercial relations with hardware retailer 5322, and its both is MMR distribution person.
Next, method 5400 is confirmed possible business transaction between the group of 5412 relations that have in step 5410 to be set up.Especially, between any two or more commercial entities 5300 multiple transaction can take place.Exemplary transaction comprises: purchase information; Buy actuals; Buy service; Buy bandwidth; The storage of purchase electronics; Buy advertisement; Buy the advertistics amount; Transport commodity; Sale information; Sell actuals; Sell service; Sell bandwidth; Sell electronic memory; Sell advertisement; Sell the advertistics amount; Lease/hire out; And opinion collection/grading/ballot.
In case method 5400 has been confirmed business transaction possible between the group, just use MMR system 100 to reach the agreement of 5414 at least one business transaction.The result's of conduct transaction multiple action possibly take place between any two or more commercial entities 5300 especially.Exemplary action comprises: purchase information; Receive order; For the more information point advances; Create advertising space; Part/remote access is provided; Sponsor; Transport; Create commercial relations; The storage private information; Information is passed to other object; Add content; And blog.
In case method 5400 has been reached the agreement of business transaction, just use MMR system 100 to pay the product or the service of 5416 these transaction, for example, to MMR consumer 5312.Especially, as the result of the business transaction of in method step 5414, being reached, between any two or more commercial entities 5300, can exchange plurality of kinds of contents.Exemplary content comprises: text; Web page interlinkage; Software; Still photo; Video; Audio frequency; With above any combination.In addition, for the facility transaction, between any two or more commercial entities 5300, can utilize multiple delivery mechanisms.Exemplary delivery mechanisms comprises: paper; Personal computer; Network computer; Acquisition equipment 106; The individual video device; Personal audio set; With above any combination.
Except like the present invention claiming according to right in the above embodiments and describing, following embodiment is disclosed.
According to one embodiment of present invention, Figure 55 illustrates the method that is used to visit mixed media document.
Obtain 5,510 first documents.For example, can obtain 5,510 first documents through using any expression of catching first document in the catch mechanism 230 of this description with reference to figure 2E.According to various embodiment, can obtain the integral body of 5,510 first documents, perhaps can obtain the part of 5,510 first documents.Can use be used for obtaining 5214 with reference to Figure 52 any in the method for the document of this description, obtain first document.
Retrieve 5,512 second documents.For example, can retrieve 5,512 second documents from central repositories.According to one embodiment of present invention, second document for such as with reference to figure 5 in the such MMR document of the MMR of this description document 500.Can retrieve 5,512 second documents based on first document with such as the such index of PD index 322.
Compare 5,514 first and second documents.For example, relatively 5,514 first and second documents can comprise that analyzing first and second documents confirms whether they are similar fully.In one embodiment, this accomplishes through the document fingerprint matching of carrying out from second document to first document.
What should pay attention to is in certain embodiments, can and compare 5,514 first and second documents with retrieval 5,512 second documents and be incorporated in the document coupling step.For example, can on first document, carry out document fingerprint matching operation through inquiry PD index 322.As another example, can on first document, carry out document fingerprint matching operation through the MMR index 3404 of inquiry MMR database 3400.The example of document fingerprint matching synthesizes descriptor from those characteristics, and searches document and fragment in those descriptors that comprise certain ratio for extracting characteristic from the image of step 5510, being caught.Can retrieve 5512 coupling documents then.It should be understood that and repeatedly to carry out this pattern match step for each document once, wherein the numerous document of database storing confirms in storehouse or database, whether to have document to mate first document.
Based on relatively carrying out 5516 actions.The action of execution 5516 can depend on the result of comparison.For example, in one embodiment, if 5514 indication documents couplings are relatively then carried out 5,516 first actions, and if relatively 5514 indication documents do not match, then carry out 5,516 second actions.In another embodiment, the action of execution 5516 can depend on second document of coupling first document.For example, can second document be associated with predetermined actions, and can carry out 5516 predetermined actions.In addition, the action of execution 5516 can be in response to user's selection.According to various embodiment of the present invention, next to 56 (c) in the example of this description actions that can carry out, various types with reference to Figure 56 (a).
Figure 56 (a) illustrates first example of the method that is used to visit the MMR document.Obtain 5510 documents 5604.From central repositories 5610 retrievals 5512 documents 5614.In one embodiment, central repositories 5610 is a document event database 320.In another embodiment, central repositories 5610 is a MMR database 3400.In another embodiment, can central repositories 5610 be embodied as any database of the electronic representation that can store document.
In the example of being illustrated, will move or medium 5618 is associated with document 5614.For example, can document 5614 be embodied as MMR document 500, and document 5614 can comprise the action or second medium 504.
Compare 5514 documents 5604 and documents 5614.In one embodiment, based on comparing execution 5620 and document 5614 associated action 5618.In another embodiment, based on comparing the medium 5618 that demonstration 5620 is associated with document 5614.
The action of relatively carrying out based on first and second documents allows document printing to comprise information, other dimension valuably.For example, document 5604 can be the part of tourism with the books of travelling.Given many travel purpose economy ground, that develop rapidly, when they arrived shop shelf, travel book usually was out-of-date.The user can adopt MMR system obtain and travel specific part (for example, the special article) information that be associated, that replenish of books or tourism books.
For example, through with the integrated camera of user's mobile phone obtain 5510 the tourism books chapters and sections.Retrieval 5512 and comparison 5514 have the MMR document 5614 of the fingerprint of the interested chapters and sections of coupling.MMR document 5614 comprises the information 5,618 5510 the part obtained, additional about the tourism books.For example, the information 5618 of replenishing can be for the electronic representation of the renewal of article, about the health consultation in special zone, or advertisement.Said additional information 5618 is shown to the user.Advantageously, information renewal or other is available to the user.
Can the information of replenishing be embodied as the medium file of any kind of.Can with based on first and second documents relatively carry out action, be used for showing the medium file that is associated with paper document.For example, document 5604 can be the page of sports journal.Retrieval 5512 and comparison 5514 have the MMR document 5614 of the fingerprint of coupling document 5604.Document 5614 comprises the medium 5618 that relates in the physical culture story of obtain on the page 5510, magazine, for example, and the video clipping of the ball of winning victory that article is described.Medium 5618 is shown 5620 to the user.Advantageously, the medium file that is associated with paper document is available to the user.
The information of replenishing can also be specific for the user.Based on first and second documents relatively carry out action can comprise the retrieval and the explicit user information specific.For example, document 5604 can be the newspaper article of relevant listed company.Retrieval 5512 and comparison 5514 have the MMR document 5614 of the fingerprint of coupling document 5604.Document 5614 comprises the metadata 5618 of the purport of discerning article.For example, come the retrieval user information specific based on metadata 5618.For example, can retrieve from online commission house and relate to the share-holding information of article, mentioning company, the user.User's information specific is shown to the user.Advantageously, the user's information specific that relates to paper document to the user for available.
Figure 56 (b) illustrates second instance of the method that is used to visit the MMR document.Obtain 5510 documents 5604, and from central repositories 5610 retrievals 5512 documents 5614.Compare 5514 documents 5604 and documents 5614, and the reception user imports 5628.Import 5628 in response to comparison 5514 with the user, carry out 5630 actions.
In one embodiment, document 5614 is a MMR document 500.The MMR document can comprise a plurality of second media 504.For selection, a plurality of second media 504 are presented to the user, and the user imports the second desired medium 504 of 5628 indications.For example, second medium 504 can be audio file, video file, text or application file, or the like.Show 5630 selected second media 504.As another example, the MMR document can comprise a plurality of associated action 504.For example, associated action can for retrieving information, with information write to a position, carry out to the search of information, on client apparatus display message, guiding interactive sessions, postpone the input of order of purchase and another system of arriving of action, change document, product for execution after a while.For choosing, a plurality of associated action 504 are presented to the user, and the user imports desired, the associated action 504 of 5628 indications.Carry out 5630 selected, associated action 504.As another example, the MMR document can comprise the combination of second medium and associated action 504, and the user imports 5628 and can indicate whether show 5,630 second media, or should not carry out 5630 associated action.Import 5628 through receiving the user, can advantageously carry out 5630 action for user's expectation customizes.
In another embodiment, document 5614 is for comprising the MMR document 500 of a plurality of focuses 506 (that is the position within document 5604).The user imports 5628 can indicate desired focus 506.In some cases, can desired focus 506 be associated with a plurality of actions and/or second medium 504 successively, and the user imports 5628 and can indicate further and will carry out which action and/or will show which medium.Advantageously, import 5628, can select special focus, and can carry out 5630 action for user's expectation customizes further by the user through receiving the user.
Figure 56 (c) illustrates the 3rd instance of the method that is used to visit the MMR document.Obtain 5510 documents 5604, and from central repositories 5610 retrievals 5512 documents 5614.Compare 5514 documents 5604 and documents 5614.For example, document 5604 can be ready paper document that it is associated with medium file for the user.
Obtain 5622 media 5624.For example, medium 5624 can for video file, audio file or application file (as, spreadsheet or word processing file etc.).Not deviate from scope of the present invention, another example of medium 5624 will be tangible to those skilled in the art.For example, can wait from camera, microphone, network, computer-readable medium and obtain 5622 media.
In one embodiment, document 5614 is a MMR document 500, and medium 5624 is stored as second medium 504.Can medium 5624 be associated with one or more focuses of document 5614.In one embodiment, receive 5625 users input.For example, the user imports 5625 and can indicate medium 5624 with which focus is associated.As another example, document 5614 can comprise various entities, and the user imports 5625 and can indicate medium 5624 and which entity associated.For example, can during normal printing, obtain the user is chosen as available entity from print driver.Describe at Figure 37 to 43 with in this subsidiary description and to be used for obtaining entity, various exemplary methods from print driver.
Medium is associated with the MMR document allows the user to replenish paper document valuably.For example, the user who prepares travelling can obtain the paper document that helps its portable and low alternative costs, and upgrades the electronic edition of document with the content of replenishing from mobile platform (such such as mobile phone).
Retrieve the MMR document based on expression that obtained, paper document and advantageously other dimension is introduced into paper document.Can other dimension be thought of as and can interior perhaps advertisement be provided to document space wherein, extra.For example, the newspaper advertisement that is used for new film can " link " to the consulting of film, and this means that the result who obtains of newspaper advertisement will be for such as the demonstration of consulting information such, that replenish.As another example, consulting of the film in magazine can " link " to the advertisement that is used for cinema.Content can link to other guide, and advertisement can link to other advertisements.Retrieve specific MMR document and can cause, perhaps can cause debit or note on user's account (for example, the exchange that is used for checking the bonus content) to user's credit or payment (for example, the exchange that is used for checking advertisement).Thereby various method of the present invention can be for useful to advertiser, content provider and user etc.
In one embodiment, retrieve the MMR document based on the identification of paper document.Obtain the expression of paper document, and with the fingerprint of itself and MMR document relatively.The MMR document of coupling can comprise medium, perhaps can the MMR document of said coupling be associated with special action.In response to the comparison of paper document and virtual multimedia document, carry out action.For example, can show the medium of the MMR document of coupling, the MMR document associated action that perhaps can carry out Yu mate.Retrieve the user that the MMR document advantageously allows paper document based on the identification of paper document and benefit from the information that paper document is replenished.In addition, retrieve the user that virtual multimedia document advantageously allows paper document, perhaps said content is added into virtual multimedia document in changing based on the identification of paper document.
In one embodiment, will combine with user's information specific, the information of personalization is provided to the user from the data of MMR document.In another embodiment, solicit in a plurality of actions that the user imports to confirm to carry out with the MMR document is associated which.
In one embodiment, retrieve the MMR document based on the identification of paper document.In response to the comparison of paper document and virtual multimedia document, carry out action.For example, can show the medium of the MMR document of coupling, the MMR document associated action that perhaps can carry out Yu mate.
Be described below another embodiment.
Figure 57 illustrates according to an embodiment of the invention, is used to visit the method for mixed media document.
Obtain 5,710 first documents.For example, can catch the expression of first document with reference to figure 2E any one in the catch mechanism 230 of this description, obtain 5,710 first documents through using.According to various embodiment, can in the entity of first embodiment, obtain 5710 said first documents, perhaps can obtain the part of 5,710 first documents.Can use any one in method this description, that be used for obtaining 5214 documents, obtain first document with reference to Figure 52.
Retrieve 5,712 second documents.For example, can retrieve 5,712 second documents from central repositories.According to one embodiment of present invention, second document be such as with reference to figure 5 in the such MMR document of the MMR of this description document 500.Can retrieve 5,712 second documents based on first document with such as PD index 322 such index.
Compare 5,714 first and second documents.For example, relatively 5,714 first and second documents can comprise that analyzing first and second documents confirms whether they are fully similar.In one embodiment, this accomplishes the document fingerprint matching that second document matches first document through execution.
What should pay attention to is in certain embodiments, can and compare 5,714 first and second documents with retrieval 5,712 second documents and be attached to document coupling step.For example, can come on first document, to carry out document fingerprint matching operation through inquiry PD index 322.The example of document fingerprint matching synthesizes descriptor for extracting characteristic from the image of step 5710, being caught from those characteristics, and document and the fragment of searching those descriptors that comprise certain ratio.Then, can retrieve the document of 5712 couplings.It should be understood that and repeatedly to carry out this pattern match step for each document once, wherein the numerous document of database storing confirms in storehouse or database, whether to have document to mate first document.
Optionally, can carry out the action that 5718 users select.Select in response to one user in the action in tabulation, can carry out the action that the user selects.According to various embodiment, the example of the action that can select to carry out in response to the user in this description with reference to Figure 58.
Figure 58 illustrates according to an embodiment of the invention, the user interface of visit mixed media document.Figure 58 is illustrated in demonstration on the display 212 of acquisition equipment 106, user action.
In the example of being illustrated, display menu 5801 on display 212.Menu 5801 comprises the document information 5802 that is associated with the document of being retrieved 5712.For example, document information 5802 can comprise the title of the document of being retrieved 5712.As another example, document information 5802 can comprise title, the page, author, publication date, ISBN number or relate to retrieve 5712 or obtain any other information or its part of 5710 document.For the purpose of illustrating provides these embodiment, and it is not for limiting.Can not deviate from scope of the present invention, another example of the document information 5802 that in menu 5801, shows will be tangible to those skilled in the art.
The action 5804 that on menu 5801, is shown can also depend on obtain 5710 zone document, special or position.For example, when the primary importance of obtaining document (or focus), show the action of confirming 5804, and when the second place of obtaining identical document (or focus), show that other move 5804.According to one embodiment of present invention, for selecting a plurality of focuses within document are shown to the user.In menu 5801, show the focus associated action of selecting with the user 5804.
In the example of being illustrated, menu 5801 comprises the action 5804A that is used for " seizing " document.Select in response to the user that seizes action,, be stored on the acquisition equipment 106 identifier of retrieve 5712 document or on other device for demonstration immediately or after a while, document.According to one embodiment of present invention, select in response to the user that seizes action, with retrieve 5712 document storage on acquisition equipment 106.For the reference in future, it is available for the user that the identifier of storage document or document itself make document valuably.The document of being retrieved can be for such as in such many forms such as Microsoft Word, Adobe PDF, Microsoft XML Paper Specification (XPS), Postscript and PCL a kind of.
For example, when when magazine is read by dentist's office, the user possibly meet interested article by chance.The user catches document, and selects action to seize document.Recognize document, and the identifier of document is stored on the acquisition equipment.Some place in the time after a while, for user's enjoyment, document is available.
In the example of being illustrated, menu 5801 comprises the action 5804B of the plan that is used to formulate reservation.For example, can document (or the position in document) and the activity that occurs at special date and time (such such as doctor's reservation and concert etc.) be associated.Select in response to the user of the action of making a plan, with the information stores of routing on acquisition equipment 106 or another device.In one embodiment, acquisition equipment 106 comprises the digital calendar application program, and selects in response to the user of the action of making a plan, and on digital calendar, formulates the plan for the reservation of the date and time that is associated with document.In another embodiment, the remote server storage reservation data, and select storage is associated with document on remote server reservation data in response to the user of the action of making a plan.In another embodiment, another computer installation comprises the digital calendar application program, and selects in response to the user of the action of making a plan, and on digital calendar, formulates the plan for the reservation of the date and time that is associated with document.For example, can on user
calendar, customize the plan of reservation.That the information of storage routing allows to make a plan based on document valuably, easily with efficient reservation.
For example, the user can see for making him bear the advertisement of the TV programme of interest.The user catches document, and selects action to come based on document to formulate the plan of reservation.Select in response to the user, reservation is added into the user's of the description that comprises time, date and incident calendar.According to various embodiment, can also reservation be added or instead be added into other calendar.For example, the characteristic that first user can show service time is given second user with the incident that is associated with document " recommendation ", and reservation is added into second user's calendar.As another example, the characteristic that the user can show service time is added into the calendar such as the such electronic installation of digital video recorder (DVR) or online automobile share service (such such as CityCarShare or ZipCar) with incident.In the example of television advertising, on user's DVR, formulate the plan of recording of TV programme.In the example of doctor's reservation, the time window for around doctor's reservation keeps the automobile of being shared.
Can obtain to be used for to formulate the reservation date of the plan of reservation in many ways.For example, can the reservation date be associated with retrieve 5712 document.As the result of document printing, can confirm the reservation date by document analysis device module 326.As another example, can obtain the reservation date through on obtain 5710 document, carrying out optical character identification (OCR).Can analyze such as description, the such reservation data of time and date from the character of being recognized.In a realization, the Entity recognition of appointment is extracted date and time from the character of being recognized.As another example, reservation data can comprise be associated and/or institute data retrieved and combining from the data of user's input.For example, can the description of the document of being retrieved 5712 with reservation be associated, and the user can import such as such, the other reservation data of time and date.
In the example of being illustrated, menu 5801 comprises the action 5804C that is used to make purchase.For example, can document (or the position in document) be associated in the commodity or the service (such such as document, medium file, ticket and magazine subscription etc.) of selling with capable of using.The user who moves in response to purchase selects, following order to commodity or service.The user who moves in response to purchase selects, can also be to account's charge of user.In one embodiment, the user who moves in response to purchase selects, for paying the plan of formulating the project of being bought.Can be immediately or the some place in the time after a while, the project of being bought is passed to acquisition equipment 106 or to another device.For example, can be at the next place synchronously of mancarried device, for the plan of formulating the project of being bought to the transmission of the mancarried device that video capability is arranged.The project of being bought can be for the copy of document itself (electronics or physics), relate to the project of document or the project of on the basis of document, being advised.For example, 5710 the document of obtaining can be the consulting of new TV programme, and in response to the selection of buying action, can buy the episode of program and make it capable of using on the mancarried device that downloads to video capability.Select in response to the user, the order for commodity or service allows the user to use easily the document based on papery to buy valuably down.
Can also be with buying the payment that action is used for commodity and service.For example, 5710 the document of obtaining can be the restaurant receipt, and in response to the selection of buying action, forwards fund the procuratorial account of restaurant or restaurant to from user's account, thus with user's debt payment to the restaurant.
Figure 59 illustrates according to an embodiment of the invention, is used for to the tagged method of medium file.For example, will can be image, audio frequency, video, text, portable document and report etc. by tagged medium file.For example, can be from camera on acquisition equipment 106 or microphone, or wait from the file transfers of this locality or long-range position and to obtain medium file.
The tabulation of catching title can comprise analyzes source file 310 (or other documents), so that location " entity of appointment " maybe can make multimedia explain interface information propagation, that other are interesting.For example, the entity of appointment is for can be applied to multimedia label after a while.For example, can document with (thereby) survey title, author, time or position in the tabulation of 5910 the title of catching.
With reference to Figure 60, illustrate sample source file 310.In the example of being illustrated, the document or the webpage that comprise for the agenda of " SanFrancisco Walking Tour " are shown.As directed, source file 310 comprise can catch 5910, such as so a plurality of titles such as several, time and positions.For example, in response to the printing of source file 310 or as the part of said printing, can; For example; Catch title " San Francisco ", " City Hall ", " Rose Garden " by PD trapping module 318; " McAuliffPark ", " Bayshore " and " Robin Williams ".As another example, source file 310 can be webpage, and in response to the checking of source file 310, can capture time " 9am ", " 10am ", " noon ", " 1pm " and " 2pm ".Can be not deviating from scope of the present invention, will be from another example of the title of catching to those skilled in the art for significantly such as source document 310 such documents.
The tabulation of title is shown 5912 to the user.Can the tabulation of title be shown 5912 on any device such such as MMR computing machine 112 or acquisition equipment 106.Show 5912 titles tabulation device can for or can be not be the device identical with the device of the tabulation of catching 5910 titles.For example, according to one embodiment of present invention, on first device, catch the tabulation 5910 of title, and said tabulation is delivered to second device electronically.Second device shows 5912 to the user with the tabulation of title.
With reference to Figure 60, display menu 6001 on display 212.Menu 6001 comprises a plurality of titles 6004 as user option.Show that 5912 titles 6004 to the user comprise the subclass of the tabulation of 5910 the title of catching at least.In the example of being illustrated, will from document 310 catch the title " City Hall " of 5910 title, " Robin Williams ", " McAuliff Park ", " RoseGarden " and " Bayshore " is rendered as user option.Can also from multiple document catch 5910 show 5912 title.According to one embodiment of present invention, the order that shows 5912 title is different from the order of catching 5910 titles with it.For example; Can based on the frequency (top that is placed on tabulation such as the title that will more generally select is such) of selecting title, the document of catching 5910 titles from it title conspicuousness and from its importance and conspicuousnesses etc. of catching the document of 5910 titles, come to arrange again show 5912 title.In addition, catch 5910 title therein, can use similar metering to confirm show 5912 which title than under show 5912 the more situation of title.
The tabulation that shows 5912 titles based on catch tabulation 5910, title advantageously is presented to the user with the option relevant with user's activity, for example, and the document of having printed or having checked recently based on the user.Thereby, present easily user interface with intelligence.
The user is chosen in one that shows on the menu 6001 in 5912 the title 6004.In response to user's selection, the name 5914 that medium file and user are selected.The name of can be in many ways medium file and user being selected 5914.For example, medium file can comprise one or more metadata fields, and the name storage that can the user be selected is in metadata fields.As another example, can title and medium file that the user selects be stored discretely, and pointer or other associations can be associated the title of user's selection with medium file.As another example, can the title that the user selects be used as the filename of medium file.As another embodiment, can the title that the user selects be used as medium file is stored in folder name wherein.Purpose in order to illustrate provides these examples, but is not limited thereto.Not deviate from scope of the present invention, other instances of the method for the name that is used for medium file and user are selected will be to those skilled in the art for significantly.
With reference to Figure 60, the related of title that medium file and user select is shown.In the example of being illustrated, medium file is with the image file 6005 of image with the stored in form of view data 6008.Select in response to the user, the title (" Rose Garden " in this example) that the user is selected as metadata 6006 storages 5914 in image file 6005." Rose Garden " is associated with medium file with label.Advantageously, the name selected of user is called tissue, the search in the future in medium file capable of using and/or shows.
In one embodiment, retrieve the MMR document based on the identification of paper document.Obtain the expression of paper document, and with the fingerprint of itself and MMR document relatively.In response to the comparison of paper document and virtual multimedia document, set is shown to the user.In response to user's selection, carry out the action that the user selects.For example, can retrieve and store the document of being recognized (or its identifier) for checking after a while.As another example, can formulate the plan of reservation.As another example, can buy product.
In another embodiment, the tabulation with the title of being caught comes to the medium file tagging in order to help the user.For example, catch the tabulation of title from document.The tabulation of title is shown to the user.In response to user's selection, the name that medium file and user are selected.
In one embodiment, retrieve the MMR document based on the identification of paper document.In response to the comparison of paper document and virtual multimedia document, set is shown to the user.In response to user's selection, carry out the action that the user selects.In another embodiment, the tabulation with the title of being caught comes to the medium file tagging in order to help the user.
Be described below another embodiment.
The generation of advertisement
Figure 61 A illustrates the process flow diagram of use according to the method 6100 of MMR document 500 of the present invention and the 100b of MMR system.Designing said method comes the mode that action is associated with image to MMR user 110 to be provided.
Through the expression and the action of obtaining 6102 images or image, method 6100 beginnings.In one embodiment, configuration is associated said action with image.Can image be included in the document of being printed or webpage.Image can comprise to the video camera 232 of acquisition equipment 106 or camera 234 its for visible but human its is sightless or very faint, machine-readable mark.The example of image comprises the image of trade mark and logo.Image can be with such as one in the such many standards of TIFF, BMP, GIF and JPEG.The action example comprise: (1) from document event database 320 (In a particular embodiment; Document event database 320 (comprising PD index 322) is embodied as the Database Systems 3400 shown in Figure 34 A), the Internet or other places (as, with movie trailer be downloaded to acquisition equipment 106 or such as iPod video video display devices such, that separate) come retrieving information; (2) information is write to the position by the 100b of the MMR system checking of the output that is ready to receiving system; (3) search information (as, the guiding this locality/global search); And (4) display message on the client apparatus such such as acquisition equipment 106, and guiding and user's interactive sessions (as, carry out general view).Yet, will can image be associated with a plurality of actions for being apparent that to those skilled in the art.
In one embodiment, create image, and obtain 6102 said images by the 100b of MMR system after a while by MMR user 110.The user can use by MMR ISP 5310 or by the third party software supplier (as, from the Paint of Microsoft company with from the Photoshop of Adobe company) software application that provides creates image.Can use diverse ways to obtain image.The illustrative methods of obtaining image comprises following: (1) is through the layout of the document printing within the operating system of coming automatically to be captured in MMR computing machine 112 via PD trapping module 318, to obtain image; (2) layout through the document printing within the printer driver 316 that automatically is captured in MMR computing machine 112 is to obtain image; (3) through via being connected to, for example, file scanning instrument apparatus 127 MMR computing machine 112, standard scans paper document, to obtain image; (4) obtain image with the image capture apparatus 106 that is included in the catch mechanism 230 shown in Fig. 2 E; And (5) through automatically or manually will be for the file transfers of the expression of image, upload or be downloaded to MMR computing machine 112.In another embodiment, method 6100 can be used the user to import (such as text or figure like this) and create image, to replace obtaining 6102 said images.Can use similar in appearance to those methods of image, diverse ways obtains 6102 actions.
For example, MMR user 110 plan will be associated with the official website (www.cocacola.com) that visits Coca-Cola company for the logo of the Coca-Cola of the trade mark of Coca-Cola company.Obtained the printing of logo representes by PD trapping module 318.The user inputs to action (visit www.cocacola.com) in the MMR computing machine 112 through the UI that the printer driver by MM 330 produces.Alternately, the user uses document scanner 127 or image capture apparatus 106 to scan the printed copies of logo, and comes input action with the user interface (UI) similar with described that user interface with regard to Figure 50 A.
In case carry out obtaining step 6102, method 6100 is just carried out the operation that 6104 index are demarcated on image.The operation that index is demarcated allows the electronic representation of corresponding image and identification input, associated action that is used to mate the image that is obtained.In an embodiment of step 6104, execution index is demarcated in two steps.In a first step, carry out at image based on the searching part of picture material 414 and be stored between the expression of the image in the document event database 320 image relatively.If parts 414 confirm that image matees institute's image stored at least with high confidence level, then method 6100 can confirm that image is not unique, and notice MMR user 110 is said definite, and ending method 6100.The user can select to change image and restart said process.
Suppose to confirm image for unique fully, second step of the operation that execution index is demarcated, it is added into document event database 320 with image and associated action.In a realization, described with regard to Figure 35, upgrade PD index 322 (or being equal to ground, MMR concordance list 3404).The example that can be added into the characteristic of said concordance list comprises following: (1) is in the x-y position of characters in images; (2) the x-y position of the word in image; (3) the x-y position of a part of image; (4) carry out the OCR imaging operation, thereby and for indexing the x-y position of character and/or word; And (5) carry out the feature extraction from image, and for indexing the x-y position of characteristic.The instance of feature extraction comprises Region Segmentation, color analysis, edge detection, face detection, texture analysis or is used for any one in the common employed method of the feature extraction of content-based retrieval algorithm.The operation 6104 that index is demarcated can comprise any one or the cohort wherein in the operation that index top, that depend on application program of the present invention demarcates.Document event database 320 with moving the result of the operation of demarcating with the image associated index, is stored in image in first medium part of MMR document 500, and action is stored in second medium part.
Continue the example of the Coca Cola of preceding text, carry out in CocaCola logo and the image comparison that is stored between the image in the document event database 320 based on the searching part 414 of picture material.Parts 414 confirm that logo is unique.Subsequently, PD trapping module 318 upgrades the PD index 322 that is used for logo (or be equal to ground, MMR concordance list module 3404 or some other special-purpose modules are upgraded index), and as described with regard to Figure 35, will move to be associated with image.If import image that scanned, logo, then carry out similar one group of step (as described with regard to Figure 35).In any case, in database 320, create MMR document 500, and the institute of logo first medium that is expressed as of (printed or scanned) that forms images, and to move be second medium.
Continue the example of the Coca Cola of preceding text, document can be the label on container for drink.Label comprises the image of CocaCola logo.MMR user 110 can use camera cell phone to obtain the image of label, obtains the image of logo thus.
In case executed obtaining step 6106, method just carry out 6108 at file and picture and be stored in the pattern match between the image in the database 320, confirm whether file and picture comprises any one in the database images.In one embodiment, carry out said pattern match step 6108 by document fingerprint matching module 226.Because file and picture can comprise a plurality of subimages that are stored in the database 320; So it should be understood that and repeatedly to carry out said pattern match step; For each subimage in file and picture once, determine whether in the subimage any one is included in the database.In another embodiment, by carrying out said pattern match step 6108 based on the searching part 414 of picture material.If the confidence level that is associated surpasses predetermined threshold value, any one in then definite file and picture or its subimage is in database.
In another embodiment, method is carried out at the document that is obtained and is stored in document or the pattern match 6108 between the document in the document event database 320.This can accomplish through carrying out document to the document fingerprint matching that is stored in the document in the database 320 of being obtained.
Continue the example of the Coca Cola of preceding text, method is carried out the pattern match between beverage label and CocaCola logo, and definite label comprises said logo.
Finally, method 6100 is based on the result of step 6108 and optionally based on user's input, carries out 6110 actions of in obtaining step 6102, being obtained.If step 6108 confirms that document comprises image, then method 6100 is carried out 6110 actions.It should be understood that pattern match step 6108 can confirm that document comprises other images that are stored in the database 320.For this reason, method 6100 can be carried out a plurality of actions, wherein with each all be included in document in image be associated.In case carry out step 6110, method 6100 just also finishes for what accomplish.
Continue the example of the Coca Cola of preceding text, because step 6108 confirms that document comprises the CocaCola logo, so method 6100 is carried out 6110 associated action---the official website of visit Coca-Cola company.As a result, www.cocacola.com comes searching web pages to camera cell phone from the website through the Internet.
Use the cooperation of the real border of blending agent document to explain
Figure 61 B illustrates the process flow diagram of use according to the method 6120 of MMR document 500 of the present invention and the 100b of MMR system.Method for designing 6120 comes to provide cooperation ground to add the mode for the easy note of utilizing to other MMR users to MMR user 110.
Through obtaining the expression of 6,122 first documents or first document, method 6120 beginnings.First document is the works that MMR user's 110 plans are added into note.Owing to can use with preceding text those method similar methods of described obtaining step 6102 with regard to Figure 61 A and obtain first document, therefore not be repeated in this description here.For example, the publisher of book " (Harry Potter and Sorcerer ' s Stone " can use document scanner 127 to scan this book, representes with the printing of obtaining 6122 books.
In case carry out obtaining step 6122, method 6120 is just carried out the operation that 6124 index are demarcated on first document.The operation 6124 that index is demarcated allows the electronic representation of corresponding first document and from the identification of second media type input, that be associated of " coupling " first document (the document fingerprint matching module 226 that use further describes) or its part with regard to Fig. 6.In an embodiment of said step, carry out the operation that document index is demarcated by the PD trapping module 318 that produces PD index 322.Be equal to ground, can be like the operation that with regard to described with regard to Figure 35, comes execution index to demarcate.The operation 6124 that index is demarcated can be included in reference in the operation of index demarcation described in the index demarcating steps 6104 of figure 61A, that depend on application program of the present invention any one.System will the result of first document in first medium of MMR document 500 part and index demarcating steps 6124 (as, PD index 322 or be equal to MMR concordance list 3404) together, be stored in the document event database 320.
In one embodiment, the 100b of system determines whether the different editions of first document is stored in the database 320.For example, first document is the copy of paperback edition, and the 100b of system possibly previously obtain the copy of the de luxe edition of identical document.Alternately, database 320 possibly have the different editions of one or more first documents.When the definite different editions with first document of the 100b of system was stored in the database 320, collaborative document is consulted parts 422 can be created in the intermediate line link layer between the version.The intermediate line link layer is associated the identical chapters and sections of different editions together.When MMR user 110 is just visiting version of first document, can dynamically create the intermediate line link layer.Alternately, when obtaining different versions, can create said intermediate line link layer.For this reason; When the chapters and sections of a version of MMR user 110 visit first document (as; First section of the preface of the copy of the paperback edition of " Harry Potter and Sorcerer ' s Stone ") time; Parts 422 can be identified in chapters and sections in other versions, corresponding (like first section of, the preface of the copy of the de luxe edition of " Harry Potter and Sorcerer ' s Stone ").
Continue the example of the Harry Potter of preceding text, the PD index 322 that PD trapping module 318 produces for the paperback edition version.Be equal to ground, can be like the operation that with regard to described with regard to Figure 35, comes execution index to demarcate.Database 320 is represented the printing of books to be stored in first medium part of MMR document 500 with PD index 322.Suppose that the MMR document 500 that also will be associated with de luxe edition " Harry Potter and Sorcerer ' sStone " is stored in the database 320, will create the intermediate line link layer, and said intermediate line link layer is stored in the database 320 between two versions.
Method 6120 is obtained 6126 relevant data, and said data are associated with first document.Relevant data can be relevant first document or its a part of note or note.Can do relevant data by one or more MMR users 110.In one embodiment, relevant data are in addressable content on the Internet.For example, the do for oneself entity of the BLOG that Harry Potter assigned can be the part of related data.The content that in the community website that said books are discussed, is provided by the user also can be the part of related data.For example, MMR user 110 feels that the note of the preface of relevant " Harry Potter and Sorcerer ' sStone " is interesting.Said user can comprise the focus 506 of preface and note is associated the preface and the note of next related said book with focus 506 through establishment.Alternately, the user can import relevant role's in said books his note, and said input is associated with said books, relevant chapters and sections.The user can use document production parts 442 to create association.Alternately, the user can use multimedia to explain software 410 and create note.Can be in the MMR document of creating for first document 500 with these notes and relevant data storage.What notice is can the multiple source of data be associated with first document, identical chapters and sections, and a plurality of MMR user 110 can create note.When in database 320, a plurality of version of first document being arranged, MMR user 110 can select to accomplish any one association in the version.
In case carry out obtaining step 6126, method 6120 is just obtained 6,128 second documents.In this step 6128, second document that is obtained can be all document or the just part of second document.Owing to can use with preceding text and obtain second document, therefore not repeat said description here with reference to those method similar methods of the described obtaining step 6106 of figure 61A.For example, can his camera of camera cell phone be pointed to said books to the interesting MMR user of the suggestion of the preface of the relevant books " Harry Potter and Sorcerer ' s Stone " of knowing other people, and obtain the picture of preface.As a result, camera cell phone obtains the image of the part that the printing that comprises preface representes.
In case executed step 6128, method 6120 are just carried out 6130 documents or pattern match between first document and second document.Owing to can therefore not repeat said description here to come the similar mode of described pattern match step 6108 to carry out said step with reference to figure 61A with preceding text.Method 6120 is also discerned and the part of second document that is obtained chapters and sections coupling, first document.
Continue the example of " Harry Potter and Sorcerer ' s Stone "; The feature description that document fingerprint matching module 226 produces for the part of second document that is obtained; In database 320, inquire about (its specific realization is provided) for the coupling document in Figure 34 A, and confirm first document and second document coupling.The module 226 further part of definite second document that is obtained is the preface of books.
Finally, method 6120 retrieval 6132 is stored in based on the result of step 6130 or optionally matees the data in second medium part of MMR document 500 of second document based on user's input.In first embodiment, the focus in the chapters and sections of first document that method 6120 is identified in the step 6128 to be obtained, and the data of second media type that is associated with the focus of being discerned of retrieval.Based on user input, method 6120 can show with focus in one or more data that are associated.When a plurality of version of second document, collaborative document is consulted parts 422 can retrieve the data that are associated with the focus of each version, and said version is corresponding to the part of second document that is obtained, and makes focus be ready to let the user consult.For example, the note that is associated with focus in the preface of the paperback edition copy of " Harry Potter and Sorcerer ' s Stone " and preface that de luxe edition copies of method 6120 retrieval.It is that method 6120 can point out his institute of user interested for which focus, thus and demonstration note.
In another embodiment, method 6120 is carried out action based on the result of step 6130, to replace the data of retrieve stored in second medium part.For example, method 6120 can running browser visits the webpage of note of the part correlation of second document that comprises and in step 6128, obtained.Alternately, method 6120 can move enable MMR user 110 import relative section, his application program of note, note is associated with said part, and with annotation storage in second medium part of MMR document 500.In case carry out step 6132, method 6120 just also finishes for what accomplish.
The media link
Fig. 6 C illustrates the process flow diagram of use according to the method 6140 of MMR document 500 of the present invention and the 100b of MMR system.Method for designing 6140 comes to provide instrument to come the relevant document of cross reference to MMR user 110.
Through obtaining 6,142 first documents (or expression of first document) and second document (or expression of second document), method 6140 beginnings.First and second documents are relative to each other.For example; First document can be " Hamlet " that is write by William Shakespeare; And second document can be for being " Cliff Notes:Hamlet Book Summary, the Monarch Notes-Hamlet " of the abreviation version of Hamlet, or the test of relevant Hamlet.
Owing to can use with preceding text those method similar methods of described obtaining step 6102 with regard to Figure 61 A and obtain two documents, therefore not repeat said description here.For example, the publisher of books " CliffNotes:Hamlet Book Summary " can use document scanner 127 to scan said books " Hamlet " and corresponding " Cliff Notes ", representes with the printing of obtaining 6,142 two documents.
In case carry out obtaining step 6142, method 6140 is just carried out the operation that 6144 index are demarcated on these two documents.The identification of the electronic representation that the operation that index is demarcated allows corresponding document and second media type input, that be associated of each or its part that are used for mating document.Index proving operation 6144 can be included in any one in the operation of demarcating with reference to the index described in the index demarcating steps 6104 figure 61A, that depend on application program of the present invention.The operation 6144 that index is demarcated can be created focus 506 in two documents, relevant chapters and sections, and with focus 506 with in other documents, the references of chapters and sections accordingly are associated.
For example; Method 6140 can be with the first act of " Hamlet ": the 4th be associated to reference in " Cliff Notes ", corresponding, and will represent to preserve with the printing of " Hamlet " in the reference in second medium part of MMR document.Method 6140 can also be created the reference of getting back to " Hamlet " accordingly, from " CliffNotes ".Alternately, method 6140 can (present the role in " Hamlet " with role analysis corresponding, in " Cliff Notes " like, each in Claudius) and to be associated.Can be automatically or come execution index proving operation 6144 with people's interference.For example, the publishing company of " Cliff Notes " can be provided at the information of the cross reference between " Hamlet " and " Cliff Notes ".Alternately, method 6140 can be used the software application information of the cross reference of living two documents of real estate automatically.Yet, for being apparent that, a plurality of documents associated with each other can be arranged to those skilled in the art, for example, the textbook of test book, teacher's guide and relevant identical purport.
In another embodiment, index demarcating steps 6144 can be created in the reference within the identical document.For example, can each the appearing subsequently to the reference of the chapters and sections of wherein at first introducing the role and role in identical document be associated.Can with on the document text in being stored in database 3400, realize this for the insertion of the text search of role's name and the link in the database of realizing cross reference.
In case execution index demarcating steps 6144, method 6140 are just obtained 6146 the 3rd documents.In this step 6146, the 3rd document that is obtained can be all document or the just part of the 3rd document.Owing to can use and preceding text those method similar methods with reference to the described obtaining step 6106 of figure 61A, obtain the 3rd document, therefore do not repeat said description here.For example, by " Hamlet " chapters and sections confused and be ready to know how " Cliff Notes " or " Monarch Notes " explains that the MMR user 110 of said chapters and sections can point to said chapters and sections with his camera of camera cell phone, and obtain its picture.As a result, camera cell phone obtains the image that the printing that comprises chapters and sections is represented.
In case executed step 6146, method 6140 are just carried out 6148 documents or pattern match between the 3rd document and the document that in step 6142, obtained.Owing to can carry out this step with reference to the similar mode of the described pattern match of figure 61A step 6108 with preceding text, therefore not repeat said description here.Suppose location coupling document, method 6140 is also discerned the chapters and sections of the document that matees with the part of being obtained of the 3rd document.
Continue the example of " Hamlet "; Document fingerprint matching module 226 produces the character representation of the part of being obtained that is used for " Hamlet "; In the document event database that is used for mating document 320 inquiries, and confirm that the 3rd document be " Hamlet " by the WilliamShakespeare of version in 2003 of Washington Square Press publication.The first act that the part of the 3rd documents that module 226 is further confirmed to be obtained is met for Hamlet and his father's ghost wherein: the part in the 4th.
Finally, based on the result of step 6148 with optionally based on user input, method 6140 retrievals 6150 are stored in the information of the cross reference in second medium part of MMR document 500 of coupling document.In one embodiment, the focus in the part of the coupling document that method 6140 is identified in the step 6146 to be obtained, and obtain the data of second media type that is associated with focus.Based on user input, method 6140 can show with focus in one or more data that are associated.For example, method 6140 retrieval and Hamlet wherein meet the reference information that the focus of said field of his father's ghost is associated.Method 6140 can point out his interested purport of user (as, historical background, said hint, relevant field or expert's note), and be shown to the reference of " Cliff Notes " document relevant based on user's input with other.In case carry out step 6150, method 6140 just also finishes for what accomplish.
In another embodiment, the text that method 6140 can retrieving reference be quoted, and with text display to the user, to replace the reference of retrieve stored in second medium part.
Dynamic document production
Figure 61 D illustrates the process flow diagram of use according to the method 6160 of MMR document 500 of the present invention and the 100b of MMR system.Method for designing 6160 provides instrument with dynamically creating multimedia document for MMR user 110 in capture content.
Through creating 6,162 first documents, method 6160 beginnings.First document is that MMR user 110 plans to be added into the document in will captive content.First document can be for such as the so any form of PowerPoint document, Word document and PDF document.Can in MMR computing machine 112 or network medium server 114, create first document.Can after a while first document be passed to acquisition equipment 106.Alternately, MMR user 110 can create first document in acquisition equipment 106.
For example, MMR user's 110 plans participate in a conference, and create the PowerPoint document of relevant meeting during participating in, and use the PowerPoint document to give a report then.Before leaving for meeting, MMR user 110 creates the PowerPoint document on MMR computing machine 112.Document is for empty or only comprise introduction meeting, brief.Then, MMR user 110 is pressed into document the memory storage 216 of acquisition equipment 106 through wired or wireless equipment.
Then, method 6160 is carried out 6164 and is obtained operation in the expression of second document or second document.Second document is planned to help the user to create first document.For example, second document can be for for the routing of meeting with for the plan of highway travelling etc.Owing to can use with preceding text those method similar methods of described obtaining step 6102 with regard to Figure 61 A and obtain second document, therefore not repeat said description here.For example, MMR user 110 can use printer 116 to print to comprise the webpage that conference journey is arranged.PD trapping module 318 is captured in the text layout that the conference journey within the operating system of MMR computing machine 112 is arranged.
In case carry out obtaining step 6164, method 6160 just can optionally be handled the entity that second document is discerned appointment.(note: the Entity recognition of appointment is the technology of knowing.) for example; Document analysis device module 326 can be analyzed second document, locate such as the lecture title (as, " An InconvenientTruth "), speaker's name (as; " Al Gore ") and the position (as, " Center Building ") entity of such, appointment of embedding.Can select the entity of appointment to describe the content of being caught after a while by the user.Document analysis device module 326 can also analyze such as such, that other the are relevant information with the duration of time for each incident in routing.Can the entity title of being located be stored in the memory storage 216 of acquisition equipment 106.
Then, method 6160 can be carried out the operation that 6166 index are demarcated on second document that is obtained.The operation that index is demarcated allows the electronic representation of corresponding second document and is used to mate second document that obtained or the identification of second media type input, that be associated of its part.The operation 6166 that index is demarcated can be included in reference in the index proving operation described in the index demarcating steps 6104 of figure 61A, that depend on application program of the present invention any one.
In case execution index demarcating steps 6166, method 6160 is just obtained 6168 contents.Can obtain said content by in the catch mechanism 230 of acquisition equipment 106 any one.For example; Always in a lecture end of meeting; MMR user can use video camera 232 to obtain speechmaker's video clipping, and use is looked screen video recorder 236 and write down the part of lecture, or uses keypad 214 to import the note of his relevant lecture.
In case carry out obtaining step 6168, method 6160 is just given the content tagging that in step 6168, is obtained, and changes 6,170 first documents through adding tagged content based on second document.For example, after the user caught the video clipping of lecture, acquisition equipment can point out the user to come from entity title (it is what earlier locate from second document) selection.The user can select such phrase such as " An InconvenientTruth " and " Al Gore ".As a result, will be inserted into video clipping and note " An InconvenientTruth, Al Gore " together in the PowerPoint document of before being created.In one embodiment, method 6160 can be used the machine time of processor 210 and time of in second document, listing is confirmed current incident.For example; Suppose that the lecture of origin that second document lists relevant universe is in beginning at 10 in the morning and in 11 end in the morning; And the user obtained audio clips in 10: 30 morning, and method 6160 can automatically add the label of " audio clip of the lecture the originof the universe " to audio clips.Alternately, the user can use keypad 214 that label is got to the content of being caught with explaining, or uses sound-track engraving apparatus 236 to write down the spoken annotations of his related content.
In another embodiment, method 6160 can also be to obtain the part of second document with preceding text with reference to the similar mode of those modes of the described obtaining step 6106 of figure 61A.For example, MMR user 110 can point to his camera of camera cell phone the chapters and sections that the conference journey printed is arranged, and obtains its picture.As a result, camera cell phone obtains and comprises image chapters and sections, the printing expression that conference journey is arranged.Then; Method 6160 can the printing of being obtained represent and second document between execution pattern coupling (using document fingerprint matching module 226); Confirm the content of the said chapters and sections of second document, and use the content of said chapters and sections to come to the content tagging of being caught.
Finally, method 6160 make the document changed to MMR user 110 for available.Then, the user can with or do not use document with further editor.For example, after meeting, the user can make expression not use the PowerPoint document of being changed with further editor.For this reason, when the user is just obtaining the content that is used for document, at least in part, dynamically create the PowerPoint document.
In another embodiment, real-time informing parts 424 can with the content of being obtained and note be published in real time webpage (as, BLOG), to replace dynamically creating document.
In of the present invention, the embodiment that separates, method 6160 need not created 6,162 first documents, and said method need not obtained 6168 data yet or change 6,170 first documents.Instead, method 6160 is obtained second document and indexing for it, and then, obtains the part of second document, carries out the content of the part that the document fingerprint matching confirms to be obtained, and is stored by the user and to be used for the content of further storing.For example, the MMR user 110 who engages in the short essay of relevant Hamlet can be scanned the books in identical field.When the user read said books after a while, no matter when he felt a certain section for useful, and he can both use his camera cell phone to obtain the picture of said chapters and sections.Camera cell phone can execution pattern mate the position of confirming the said chapters and sections in said books, and retrieves the content of said chapters and sections.When he accomplished the said books of reading, he can use the chapters and sections of text to write his short essay.
Unique image that at least one aspect of one or more embodiment of the present invention provides method to produce which user can relevant action.Method is obtained image, confirms that image is is unique, and produces the index that is used for image.After indexing for image, method is obtained document and execution pattern coupling between the document and the image of being indexed.If the image that the part coupling of the definite document that is obtained of method is indexed, then its carries out associated action.
It is convenient that at least one aspect of one or more embodiment of the present invention provides method that cooperation is explained.At first, method is obtained document and is indexed for it.The annotating information that joins with the part correlation of document is provided to user's cooperation.Then, method is obtained the part of document, the part that identification is obtained, and the annotating information that joins with said part correlation is provided.
At least one aspect of one or more embodiment of the present invention provides method to be provided at the link between the document.Method is obtained two documents that are associated, and with relevant partial association together.When method is obtained in the document one a part of, the part that its identification is obtained, and the relevant portion of other documents is provided.
At least one aspect of one or more embodiment of the present invention provides method dynamically to create document.Method is obtained document, the content of identification document, and be that the document that is obtained is indexed.Method is obtained data after a while, uses the content of document to explain the data of being obtained, and uses the data of being explained to create document.
In a particular embodiment, the MMR system provides method to come related relevant document.Method is obtained two documents that are associated, and with relevant partial association together.When method is obtained in the document one a part of, the part that its identification is obtained, and the relevant portion of other documents is provided.
Be described below another embodiment
The geolocation application program
Figure 62 illustrates the process flow diagram of use according to the method 6200 of MMR document 500 of the present invention and the 100b of MMR system.
Through obtaining the expression of 6210 documents or document, method 6200 beginnings.Document comprises some geographical location information.The example of document comprises following: (1) html file (as, the webpage of Google Maps); (2) comprise the address or the position of position title (as, comprise PDF document for the traffic route of National Historical museum); (3) document printing of relevant specific geographic area (like, the Rome city guide of publishing by LonelyPlanet publishing company).
Can use diverse ways to obtain document.The illustrative methods of obtaining document comprises following: (1) through the text layout of the document printing in the operating system of coming automatically to be captured in MMR computing machine 112 via PD trapping module 318 (or MMR type-script architecture of with regard to Figure 37 to 47, earlier describing), to obtain document with being equal to; (2) through the text layout of the document printing in the printer driver 316 that automatically is captured in MMR computing machine 112, obtain document; (3) through via being connected to, for example, file scanning instrument apparatus 127 MMR computing machine 112, standard scans paper document, obtains document; And (4) through automatically or manually will be for the file transfers of the expression of document printing, upload or be downloaded to MMR computing machine 112, obtain document.Although obtaining step is described as obtaining the great majority or all in the document printing, it should also be understood that can be only to the part of the minimum of document printing, carry out obtaining step 6210.In addition, although just with regard to catching single document in the said step, come describing method, those skilled in the art also will recognize can carry out said step, with the storehouse that obtains a plurality of documents and create document.
For example, MMR user's 110 plans are had a meal in coming at the restaurant, and use MMR computing machine 112 and user's printer 116 to print the html file of the menu that comprises the restaurant.PD trapping module 318 obtains 6210 and is represented by the printing of the html file of the text layout that obtains print menu.
In case carry out obtaining step 6210, method 6200 is carried out 6212 index proving operations on document.The identification of electronic representation that the index proving operation allows corresponding document and second media type input, that be associated that is used to mate first document that obtained or its part.In an embodiment of said step, carry out the document index proving operation by the PD trapping module 318 that produces PD index 322.Exemplary index proving operation comprises following: (1) indexs for the x-y position of the character of document printing; (2) index for the x-y position of the word of document printing; (3) index for the x-y position of the part of image or image in document printing; (4) carry out the operation of OCR image conversion, thereby and index for the x-y position of character and/or word; (4) carry out feature extraction, and index for the x-y position of characteristic from the image of the page that is appeared; And the feature extraction of (5) simulation on the symbol version of the page, and index for the x-y position of characteristic.Index proving operation 6212 can comprise any one in the cohort of the index proving operation that depends on index proving operations application program of the present invention, preceding text and preceding text thereof.Document event database 320 is together stored the result of document (or expression of document) with the index proving operation.
Continue the example of restaurant menu, when printing comprised the html file of menu, PD trapping module 318 represented to carry out 6212 index proving operations in the printing of html file, and generation is used to print the PD index 322 of expression.The printing of menu is represented that both are stored in the document event database 320 with PD index 322.
In case carry out obtaining step 6210, method 6200 is just carried out 6214 and is extracted and operate on document, the geographic position that is associated with identification.In an embodiment of extraction step 6214, document analysis device module 326 analytical documentations to locate the relevant information of embedded geography, are for example represented, as HallidieBuilding, the address of position and entity title.The address need not be the form with address, and it can be for cross street or only be the city title.Then, document analysis device module 326 can be applied to the entity title of being located with entity title identifying, confirms the address of entity, or latitude and longitude coordinate.Document analysis device module 326 can be used business database, for example by Google Earth use that, guides entity title identifying.
Then, can the information conversion that geography is relevant become the geographic position.For example, can use geographical data bank, that is for example used by Google Earth is transformed into the title of address and position their corresponding latitude and longitude coordinate.Then, document analysis device module 326 can transfer to the geographic position MMR user 110 acquisition equipment 106.Can the geographic position be stored in the memory storage 216 of acquisition equipment 106.
In another embodiment of extraction step, can the geographic position be provided by MMR user 110.For example, when handling following geographic position by the 100b of MMR system, Lonely PlanetPublications can provide place and the address of map and/or the geographic position of latitude and longitude coordinate in the books that are included in its travelling.Lonely Planet Publications can use document production parts 442; Create each MMR document 500 of books that are used for travelling; Establishment is for the place in the books of travelling and the focus 506 of map; In second medium part of MMR document, add geographical location information, and the geographic position is associated with corresponding focus 506.When MMR user 110 points to the page of the books of travelling with acquisition equipment 106; Document fingerprint matching module 226 execution patterns coupling; Identification is extracted the geographic position that provide, that be associated by Lonely Planet Publications thus with the MMR document 500 of the books focus on the said page 506, travelling.Document fingerprint matching module 226 can also transfer to acquisition equipment 106 with geographical location information.
In another embodiment of extraction step, can in the metadata of document, the geographic position be provided.For example, in html file, can geographical location information be provided such as crossing in the such html tag of label.There is the leap label of attribute " EXECUTE_WHEN_PRINTED " can point to the quoted string that comprises order and parameter.When printing comprises the document of such leap label, needn't be on document printing display label.Yet; By document analysis device module 326 analyze order with it parameter (as; Represent by the leap label that attribute " EXECUTE_WHEN_PRINTED " is arranged); And it is stored in second medium part of MMR document of html file, and said order and parameter and document printing, tagged part correlation couplet.The label of position sensing is specified the geographical location information for the part that is associated, and the action that can trigger based on the position of acquisition equipment 106 can optionally be provided.
For example, the following content of html file for the label that do not have position sensing:
<HTML>
<BODY>
The?Corner?Restaurant?at?Main?and?Cherry
Has?a?special?habanero?burrito.
</BODY>
</HTML>
After the responsive label in insertion position, the content of html file can become:
<HTML>
<HEAD>
<STYLE?type＝”text/css”>
span.send_to_phone_when_printed{color:#ff0000)
</STYLE>
</HEAD>
<BODY>
The<SPAN?class＝”send_to_phone_when_printed”
EXECUTE_WHEN_PRINTED＝”SEND_TO_PHONE?COMMAND＝PROVIDE_GUIDANCE”
LAT＝24:36:21?LONG＝56:45:06?THRESH＝1km?OUTPUT＝audio”>Corner
Restaurant</SPAN>at?Main?and?Cherry
Has?a?special?habanero?burrito.
</BODY>
</HTML>
Document analysis device module 326 can be from document extraction order (" SEND_TO_PHONE ") with the content that is sent out (" COMMAND=PROVIDE_GUIDANCE LAT=24:36:21 LONG=56:45:06THRESH=1km OUTPUT=audio ").Document analysis device module 326 can also be with the acquisition equipment 106 of content delivery to MMR user 110.Under said situation, when the user is in 1 kilometer in restaurant the time, content is called to the label Corner restaurant, position sensing for audio frequency is guided.
The order that those skilled in the art will notice in html tag to be provided has the application program that is different from like the geographic position application program described in the method 6200.For example, can come configuration order to upgrade user's calendar through being added on the reservation of appointment in the html tag (or the other places in document).Order can also be upgraded contact manager with associated person information; With notice (as; Email message and/or voice mail) be sent to the user or the personnel that indicated (as; The document distribution manager of relevant entity), and said user or the personnel that indicated announce document printing, and should it be torn to shreds.
Continue the example of restaurant menu, document analysis device module 326 is from the leap tag extraction geographical location information html file, and with the camera cell phone of information transmission to MMR user 110.Optionally, document analysis device module 326 also transfers to camera cell phone with the graphical representation of menu.In alternative realization; To extract the leap label by the MMR type-script architecture of with regard to Figure 37 to 47, earlier describing; It is kept in the database 3750 (being equal to ground MMR database 34A); And, transmit it to acquisition equipment 106 by unconscious upload component 446 in response to document printing.
In case carry out extraction step 6214, method 6200 is just carried out 6216 identifying operations, confirms the geographic position of acquisition equipment 106.As described in Fig. 2 E, the GPS device 242 of acquisition equipment 106 can rely on triangulation to confirm in real time the geographic position of acquisition equipment 106.With regard to Figure 1B 142 with regard to the method geographic position, alternative of confirming is described, and do not require the GPS device in acquisition equipment 142 the above method.
Continue the example of restaurant menu, have his camera cell phone and the MMR user 110 of the menu of being printed to drive to the restaurant.As MMR user 110 during near the restaurant, the GPS device 242 of camera cell phone (or method definite geographic position, alternative) is discerned the position of phone in real time.Notice, because camera cell phone has earlier from the copy of document analysis device module 326 graphical representations that receive, menu, so MMR user 110 need not bring the menu of being printed.
Finally, method 6200 is based on the geographic position of geographic position of being extracted and acquisition equipment 106, and optionally based on user's input, carries out 6218 actions.In one embodiment, method 6200 is searched and the geographic position associated action of being extracted, and as for example, it is stored in the memory storage 216 of acquisition equipment 106 with the geographic position that is associated.Can will move and transfer to acquisition equipment 106 by document analysis device module 326 with the geographic position.Alternately, action can be predetermined actions.The example of predetermined actions comprises: (1) provides directional information to the geographic position that is associated; (2) come retrieving information from document event database 320, the Internet or other places; And extract and the geographic position associated action (3).In case carry out step 6218, method 6200 just also finishes for what accomplish.
In one embodiment, method 6200 is carried out 6218 actions through the geographic position of being extracted being transferred to the third party software module 228 of highway section homing capability.Then, third party software module 228 can be directed to the geographic position of being extracted with MMR user 110 based on the position data that is provided by GPS device 242.Alternately; Acquisition equipment 106 can pass through wireless communication link 218; With the combining of Wi-Fi technology 134 or
technology 136 in; Or by wired connection to external device (ED); The geographic position of being extracted is transferred to external device (ED) (like, the gps system in MMR user's automobile).
When MMR user 110 when contiguous, acquisition equipment 106 can utilize previous employed geographic position, and prompting MMR user 110.Alternately; Suppose that MMR user had before used the MMR navigation Service to visit the subway restaurant; Even the previous subway restaurant of visiting is not in the positional information in contiguous subway restaurant; Acquisition equipment 106 also can come to provide the positional information in said contiguous subway restaurant to the user based on user's position.When being applicable, acquisition equipment 106 can be pointed out MMR user 110 through display 212, to select one as a plurality of geographic position of extracting.
In another embodiment, method 6200 provides to MMR user 110 through the advertisement that in the display 212 of acquisition equipment 106, will aim at reluctantly, carries out 6218 actions.For example, method 6200 can provide for the advertisement that is positioned near the commerce the geographic position of being extracted on the geography, and service that is provided and product felt to be given to MMR user 110.Alternately, method 6200 can provide the advertisement of the commerce between the geographic position of extracting for being positioned and MMR user's 106 the current location.
Method 6200 can further customize the advertisement that will show through MMR user 110 preference.Method 6200 can be confirmed MMR user 110 preference through his the previous document of printing.Method 6200 can be from these documents of document event database 320 retrieval, and handle the purport that they discern document (as, the keyword of document and theme).For example, MMR user 110 prints and is listed in the travel folder that the place of historic significance is arranged in the San Francisco zone, several.Method 6200 will show the advertisement in the restaurant of historic significance, to replace showing the advertisement near, conventional restaurant.Said advertisement can also comprise clocking information (like, the timetable of cinema) and sales promotion information (like, electronic coupons).Can said advertisement be provided to acquisition equipment 106 through local superminiature base station or through advertising company 740 by cellular service supplier 718.
In another embodiment, the geographical location information that transfers to acquisition equipment 106 comprises the entity title of representing the geographic position.For example, MMR user 110 prints the route timetable that comprises Yellowstone National Park (Yellow Stone), Gold Gate Bridge (Golden Gate) and Grand Canyon (Grand Canyon).Except that the latitude and longitude coordinate in three places, document analysis device module 326 also transfers to acquisition equipment 106 with the entity title that is associated (" Yellow Stone ", " Golden Gate " and " Grand Canyon ").Then, can use the entity title to explain the data of in these places, being caught.For example, camera 234, phonographic recorder 236 and video camera 232 that the user uses acquisition equipment 106 respectively take several pictures that magnify the valley, are recorded in the howl of a wolf in the Yellowstone National Park, and on the Gold Gate Bridge, catch the video in SanFrancisco urban district.Because acquisition equipment 106 can be confirmed its real time position through GPS device 242 (or confirming method alternative, the geographic position); So acquisition equipment 106 can confirm that photograph is at the Grand Canyon photograph; Audio clips is recorded at Yellowstone National Park; And video clipping is taken the photograph on the Gold Gate Bridge, thereby and it is explained also/or with its name.When catching photograph, audio/video clips; Acquisition equipment 106 can also be with the geographic position of acquisition equipment 106; Explain said photograph, audio/video clips, and use the geographical location information explained that they are mapped to geographic application program (like, Google Earth).
Continue the example of restaurant menu, camera cell phone uses audio frequency to guide, and MMR user 110 is guided to the restaurant.Camera cell phone can also show the advertisement near restaurant.
At least one aspect of one or more embodiment of the present invention provides method, comes to provide the information of the position that relates to the user to the user.Said method is obtained document, from the relevant information of document extracting position, discerns user's position, and based on position of being discerned and the information of being extracted information is provided to the user.
In a particular embodiment, the MMR system provides the information of the position that relates to the user to the user.System obtains document, from the relevant information of document extracting position, discerns user's position, and based on position of being discerned and the information of being extracted information is provided to the user.
Be described below another embodiment.
MMR system with MMR massaging device
Figure 63 illustrates the functional block diagram of the embodiment that comprises the MMR system 6300 that does not have the MMR of optical acquisition massaging device 6301.Figure 64 be illustrated on the MMR massaging device of MMR system of Figure 63, for the user interface of the menu of MMR document 500.Figure 65 is illustrated in the user interface of the MMR document that shown on the MMR massaging device 6,301 500 and hot information.
In said embodiment, the user does not have the acquisition equipment 106 of optical acquisition and pattern-recognition.Instead, MMR massaging device 6301 can guide to the focus in document with the user, and hot information is provided to the user.In alternative embodiment, 6301 storages of MMR massaging device also show MMR document 500 and the focus in MMR document 500.MMR massaging device 6301 can with the similar like this mode of the described acquisition equipment 106 of preceding text, come to communicate by letter with MMR system 100.
MMR massaging device 6301 can be stored the hot information of MMR information, the focus identifier within MMR document 500 and the MMR document 500 of the text that comprises MMR document 500 and picture.For example, the focus identifier can be icon or link.The user uses the user interface on MMR massaging device 6301, is chosen in the focus in the document.In response to user's selection, MMR massaging device 6301 will provide to the user like the hot information text, image, the audio or video.For example, the user can check the user interface of Figure 64, and selects the article of autograph into " Prime Minister Koizumi Visits Graceland ".In response to user's selection, MMR massaging device 6301 can display video or text, perhaps plays the audio frequency of prime minister to the visit of Graceland.
MMR massaging device 6301 comprises through network 128, from the system interface 6310 of network medium server 114 or MMR medium memory device 162 download MMR information and hot information.MMR information can comprise text, image, audio frequency or video, or its any combination, and can be for the form of blog.Hot information can be provided for buying the product that relates to focus or the information of service.Can MMR information and hot information be formatted as MMR database 320.
For the user search that uses user interface 6311, MMR massaging device 6301 is housed in MMR information and hot information in the storer 6313.In one embodiment, 6301 downloads of MMR massaging device are for the MMR information and the hot information of goods catalogue, magazine, newspaper and the periodical of consumer premise.The user can select for the information of downloading on MMR massaging device 6301 or through MMR computing machine 112.MMR massaging device 6301 can be downloaded said information when the user selects information, or can be in the time after a while, if for example MMR massaging device 6301 is docked download message with MMR computing machine 112.
MMR massaging device 6301 comprises in response to the user corresponding to the menu item of focus to be selected, and shows the display 6312 of text of menu and hot information of the optional part of MMR document 500.In an embodiment shown in Figure 64, menu is included in the article in the newspaper that comprises focus.Optional part can comprise the text of the part, whole of MMR document 500 or part, and can be included in the designator of the focus in these parts.Figure 65 illustrates the whole text of the article that is presented in the MMR document and the user interface of focus designator 6501, and said focus designator can be for similar with focus 506 (see figure 5)s in document.The user can select focus, retrieves the hot information that is associated.Its focus Duan Jie, above the selection that relates in the house visit impels MMR massaging device 6301, comes when in visit, escorting the prime minister, plays the audio or video of prime minister to the part of the visit of Graceland.Its relate to Duan Jie in prime minister's singing, select lower focus to impel MMR massaging device 6301, play the audio or video of at first singing.
In one embodiment, user interface 6311 is identical with display 6312.
For example, MMR massaging device 6301 can comprise audio player 6315 or video player 6316 or both.
In one embodiment, audio player 6315 is selected in response to the user corresponding to the menu item of focus, plays the audio frequency of hot information.For example, MMR document 500 can comprise such as museum or the such amusement of film, and hot information can be audio frequency, direction or the telephone number of trip time.In one embodiment, video player 6316 is selected in response to the user corresponding to the menu item of focus, plays the video of hot information.For example, MMR document 500 can comprise movie listings, and hot information can be the movie trailer for selected film.
In one embodiment, MMR massaging device 6301 comprises to the audio frequency of digital quantizer 6317, and said converter will be transformed into user command through the voice command that microphone 6318 receives from the user.In response to the user command of institute's conversion, the selection of the part of MMR massaging device 6301 Control-Menu items or MMR document.The user reads text from MMR document 500.MMR massaging device 6301 is transformed into text with audio frequency, and with text and the MMR information of being stored relatively.
Figure 66 illustrates the process flow diagram of the operation of MMR massaging device 6301.MMR information and hot information are stored 6602 in storer 6313.Processor 6314 shows that 6604 have the part of the MMR document 500 of focus indicated on display 6312, that kind shown in Figure 65.User in response on user interface 6311 selects 6606, and processor 6314 provides 6608 to be text or video hot information on display 6312, or provides 6608 to be audio frequency said hot information.
Figure 67 illustrates the functional block diagram of embodiment of the MMR system 6700 of the MMR massaging device 6301 that comprises acquisition equipment 106 and do not have optical acquisition.
Though MMR system 6700 is similar with MMR system 6300, it further comprises acquisition equipment 106, and it locatees the focus in the MMR document 500 in order to help the user, from MMR massaging device 6301 reception information.
In alternative embodiment, have only in acquisition equipment 106 and the MMR massaging device 6301 one to communicate by letter, and information is forwarded to other device with MMR system 100.
In alternative embodiment, acquisition equipment 106 and MMR massaging device 6301 are constituted integrated device.
In one embodiment, acquisition equipment 106 is surveyed focus, and order is sent to the MMR massaging device 6301 for device 6301, retrieves the hot information that is associated.In one embodiment, MMR massaging device 6301 can provide hot information as described in the associating of preceding text and Figure 63.In alternative embodiment, in order to provide to the user, MMR massaging device 6301 can be sent to acquisition equipment 106 with hot information.In alternative embodiment, MMR massaging device 6301 can postpone to carry out the order of deriving from focus (as, buy order at the product shown in the MMR document 500), communicate by letter with MMR system 100 up to it.
In another embodiment, for store M MR document information and hot information, can be with storer subregion between MMR massaging device 6301 and acquisition equipment 106.
In one embodiment, acquisition equipment 106 or MMR massaging device 6301 do not comprise the display that presents MMR document information and hot information.Device 106 and 6301 intercommunications mutually, the information that is provided for showing.
Figure 68 illustrates the process flow diagram of the operation of MMR system 6700.As the part of downloading, processor 6314 stores 6802 in storer 6313 with MMR information and hot information.MMR massaging device 6301 provides 6804 to acquisition equipment 106 with the position of focus.Acquisition equipment 106 is surveyed focus.If the user selects 6806 focuses on acquisition equipment 106, then for the retrieval of the hot information of being asked, acquisition equipment 106 is with recall signal transmission 6808 to MMR massaging devices 6301.MMR massaging device 6301 with hot information as text, image or video on display 6312, or provide 6810 for audio frequency with it, perhaps hot information is provided to acquisition equipment 106.
At least one aspect of one or more embodiment of the present invention provides the real border (MMR) of blending agent system and the method that is associated.MMR of the present invention system is provided for forming the mechanism of mixed media document, and said mixed media document comprises such as as the print paper of first medium with as the text in electrical form or the two types such medium of data, digital picture, digital photo, digital movie, digital audio file or network linking of second medium at least.In addition, MMR of the present invention system makes business method convenient, and the such portable electron device of said business method utilization such as portable video or audio player and the combination of paper document.MMR of the present invention system comprises MMR processor, acquisition equipment, communication mechanism and comprises the storer of MMR software.Can also the MMR processor be connected to storer or source, input media and the output unit of media type.In order to visit the MMR document of other digital content, to create or change MMR document, and use document to carry out such as other such operations such as business transaction, data query and report, MMR software comprises can be by the routine of MMR processor execution.The MMR system comprises the MMR massaging device of the MMR information that storage is downloaded, and said MMR information relates to the MMR document that is used in response to inputing to the retrieval MMR massaging device or that select from user that acquisition equipment provided.
The real border (MMR) of blending agent system be provided for forming the medium that comprises at least two types (as, as the print paper of first medium with as the digital content and/or the network linking of second medium) the mechanism of mixed media document.The MMR system is provided for forming and comprises such as as the print paper of first medium with as the mechanism of the mixed media document of at least two types such medium of digital photo, digital movie, digital audio file or the network linking of second medium.In addition, the MMR system makes business method convenient, and the such portable electron device of said business method utilization such as portable video player or portable audio player provides information with the combination that is used for the paper document of portable electron device.
Be described below another embodiment.
Voice mail
Figure 69 illustrates the block diagram of the MMR system 6900 that comprises the voice-mail server 6902 of handling hot information.
The hot information that voice-mail server 6902 will be referred to MMR document 500 provides to the user with the form of voice mail.MMR system 100 can provide the voice mail content to voice-mail server 6902.The user can also provide the voice mail content to upgrade hot information.In the embodiment shown in Figure 69 to 74, MMR system 100 can be 100a of MMR system or 110b.
In the embodiment of illustrative, the user can stay relate to such as the so special item in focus of restaurant, museum or film, such as praising or complaining such note.MMR system 6900 can collect the expense that in voice mail, stays message to the user.
In one embodiment, MMR document 500 can be for focus link being provided to voice mail but is not comprised the personal advertisement of telephone number.This allows the user not issue their telephone number, places personal advertisement, but allows answer person to stay the ability of voice mail.
In another embodiment, MMR document 500 can comprise the advertisement that is used to place an order, focus is arranged.The user stays the details of ordering and places the entity that can retrieve the advertisement that is used to handle the message of ordering in voice mail.
Figure 70 illustrates the process flow diagram of the method that is updated in the hot information in the MMR system 6900.
The user of the focus of MMR system 6900 receptions 7002 on MMR document 500 selects.MMR system 6900 receives 7004 users request, upgrades from the hot information of acquisition equipment 106 to voice-mail server 6902.MMR system 6900 connects 7006 to voice-mail server 6902, and points out 7008 users in voice-mail server 6902, to stay voice mail.Through using focus and prompting, the user need not dial the number or Voice mail Number just can the access voice mailbox.MMR system 6900 can point out the user that voice mail is classified.For example, for consulting of film, input code on acquisition equipment 106 can be pointed out the user by MMR system 6900, indicates whether said consulting is that be satisfied with, unsatisfied or general.
Figure 71 is illustrated in the process flow diagram of the method for retrieve hotspot information in the MMR system 6900.
The user search of MMR system 100 control voice mail hot informations.The user's of the focus of MMR system 100 receptions 7102 on MMR document 500 selection.MMR system 100 provides the menu of 7104 voice mail focuses for acquisition equipment 106.Can menu be presented on the acquisition equipment 106, perhaps said menu can be audio menu.Menu can comprise such as satisfied note classification such, voice mail.MMR system 100 retrievals, 7106 voice mails corresponding to focus (it is in response to the selection from one user in the focus of menu), and, provide 7108 to acquisition equipment 106 with voice mail for the user hears.In alternative embodiment, MMR system 6900 select to provide to subscriber voicemail.
As the example of illustrative, the user can be read as the MMR document 500 of the newspaper tabulation of film.The user can select the film focus with acquisition equipment 106, and asks consulting of film, and can it be provided as speech message.Hear consult after, said film is watched in user decision.After film, the user wants to upgrade hot information through staying voice mail.Though the user reuses the newspaper tabulation of film and visits focus, and is current, the user sends focus and upgrades request, and stays the comment of film.
Email
Figure 72 illustrates the block diagram of the MMR system 7200 that comprises the email message server 7204 of handling hot information.
For the search in response to the inquiry of using traditional research tool, MMR system 100 can index to Email.
As the example of illustrative, the user is just reading guide book, and the request more information.The user selects focus to receive hot information.If the size of hot information is big, then the user selects to receive hot information with Email, and the user will be after a while or on the computing machine by giant-screen more, read, or document is printed reading.
Figure 73 is illustrated in the process flow diagram of the method for retrieve hotspot information in the MMR system 7200.
The user's of the focus of MMR system 7200 receptions 7302 in MMR document 500 selection.MMR system 7200 connects 7304 to email message server 7204, and points out 7306 users to come to select from email message server 7204 destination of Email.Through the use of prompting, the user can guide to acquisition equipment 106 or subscriber computer 7206 with Email.If the user has selected Email is sent to subscriber computer 7206 or is different from the destination of acquisition equipment 106, then MMR system 7200 sends 7308 to the address of being asked with Email, and will confirm that Email transmission 7310 is to acquisition equipment 106.
In one embodiment, MMR system 7200 is provided with high priority flag on Email.
In one embodiment, can acquisition equipment 106 be broken off from MMR system 100 provisionally.At off period, acquisition equipment 106 storage is selected for the focus of after a while transmission.When acquisition equipment 106 is connected to MMR system 100 again, acquisition equipment 106 will transfer to MMR system 100 for the request of handling.
Figure 74 illustrates the block diagram of the MMR system that comprises document database 7404.Though MMR system 7400 is similar with MMR system 7200, it further comprises document database 7404.The MMR document 500 that document database 7404 storages are read by the user, and storage is by the hot information of user's selection.Can come to index by the user, and it can be what can search for for the user to document database 7404.Document database 7404 is stored reading for the user.
Figure 75 comprises that document transports the block diagram of the MMR system 7500 of database 7504.
Though MMR system 7500 is similar with MMR system 7200, it comprises that document transports database 7504.Document transports the seller's of database 7504 storage productss or service MMR information.In said embodiment, MMR system 100 provides the user capture of transporting database 7504 to document.Document database can be the tourism publisher, and it provides the database of the guide book that whole MMR enable.The user asks MMR system 100 to come to transport database 7504 search files from document, and the said document that can send asynchronously provides to the user.The document supplier is sent to the user with hot information through Email.
In one embodiment, document transports database 7504 for through the Internet or through the internet search engine inaccessible.Through MMR system 100, visit document and transport database 7504.
In another embodiment, document can use unique electronic product code (EPC) or similar code to discern uniquely each page, transports in the database 7504 with the identification document and at document and finds document.
In another embodiment, use whole pages or its part of document printing, create and be used for identifier document printing, unique.Can unique identifier be transferred to document and transport database 7504, with the image of the hot information that replaces being used to confirming being asked.
In alternative embodiment, MMR system 7200,7400 and 7500 further comprise handle except that Email, as the voice-mail server 6902 of the hot information of voice mail.
MMR of the present invention system is provided for forming the mechanism of mixed media document; And said document comprises such as the print paper as first medium, with text in electrical form or at least two types such medium of data, digital picture, digital photo, digital movie, digital audio file or network linking as second medium.In addition, MMR of the present invention system makes business method convenient, and the combining of the such portable electron device of said business method utilization such as camera cell phone and paper document.MMR of the present invention system comprises MMR processor, acquisition equipment, communication mechanism and comprises the storer of MMR software.Can also the MMR processor be connected to storer or source, input media and the output unit of media type.MMR software comprises the routine that can be carried out by the MMR processor, and said routine is used to visit the MMR document of other digital content, creates or change MMR document, and uses document to carry out such as other such operations such as business transaction, data query, reports.For the information that relates to document printing is provided, the MMR system comprises voice-mail server or the e-mail server that uses voice mail or Email respectively.
The real border (MMR) of blending agent system comprises MMR user; The MMR computing machine; Produce user's printer of document printing; The network medium server; The office inlet; ISP's server; Be electrically connected to the electronic console of STB; Document scanner; Network; Acquisition equipment; Cellular infrastructure; Wireless Fidelity (Wi-Fi) technology;
technology; Infrared ray (IR) technology; Cable technology; With geographic position mechanism.The MMR system is provided for forming the mechanism of mixed media document, and said mixed media document comprise such as as the print paper of first medium with as at least two types such medium of digital photo, digital movie, DAB or the network linking of second medium.In addition, the MMR system makes business method convenient, and said business method utilizes the combination of portable electron device, voice mail or Email and paper document.
Be described below another embodiment.
Figure 76 illustrates MMR according to the present invention functional block diagram system, that be used for coming at the document that uses the MMR personalization embodiment of order goods or service use.Purchase and handle commodity or service in order to use the personalized commercial document printing 7601 of MMR to decide through consultation, dispose MMR system 7600 from honouring an agreement.Though Figure 76 illustrates with MMR system 7600 and uses the example of MMR individualized document 7601 with order goods or service, notice that MMR individualized document 7601 can be used for the business transaction of any other type.Except that MMR system 7600 also comprise be used for the use of the personalized document printing 7601 of MMR honour an agreement merchant's software 7602 with order the process database (DB) 7604, said MMR system 7600 is fully identical with the 100a of MMR system that in Figure 1A, is illustrated.
The personalized document printing 7601 of MMR be according to as, for example, the MMR document that the MMR system and method for being illustrated among Figure 1A, the 1B, 4 and 52 is printed.MMR document 7601 comprise the user that discerns individual or document 7601 uniquely, one or more focuses or the part of document and other focuses of identification document 7601.With as similar in these described other MMR documents, index to focus, and it be stored in the PD index 322 (or being equal to ground MMR database 3400).In one embodiment; MMR document 7601 is for being used for the printing catalogue of order goods or service (books, groceries, traveling bag or family's maintenance service etc.); And focus text or the image and the identification that comprise identification document (catalogue) be associated with catalogue, one or more users or individual focus.
Comprise under that identification is associated with document, one or more users or individual meaning text or image identifier, at focus MMR document 7601 " personalization ".For example, document 7601 can comprise the recipient's of document 7601 name and address, and can it be recognized as the focus of the identification that is used for the user.For another example, document 7601 can comprise bar code or other identification numbers, maybe can be recognized as the image of the focus of the identification that is used for the user.In special realization, customer identification information can be rendered as the mailing label on document.Can discern the user by the OCR (optical character identification) of label or the identification of postal delivery bar code (being known as the POSTNET bar code in the U.S.).The mailing label can also comprise bar code or other information (like, print text), the particular version of its identification catalogue, as, for the catalogue of Victoria Secrets San Francisco Gulf Area, in November, 2006.Thereby the mailing label comprises the information that is identified in the personal user in the database 7622, and catalogue and catalogue itself are sent to said user.The tabulation of the identifying information of retrieval user and focus is sufficient in catalogue.Hot spot data will comprise for the identification of the project in catalogue, they present the page on it and need order the information of project.User interface on acquisition equipment 106 can use said information that the ability of order goods and service is provided.Alternately, hot information can comprise the information of relevant characteristics of image, and said information corresponding to can be used in subsequently, through each focus in the comparison step of MMR recognition system.
For another example; Though document 7601 does not comprise text or the image identification, that separate that is used for the user; But it is included in variant in appearing of the interior urtext of document self, less, as, the shape of text and image, font size, orientation, color etc.; And at first said text and image are included in the document 7601, and can it be recognized as the focus of the identification that is used for the user.The description of example for will on catalogue, having project; Be included in the less variant in outward appearance, shape, font size, orientation, the color etc.; So that the difference in the MMR document 7601 that the user can easily not recognize in personalization; But focus can be mated through the fingerprint with document in MMR system 7600, with the identification user.In an exemplary realization,, be associated with customer identification information in database 7622 like, the combination of font and color with picture characteristics.For example, can catalogue be printed with the OCR-A font that blue #F86A ink is arranged with focus.The combination of that font-color is assigned to the user in database 7622, and corresponding catalogue is sent to said user.Each image that the pre-service of MMR identification software combines for font-color, and the identity of font and color is sent to orders process database 7604, and the said process database use font-color of ordering combines to search customer identification information.Can picture characteristics, unique combination be assigned to each user.This has the advantage that each focus is discerned recipient catalogue, that have a mind to uniquely.Can identical performance be provided through little amount is moved in the position flatly or vertically of character or word.Amount of movement can with the numerical information in text (as, User Recognition and focus) coding so that it is imperceptible for almost for the reader, but decode by recognizer easily.In alternative realization, select the combination of picture characteristics from the group that is limited, and it is assigned to each user in database 7622.The MMR identification software is also recognized the existence of those picture characteristics, and with their identity be sent to order process database (as, it can find OCR-A font or character in input picture, that confirm or the position of word in blue #F86A to move).Yet; It also sends less important customer identification information from MMR device 106; For example from the owner of determined that register, the device of the profile of being stored on device; Its telephone number, or the owner of the institute's device registered as other suppliers by the relevant service of cellular service supplier 5318, ISP 5336, wired TV supplier 5334 or MMR provide.Order process database 7604 and verify that those picture characteristics of being recognized by the MMR recognition system meet less important customer identification information.
The document fingerprint matching module 226 of acquisition equipment 106 (or alternately on MMR computing machine 112) is extracted the focus characteristic from the hotspot graph picture of being caught.As further described with reference to figure 6 to 33; The document index that 226 visits of document fingerprint matching module are stored in document event database 320; And with the characteristic of being extracted of the image of being caught and index coupling; Come from special personalized document 7601 and user who is associated with document or the individual of MMR of the database identification of document.In other words, confirm document (catalogue) 7601 and the user's that is associated with document identity.
The merchant's software 7602 of honouring an agreement comprises the software application that computing machine moved by the merchant 5332 that honours an agreement.For example, the personalized document 7601 of MMR is a backlist, and the bookstore of merchant 5332 for using backlist to sell books that honour an agreement.The merchant's software 7602 of honouring an agreement comprises orders process software 7610 and calendar synchronization software 7612.
In case identification document and the user who is associated are ordered process software 7610 with regard to calling party information database 7622, retrieve the user profile that is associated with the user who is discerned.User profile can comprise the user name, address, telephone number, credit card information and any can be with its other information that are associated with the personal user.
In case accomplish and to order, just as with any online ordering of commodity or service order for handling purchase so typically.Typically, will be stored in and transport in the information database 7624 such as the desired or actual such process information of ordering of data and desired or actual payment data that transports.Be stored in transporting in the information database 7624 of being associated with the user who is discerned with ordering process information.Information database 7624 is transported in 7612 visits of calendar synchronization software, and process information is ordered in retrieval.Calendar synchronization software 7612 impels orders process information, and it is added into the calendar applications 228 of operation on acquisition equipment 106.If order process information (such as desired transport or desired payment data such) in any change is arranged, then calendar synchronization software 7612 impels calendar applications 228 to upgrade, and also reflects said change.In such mode, user's his processing of ordering of real-time follow-up and transporting effectively.
Figure 77 illustrates and is used to use the personalized document of MMR to come the process flow diagram of order goods or service method 7700, for example with the user's of identification document and document hotspot graph as embedded product or service catalogue.With reference to Figure 76 and 77 both, acquisition equipment 106 is caught the hotspot graph picture of the personalized document 7601 of 7702MMR.Use the hotspot graph picture, document that 7600 identifications 7704 of MMR system are special and the user who is associated with document.Order process software 7610 comes increase in advance with user profile the form of ordering; Come remotely to impel the form of ordering for commodity or service; So that it is shown 7706 on acquisition equipment 106 through communication mechanism 104, and the subscriber identity information that is based in the step 7704 to be obtained comes retrieve user information.
The user fills in and orders form and accomplish 7708 order transaction.The processing that 7612 tracking 7710 of calendar synchronization software are ordered (as; Estimated and Shipping Date reality; Or estimated and date of payment reality); And with the said calendar applications 228 of handling the date interpolation or being updated to operation on the user's who is discerned acquisition equipment 106, so that with being stored in the information of process information synchronous 7712 in calendar applications 228 of ordering of transporting among the information D B 7624.
Though Figure 76 and 77 is illustrated in the use of the document of the MMR personalization in the scene of buying commodity or service, can also use the personalized document of MMR for other purposes.For example, want to control the information that the publisher of distribution illegal, that obtain the material of copyright can embed the user who is authorized (like, shorthand code), and said user suitably buy in the MMR document of being printed, obtain the material of copyright.Can use embedded user profile, discern the user who is authorized, and thereby will allow publisher to follow the tracks of the document that how to use them, and potentially identification distribute or duplicate the material that obtains copyright without approval and unlawfully those people.
In one embodiment of the invention, look like to embed the real border of blending agent document with hotspot graph, and the said hotspot graph picture user that is associated with document and the focus of discerning document of identification uniquely.Use said " MMR is personalized " document to discern the user who is associated with document, and the user profile that is associated with user's identity of retrieval.
An application program of MMR individualized document is for being used for order goods or service.Can use focus to discern the user and also retrieve the user profile that is associated with the user who is discerned.Can come to be increased in advance the online ordering form that shows on user's the acquisition equipment with the user profile of being retrieved.In addition, can follow the tracks of the process information of ordering, and can be updated in the calendar applications that moves on the user's who is discerned the acquisition equipment, to comprise corresponding to that order, up-to-date process information.
In a particular embodiment, mixed media document can also comprise the user's that identification uniquely is associated with mixed media document hotspot graph picture.When using mixed media document to do through the MMR system, use the mixed media document of said personalization to discern the mixed media document user for the ordering of product or service.
Be described below another embodiment.
The MMR general-purpose flip-flop of action
Mentioned with reference to figure 5 like preceding text, except that expression 502, index or the focus 506 of the part of document printing 118 with all the electronic representation 508 of documents 118, the MMR document 500 in the embodiment that confirms comprises the associated action or second medium 504.Because preceding text have fully been described these parts 502,504,506 and 508 of MMR document 500 with reference to figure 5, therefore simple and easy for what understand, will not repeat said description.Because the ability that combines with the ability of creating many MMR documents 500 for the special position on any medium, specify different actions for different MMR documents 500 allows the MMR architecture to be used as general-purpose flip-flop or the start-up routine that is used for other processing, so the structure of MMR document 500 is particularly advantageous.In other words, can discern based on MMR and trigger or start other processing or action.Can also advantageously customize other processing to consider: 1) discern documents identified by MMR; 2) page of discerning by MMR; 3) position of discerning by MMR on the page; 4) comprise the scene of use of the acquisition equipment 106 in its geographic position; 5) be known for the MMR system or be stored in the external factor in the database 3400; 6) carry out by the history of previous user's action; 7) state of MMR system unit or availability; 7) third party system and state of resources or availability; With 8) active user's identity; With 9) identity and the position of MMR database that inquired and that satisfy given inquiry.
With reference now to Figure 78,, an embodiment of the MMR system architecture 7800 that is used as general-purpose flip-flop is shown.The description of this architecture is provided as wherein an example of scene that MMR identification is used as of the present invention, the operation of general-purpose flip-flop, and will transmits as general-purpose flip-flop for the General Principle of the operation of MMR.Those skilled in the art will recognize that and in other MMR architectures of multiple description from subsequently and configuration, MMR identification to be used as general-purpose flip-flop.
Shown in Figure 78, exemplary MMR system architecture 7800 comprises the document 118, user 110, acquisition equipment 106, database 3400, MMR identification module or the software 226 that are appeared or 610 and action processor 7802.
Preceding text have been described the document that appeared 118 and user 110, and its with this exemplary scene in identical.The given document that appears 118.This can be the paper printout, and perhaps it can be for like the version such, that electricity appears shown on PC screen or e-book.The user uses acquisition equipment 106 mutual with MMR system 7800, and acquisition equipment 106 is as producing images by next the dotted line among Figure 78 is represented from the document 118 that is appeared.Preceding text are fully described the numerous example for acquisition equipment 106, and can wherein any one be used for this example.Simple and easy for what understand, acquisition equipment 106 is for having the phone of camera.For respectively through signal wire 7808 with 7804 with the communicating by letter of MMR identification module 226/610 and action processor 7802, connect acquisition equipment 106.
Through signal wire 7810 action processor 7802 is connected to MMR identification software 226/610, is connected to acquisition equipment 106, and be connected to database 3400 through signal wire 7814 through signal wire 7804.Action processor 7802 has preceding text, and described and hereinafter will be with reference to the described function of figure 80-85 with reference to action processor 3413.Action processor 7802 is connected to acquisition equipment 106; Send and receive the data and the order of the generation that is used for user interface; The data that reception is produced by action; And the needed data of input action, and said user interface allows user 110 how to carry out action and mutual with MMR system 7800 in order to control.Since increase said user interface from moving the information of standard.Action norm is for needs it to be accomplished to carry out subdivision or the one or more orders of task and the parameter that is associated of action aptly.User 110 can import the data of the execution that enables to move.The result's of relevant said action information stores in MMR database 3400, and is presented to user 110 with it.In one embodiment, 7800 storages of MMR system are such as historical information such, relevant action when formerly it carrying out with the x-y position in the piece at text.
Because can pass through acquisition equipment 106, camera cell phone points to the piece of text and carries out action, so MMR system architecture 7800 is favourable.For example, the link of a part of the text in the network version of said document is manually clicked in action, thereby will be with text retrieval to the acquisition equipment 106 of the form of XML form.In another example, move a standard part, that be used to move of clickthrough manually, thus the retrieval standard that is used to move to retrieve the text in the network version of said document.In addition, because action processor 7802 is stored in the historical information of relevant action in database 3400 or other memory storages, so said action processor 7802 is favourable.Historical information about action is stored in x-y position and the document and the page in the piece of text related.Then, can in the incident of the piece of catching identical text, retrieve, search for and analyze said information after a while.
With reference now to Figure 79,, the part of embodiment execution, MMR database 3400 who relates to action is described in further detail.Also mentioned like preceding text with reference to figure 34A, MMR database 3400 storage file and pictures 7902, action 7904 and in file and picture and action, in other words, association 7905 between the x-y position on the MMR document 500.Because MMR database 3400 is what can dynamically update, so user application can dynamically add action.MMR database 3400 allows to move adhering to of the position of x-y arbitrarily within text fragment.Clauses and subclauses in MMR database 3400 comprise the page that links to action." page side " of link is registered to specific x-y position." action side " can comprise data or mutual, almost describe arbitrarily.MMR database 3400 comprises the MMR Feature Mapping at concordance list 7908 within the document, on their the x-y position 7906.This allows MMR identification software 226/610 to import the characteristic that it is surveyed in image, and receives the output of the x-y position within the document that is illustrated in wherein those characteristics appearance.Preceding text have fully been described the embodiment of said data structure with reference to figure 34A.Also As mentioned above, MMR system 7800 can also comprise a plurality of database 3400a and 3400b.For example, MMR system 7800 comprises at least two databases: shared data bank and individual database.Shared data bank comprises that to having to its whole users of access right all be addressable information.Individual database is addressable to the personal user, and comprises the personal information of the x-y position on the page that is linked in document, and the owner does not hope to share said document with other people.It will be understood by those skilled in the art that to have other databases of on basis between user's the group, specific, sharing.Though not shown, when MMR database 3400 usually was depicted as the individual data storehouse, these databases can be the part of identical database, or the database that separates fully.
Figure 80 is the process flow diagram that is used to carry out according to an embodiment of the method for the action of MMR document 500 of the present invention.Through obtain the image of 8002 documents 118 that appeared with acquisition equipment 106, process begins.When now with process prescription when beginning with catching of special image; It will be appreciated by those skilled in the art that; Process can use acquisition equipment 106 or any information of another device input to begin with user 110, and catching as just example of image provided.Then, MMR system 7800 carries out 8004 identifications in the input that in step 8002, is received by system 7800.Secondly, MMR system 7800 confirms whether the 8006MMR identifying has returned the coupling document, and whether the coupling document comprises the action that is performed.MMR system 7800 must discern user 110 and specify the action of carrying out, so that MMR system 7800 carries out action.This require MMR system 7800 identification documents and within said document or with its associated action.If method is confirmed to find the coupling document for the image of in step 8002, catching in step 8006; Then system 7800 produces the message that 8008 indications can not be found coupling; And it is shown to user 110, for example on the display of image capture apparatus 106.Similarly, do not have and said coupling document associated action when the coupling document if method is definite in step 8006, then system 7800 produces 8008 and indicates the message that does not have with coupling document associated action, and it is shown to user 110.The step 8008 that those skilled in the art will recognize that said message is for optional.In the completion of step 8008, what method was circulated back to method begins to catch other image.On the other hand, if the MMR identifying with will be performed, associated action produces MMR document 500, then method is proceeded from step 8006 to step 8010.In step 8010, method determines whether automatically carry out action.In one embodiment, each action that is stored in the MMR system 7800 can have many settings that are associated.Whether said being provided with of being associated can be for importing with other user, or " automatically " carried out action.Method confirms whether 8010 be provided with the setting for the said action that will be performed in order to carry out automatically.If will automatically carry out action, then method directly continues to step 8018 from step 8010.Yet if will automatically not carry out action, method proceeds to step 8012.Then, MMR system 7800 produces message and is presented on the acquisition equipment 106, and notifies 8012 users just to carry out action by MMR system 7800.Method receives 8014 information from the user.If not other input, the then user of the execution of a message notification action to customer requirements.In some instances, the execution through MMR system 7800, special action can require from user 110, other input for example affirmation.Under those situations, for to the appearing of user 110, MMR system 7800 is sent to acquisition equipment 106 with message, and acquisition equipment 106 receives information MMR system 7800, other that is input to from user 110.Said information can be the identification or the selection of the different parameters of the execution that is used to move, or the input of the needed data of execution of action.Preceding text are described in mutual between user 110 and the MMR system 7800, and can alternatively use said process to allow user's 110 inputs to use needed, the other information of execution of the action of GUI of the present invention.For to the appearing of user,, just will ask user's affirmation should carry out message action, final and be sent to acquisition equipment 106 by system 7800 in case user 110 has accomplished mutual with MMR system 7800.
Secondly, method confirms whether 8016 users 110 have confirmed carry out action.If user 110 has selected not carry out action, then method also finishes for what accomplish.If the user has selected to carry out action, then method continues to step 8018, and confirms 8018 actions by 500 appointments of MMR document.Then, method is carried out 8020 actions by 500 appointments of MMR document.Mentioned like preceding text, MMR document 500 can comprise one or more actions, and each has one or more orders and parameter.Hereinafter will be described these orders and examples of parameters with reference to figure 82A and 82B.MMR document 500 can comprise the necessary one or more parameters that are implemented as for order, perhaps alternately, can directly come input parameter by user 110, and in step 8014, receive said parameter through said method.What should be noted that is that MMR document 500 can comprise several orders.As should pay attention to by those skilled in the art, this provides wherein can carry out the complicated transaction and the MMR programming language of user interactions by MMR system 7800, and MMR system 7800 has only for the MMR of trigger of execution to them and discerns.Hereinafter is described the transaction of said complicacy and the example of user interactions.In case the execution of execution, MMR system 7800 just stores the historical information of 8022 relevant actions.For example, the historical information about action can comprise: carry out action user, the moment when carrying out action, be used for carrying out the MMR system unit of action, wherein carry out action scene (geographic position that comprises acquisition equipment 106), be utilized to carry out the external resource (comprising commercial entity 5300) of action, for the execution trace of the action that comprises performed order, also have time, date, parameter and their completion status (the various degree that comprise the success of success, failure and part) for each order.In case historical information is stored in the MMR system 7800, method just is what accomplish in process finishes.
Of the present invention, noticeable aspect is its use with text fragment or text block.Two general methods using MMR architecture 7800 to create action-trigger based on text fragment are arranged.The first, user 110 can point to acquisition equipment 106 paragraph of the text in paper document 118, and retrieves and carry out the action that before had been linked to said text fragment.Second; User 110 can create such as the such action of the acquisition equipment 106 of use resemble the mobile phone, and the paragraph through acquisition equipment 106 being pointed to the text in the paper version of being printed of electronic document is with the paragraph of action link to the text in electronic document.Action can be for static state, and wherein action processor 7802 uses input parameters to carry out the order of appointment, perhaps it can be for dynamic and comprise ordering its user who adopts a series of measures 110, a series of order, and the result be the unknown.For example, can with have the result that selects and be attached to each selection, one group of menu that is linked is presented to user 110.Acquisition equipment 106 is pointed to documents to be meaned by MMR system 7800 and recognizes the image that acquisition equipment 106 obtains.This is identified in the x-y position within the page in the document, and just acquisition equipment 106 is pointed to said document.What should be noted that is the text that is not included in the document, so that the identification document.MMR identification is derived characteristic from the configuration of text, and when fully describe said text about the fragments matching preceding text, said text permission we with its be identified as on printer page, its edge effect of two dimensional topology.This discerns from bar code recognition or sequence number and distinguishes MMR identification, comprises expressly that wherein bar code or number discern document.Thereby, because for the MMR system can discern and recognize the position in document, do not add or require mark document printing, additional, so the present invention is particularly advantageous.
Another noticeable aspect of the present invention provides action-trigger for giving non-file and picture and text document.This is similar in appearance to augmented reality, and wherein it is with image of being caught and identical scene, version combination numeral or electronics.With MMR architecture 7800; Can discern whole scenes from the image of subclass; Perhaps can discern scene from some sufficient or important subclass (visually clearly or layout unique, subclass); And said subclass and the identification of MMR wherein is from any text fragment but from the document of white space, or the mode of the image within document is similar.For example, from the image of automobile, MMR architecture 7800 can be discerned unique automobile from licence plate or VIN (Vehicle Identification Number, VIN) label.Text document is considered analogy, and this will be as from being attached to the bar code recognition document of said document.MMR architecture 7800 is also discerned unique automobile from the image of the treadwear decorative pattern of the tire of automobile.In addition, an application program is identification image, said people from the people.The MMR of non-document input analyzes and can comprise video and audio frequency.Can by MMR architecture 7800 from the image of their finger tip, they face image, from they hair DNA, they tooth configuration image and when the walking their footwork video etc. relatively discern the people.MMR architecture 7800 can be assigned to the action of personalization specific finger.For example, when RIF was formed images, you point the telephone number of dialling with that root tabulation can be retrieved by system 7800.Similarly, for music and other sound, can use one group of notes can discern song, identical mode from the sequence of notes and discern MMR document 500 with the audience.
Through create or provide with the image of object with at those to the database 3400 that the x-y position within the object that moves is associated, be used to carry out the method that MMR by the image of non-document discerns the action that triggers and begin.Alternately for music or sound, database 3400 is associated to action with the waveform and the time location in those waveforms of sound.Database 3400 under two situations is what can dynamically update, and can add action in real time.Secondly, the acquisition equipment 106 that comprises storer and optional communication channel is provided.Use acquisition equipment 106 to catch still image, video clipping or recording.Analyze the object of being caught at acquisition equipment 106 or MMR processor 102 places.Secondly, identification MMR object and the position in object.For single image, it is the position within said image, and for video clipping, it is the subclass of frame, and for recording, it is sampling.For the MM object of coupling, come retrieval actions with the position of coupling.Finally, carrying out action on the image capture apparatus 106 or on MMR processor 102.Execution can be synchronous with being captured as of view data, and perhaps it can be for asynchronous.The example that synchronous action is carried out will be for from retrievals guide-book, museum's business hours.After the user is back to office and uploads the data to their PC, the example of asynchronous execution will be adhering to the image of meeting guide.
With reference now to Figure 81,, an embodiment of the method be used to check relevant historical information according to action of the present invention is described.Through obtaining 8002 images, and carry out 8004MMR and discern to confirm document and position that method begins with acquisition equipment 106.Preceding text are described these steps with reference to other images Figure 80 and preceding text, multiple, for this reason, will not repeat said description here.Secondly, method is for the relevant information that relates to the action of position, and searches for 8106 databases 3400.Identifying is output as document and the position within said document.Said output is used as the generation inquiry comes the basis of search database 3400.Mentioned that like preceding text database 3400 comprises about when carrying out the historical data with special MMR document action that be associated, special.In addition, database 3400 is stored the time such as the execution action, the number of times of execution action, the user that execution is moved and such other information or the metadata of resource that is used for carrying out action, but is not limited thereto.Then, can analyze the result who is used in step 8106 inquiry that carry out, database 3400, produce conclusion historical information, that confirm about the relevant action of storage.In the instance of confirming, can be sent to acquisition equipment 106 with the result that action processor 7802 only will be inquired about for demonstration, omit said step.Finally, in step 8110, MMR system 7800 is presented to user 110 with the relevant historical information that relates to the action of position, and method also finishes for what accomplish.
With reference now to Figure 82 A and 82B,, an embodiment of the method that is used to carry out the order that is associated with action according to the present invention is shown.Preceding text describe one group of exemplary order and parameter in detail with reference to figure 34A to 36.Have similar function with reference to the described order of figure 82A and 82B here, and in more general mode, describe said order for this reason.In said embodiment, action is specified and to be comprised for document, in the order and the parameter of the identifier of the page within the document and the x-y position within the page.Specified parameter depends on special order, and each parameter needn't all be relevant with each order.Usually, through the type of the action confirming in step 8202,8204,8206,8208,8210,8212,8250 and 8252, just be performed, be used to carry out and move the method for the order that is associated and begin; Carrying out said order then is special step.
Through confirming whether 8202 actions are " RETRIEVE " order, and method begins." RETRIEVE " order is retrieved the data that are linked to the x-y position of being recognized with document.Said data can be included in the menu of specifying step that will be performed, subsequently on the acquisition equipment 106.It can also be for being displayed on the static data on the acquisition equipment 106.If like this, then method is from MMR document 500 or other source retrieval 8214 data.In one embodiment, retrieve data is stored as the part of MMR document 500.In another embodiment, retrieve data is stored in the database 3400, and MMR document 500 is included in the reference of the position within the MMR database 3400 of storing therein data.In another embodiment, retrieve data is stored in position outside the MMR system 7800, long-range, and action processor 7802 execution necessary steps come from long-range position searching data.For example; Long-range position can be for being stored in the webserver outside the MMR system 7800; Or the locational webpage within another private system (not shown); And said another private system allow through connect such as network, VPN (Virtual Pfivate Network, VPN) connects or such, the traditional connection mechanism of secure network interface and with the interconnection of MMR system 7800.Action processor 7802 is carried out necessary step and is presented menu, receives data and mutual with user 110.Then, for to the appearing of user 110, method sends 8216 to acquisition equipment 106 with institute's data retrieved.After step 8216, method is converted to hereinafter with the step of describing 8242.
If method is confirmed action in step 8202 be not " RETRIEVE " order, then method confirms whether 8204 actions are
and order.
order is inserted in data by the specified x-y position of image fragment.The meeting guide is for wherein the user will be such as being added at the example for the x-y position within the electronics original text of paper document at the such image file of jpeg file of being caught on the acquisition equipment 106.If like this, then method identification 8218 is used for the data of order.For example, the text that can be use acquisition equipment 106 to import the data that are inserted into, or the image or the video clipping that obtain by acquisition equipment.Then, method identification 8220 should locate to insert the position of the data that received at it.It should be noted that the position of inserting the data received at its place can be within MMR document 500, such position in such as the database in MMR system 7,800 3400, or in the long-range position of the system that is connected with MMR system 7800.Then, the position of data insertion 8222 being discerned in step 8220.After step 8222, method is converted to the described step 8242 of hereinafter.
If method confirms that in step 8204 action is not
and orders, then method confirms whether 8204 actions are
and order.
order data of the determined x-y of free image fragment position in the future is inserted in another document.For example, the user can select text phrases in printout, and they are added into the PowerPoint file.If action is order for
, then method identification 8218 is used for the Data Source of order.As in the described searching step of preceding text, Data Source can be in MMR document 500, the outside of position within MMR system 7800 or MMR system 7800.In case recognition data is originated, said step just can also comprise the step of retrieval institute recognition data.This is similar with step 8214 discussed above.After step 8224, method proceeds to step 8220 and 8222, is identified in its place respectively and will inserts the position of data and carry out insertion.Moreover, will insert at its place data the position can within this or another MMR document 500, in the position within the MMR system 7800 or in the intrasystem long-range position that is connected with MMR 7800.After step 8222, method is converted to the described step 8242 of hereinafter.
If method confirms that in step 8206 action is not
and orders, then method confirms whether 8208 actions are " TRANSFER " order.The data that " TRANSFER " order will link to the x-y position are passed to the destination from the source.Source or destination can both be the MMR client apparatus.Example is used for the PDF of given paper document and its document that is passed to acquisition equipment 106 is seized the device application program for wherein retrieving.If action is " TRANSFER " order, method identification 8226 data and the source that are used to transmit then, and discern 8228 destinations then for data.These identification steps 8226 are similar with 8224 with described other identification steps 8218,8220 of preceding text with 8228.Then, method sends 8230 to the destination with data.Can be with such as through server to server communication, through file transfer protocol or through any transmission of carrying out the data in step 8230 in the such multiple mode of Email, but be not limited thereto.After step 8230, method is converted to the described step 8242 of hereinafter.
If method is confirmed action in step 8208 be not " TRANSFER " order, then method confirms whether 8210 actions are " PURCHASE " order." PURCHASE " command execution links to the purchase of the commodity of the x-y position in document.If action is " PURCHASE " order, the project just bought of method identification 8226 then from the seller.In one embodiment, this step can not be necessary, wherein specifies buyer and project and it is encoded to the parameter of action.In other words, predefine is moved from the specified seller and is bought specified project.Yet if the user is just using acquisition equipment 106 that commodity and the seller are appointed as mutual part must execution in step 5032, he and MMR system 7800, this is increased in the number of the different command in the MMR system 7800.Then, method is bought commodity from the seller, for example uses online transaction through the WWW.Those skilled in the art will recognize that MMR system 7800 can have with such as bank, cellular services supplier and courier such, being connected of various, existing entity, and can be connected with said physical system to come combine online vending machine system and traditional retail mechanism, accomplish transaction in many modes.In case complete transaction, method just proceed to the described step 8242 of hereinafter.
If method confirms that in step 8210 action is not " PURCHASE " order, then method confirms whether 8212 actions are " AUTHOR " order." AUTHOR " order is created new action and it is added into by the specified x-y position of image fragment.For example, the user can create sound annotations, and with its be added within document, x-y position arbitrarily.If action is " AUTHOR " order, then method is discerned 8228 present MMR documents 500.If the user is just creating new MMR document, then replace the step of this identification by the step of creating MMR document 500.Then, MMR system 7800 uses from user 110 through the input that acquisition equipment 106 receives, and discerns new action and is used for the parameter of said new action.When step is depicted as single step, those skilled in the art will recognize that this step can repeatedly comprise GUI to user 110 appear and from the reception user 110 of information, through acquisition equipment 106.Thereby MMR system 7800 provides to the user and carries out in real time and the change of special MMR document 500 associated action or the ability of establishment.No matter MMR document 500, is just created action and it is added into MMR document 500 in case discerned MMR document 500, action and parameter for existing MMR document 500 or of in said process, having created.In one embodiment, through new action and parameter are stored in the database 3400, and action link to the MMR document 500 that is stored in the database 3400 is accomplished this step.After step 8240, method is converted to the described step 8242 of hereinafter.
If method confirms that in step 8212 action is not " AUTHOR " order, then method confirms whether 8250 actions are " PRISTINE_PATCH " order." PRISTINE_PATCH " command search is for the electronic representation of specified document, and to extract with x-y be the center, the image fragment of predetermined radius is arranged.If action is " PRISTINE_PATCH " order, then method identification 8254MMR document 500.Because MMR document 500 be the electronic representation of document or to its reference, so will be used for the electronic representation of search file, and extract the 8256 image fragments that directly produce from electronic representation from the information of MMR document 500.After step 8256, method is converted to the described step 8242 of hereinafter.
If method is confirmed action in step 8250 be not " PRISTINE_PATCH " order, then method confirms whether 8252 actions are " ACCESS_DATABASE " order." ACCESS_DATABASE " order is changed to any given database with access control, allows user 110 to use his acquisition equipment 106 accessing databases.If action is " ACCESS_DATABASE " order, then the database 3400 of the visit on it is just being agreed in method identification 8259, and MMR system 7800 changes the visit that is used for institute recognition data storehouse then, so that the user can visit it.After step 8256, method is converted to the described step 8242 of hereinafter.
Finally, method confirms whether 8242 have action or the order that be associated, other with MMR document 500.Mentioned like preceding text, MMR document 500 can comprise many actions, order and parameter.As arriving the analogy of computer programming, can action be thought of as subroutine, can order be thought of as individual's instruction, and can parameter be thought of as variable or the data of being used by instruction.In step 8242, method determines whether to be useful on any other action or the other order of the MMR document 500 of coupling.If no, then method also finishes for what accomplish.On the other hand, if MMR document 500 comprises any other action or order, then method is retrieved 8244 Next Commands and its parameter, and is back to step 8202 and handles Next Command.
It should be noted that at an embodiment who is used for the execution ordering of action action processor 7802 or MMR server 8302 (also seeing Figure 83) comprise MMR action execution formation.Be controlled at the execution of the action in the MMR system 7800 by the formation of being sorted.When input action in formation, it is associated with right of priority with by the resource that demand is carried out action.The queue order function is repeatedly checked formation, and confirms to wait for preference action that carry out, the highest, is available for the whole required resources of said action.If reasonable time in the cycle its resource do not become availablely, then the action in formation can stop.
The RETRIEVE order of exemplary action for issuing by user interface application, and said application program provides relevant information stock price, real-time update that is linked to from the image of the text of the page of stock market tabulation.MMR architecture 7800 can be carried out in the formation in the action that the IMMEDIATE preference is arranged and insert this.If available, then can carry out action once receiving it to being linked as of current stock price.If not capable of using, then can come insert action in formation with overtime numerical value to being linked as of current stock price.For the thing of the volatibility as stock price, overtime can as 30 seconds, lacking.If required resource is not capable of using within the said time, then will stop moving (as, never carry out).
Another example of action being carried out queue order is with the interpolation of inserting the action of video clipping for the x-y position in the electronic document of being confirmed by the image of the fragment of text.Can it be assigned as preference WHEN_CONNECTED, and it will comprise the standard for needed database.This can be the MMR database on user's PC, and can only in certain following time said PC be connected to acquisition equipment 106.Overtime numerical value can will be carried out action because need only it for unlimited in next five minutes, it just can be effective and execution action in the coming years.
Unique aspect that architecture is carried out in MMR action for it, the ability suspended by the MMR document finger print matching system shown in Fig. 6 and 7.Control structure 714 is kept watch on the input from quality assessment modules 712 frequently.When its survey the frame of video that can be recognized (as, it comprises the image that appears as the text that machine prints) time, it can be sent to look-at-me the processor 210 of the execution of the MMR action that suspends current operation.Carry out to be available computational resource maximization to the document finger print matching system.
How the use that will describe order now allows MMR architecture 7800 that numerous application program or uses is provided with them.Though this tabulation is not completely, for example more suitably provide the result that illustrates the end that reaches, the use and being used to of acquisition equipment 106 through user 110 to carry out or the order and the parameter of trigger action it.For each example, the order of moving and reaching said action is provided.
The meeting guide: under this operating position, user 110 uses phone 106 to catch such as the such multimedia file of video clipping, and through the text fragment that points in the meeting guide it is attached to the electronics original text for document.Use INSERT to order the position of fragment that data and multimedia file is added on the text of corresponding M MR document 500, carry out the effect that multimedia is added into the fragment of document.
Guide-book: under this operating position, user 110 comes the retrieving multimedia file through pointing to the paragraph of text.For example, multimedia file can be for relevant in the montage by the new exhibition of described, the special places of text, and perhaps multimedia file can be the business and text lockup, that upgraded of indication enterprise.The RETRIEVE order is used in the position of the fragment of the text in corresponding M MR document 500, carries out the action that search chain is connected to the multimedia file of fragment.
The stock tabulation: under this operating position, user 110 retrieves information relevant stock, up-to-date through pointing to the stock page in the newspaper.Data at MMR document 500 refer to be back to information relevant user, personalized, and it comprises his commission account and property.This will be from the desktop PC the user; Another application program on the MMR computing machine 112 (as,
) derive., start or the action of trigger data retrieval (stock having) and financial transaction (Stock Purchase) in position by RETRIEVE order from the fragment of the text of the shared data bank of the information that comprises the stock quotation device.
With itself and the integrated MMR of advertisement in commercial document printing: under this operating position, advertisement MMR is linked to the product that is associated with advertisement.For example, can the cinema sign in printing magazine be linked to the Fandango website that people wherein can buy ticket.Action is connected to the information of advertisement for search chain, and remembers on user 110 account for said benefit.In the position of the fragment of the text of corresponding M MR document 500, order by RETRIEVE to start or trigger action, wherein the MMR document is a cinema sign.
Tagged multimedia: under this operating position, select the phrase that extracts from document printing, and it is attached to the multimedia of catching recently.Action is the selection text phrases from MMR preparation document, and just they is being attached to the multimedia of being caught by acquisition equipment 106.Order trigger action by INSERT_FROM, being inserted in the MMR document 500 with the multimedia of on acquisition equipment 106, being caught in the data of the position of the fragment of text.
The MMR document is seized device: under this operating position, user 110 uses MMR architecture 7800 to select document, and it is passed to acquisition equipment 106.Action is the electronic version of identification from the document of server 114, and it is passed to acquisition equipment 106 from server 114.Order with TRANSFER and to carry out action, being sent to the MMR client computer on acquisition equipment 106 from server 114 in the data of the position of the fragment of text.
Formulate the MMR front end of the plan of application program: under this operating position, the user catches the date, selects it from MMR preparation document, and data are added into calendar management device application program.Action is selected " date " for preparing document from MMR, and they are inserted in another software application on acquisition equipment 106., order by INSERT_FROM to start action on the calendar management device application's data on the acquisition equipment 106 in the position of the fragment of text.
MMR as the UI that buys medium file: under this operating position; Relevant scanning such as story " DesperateHousewives " such TV programme, in the magazine such such as " People Magazine " trigger option, to the appearing of user 110, buy the plot of series.Action is for the purchase of episode with to the download of user account (TiVo, iPOD or online iTunes account).Order for the TRANSFER that orders and will buy file to be passed to user's account or digital video recorder from their source in the data of the position of the fragment of text by PURCHASE and to carry out action.
There is the uniqueness of MMR to measure: under this operating position, propose sign or other figures, and measure its singularity with respect to the data in given MMR database.Mark Designing teacher that feeds back to the uniqueness of relevant sign.This is what wherein action can be for recurrence, and the example of other MMR actions of MMR action triggers.Can carry out for said RETRIEVE order, identification is near at the trigger of the number of the MMR document of the data of the position of the fragment of text with further analyze then, confirms the level of singularity.
The note that is used for the cooperation of MMR: under this operating position, a plurality of users 110 with note be added into relevant given document or books blog (as, children exchange they, the note of the chapters and sections of relevant Harry Potter).Can use identical mechanism to share exam questions and answer.To link to their answer in the exercise of the end of chapter.Can contribute answer by other people, perhaps they can be the answer author, official from books.Action for retrieval also/or just with multimedia be added into for publication, share MMR database 3400.The order that is used for carrying out said action is the RETRIEVE order, comes retrieve data and INSERT_TO to order data are placed on the position of the fragment of text.
Middle link: under this operating position, Monarch Notes (or Cliff Notes) is linked to the paragraph in Shakespeare, and vice versa.Can also it be made and be used for more relevant works or note theoretical, various editorials or comment.Action is the pointer of retrieval to the x-y position within the page of different publications.Use the RETRIEVE order for the position of the text fragment in Cliff Notes; And create extremely at the original text works; Like pointer, carry out action by the text fragment among Shakespeare ' the s Romeo and Juliet of Scholastic publishing house printing.
The MMR that is used for the dynamic document mandate: under this operating position, select phrase, and it is added into puts another document of drawing up (like, PowerPoint file) from MMR preparation document.Action is for to be added into another document with data, and use data and the position that is stored in the fragment of the text in the PowerPoint file are ordered with INSERT_FROM and accomplished interpolation.
Be used for the position and inform the MMR of application program: under this operating position,, come to be provided at the tabulation in those restaurants nearby, position of user to you through the advertisement of scanning for the restaurant chain.Action is for to come retrieving information based on the position of MMR result and acquisition equipment 106.For the matched text fragment or be near the data the current position of acquisition equipment 106, order trigger action by RETRIEVE.
The MMR that is used for the MP3 player: when MMR user 110 pointed to documents with their acquisition equipment 106, their MP3 player plays that is associated was housed in the recording on their the MP3 player in advance.Acquisition equipment 106 is wirelessly communicated by letter with the MP3 player through bluetooth.Action is the action on another device that will be enabled in the part that is not acquisition equipment 106 or MMR architecture 7800.The order that is used for accomplishing this is the RETRIEVE order.
The MMR that adds voice mail: voice mail is the infrastructure service device of the data that are used for being provided by MMR.Acquisition equipment 106 or other MMR client apparatus are retrieved its multimedia through dialling voice mail and playing the message that is stored in the there.Because MMR is required unseparated infrastructure server, and it can use available other computational resources to many companies, so this is favourable.Action is the transaction with the robotization of voice-mail system, dials in and multimedia that retrieval is associated with focus.User 110 can also be added into voice mail with audio frequency.The order that is used for accomplishing this is the AUTHOR order.
With the integrated MMR of Email: the result of MMR action impels Email that it is delivered to the startup of step of user's e-mail address.Action is the setting up procedure that causes the transmission of email message.
With the integrated MMR of calendar applications: user 110 catches and system 7800 identification advertisements and from non-MMR database entity order goods.That occurs, and the prompting of the relevant completion desired, said transaction of input in user's calendar.Action is the insertion in time-sensitive information to user's the calendar applications.Can order by PURCHASE to start or trigger action, and other processing is sightless for the user after said action.
Advertisement in commercial document printing, personalized: in the example of goods catalogue, print individual's advertisement with the mode of the personal information of secret writing in advertisement coding (like, credit card number).Under another situation, it is the identification (like, the number of goods catalogue) of publication.Extract said data through the MMR process, and it is transferred to server with id as being provided by acquisition equipment 106, the user.This guarantees that the MMR request is to do for oneself the user's of its printing goods catalogue.Action is authentication and transaction (commodity of purchase goods catalogue), and is ordered by PURCHASE and to trigger said action.
With reference now to Figure 83,, the embodiment according to distributed MMR system architecture 8300 of the present invention is shown.Distributed MMR system architecture 8300 comprise as with preceding text with reference to Figure 78 some in the identical parts that identical functions is arranged so described.For understand simple and easy with describe clear, use like terms and reference number to show as like the device of preceding text among Figure 78 those.Distributed MMR system 8300 comprises document 118, user 110, acquisition equipment 106, database 3400, MMR identification module or software 226/610 and the action processor as the part of MMR server 8302 7802 that is appeared.As can see that the parts 106,3400 and 7802 of distributed MMR system 8300 comprise software suite 222a to c, and preceding text have been described that it provides, whole function.Each example that those skilled in the art will recognize that software suite 222a to n can be all versions, those parts of having only the software suite 222 relevant with the operation that can on special parts, be performed or any part of between of the software suite 222 of repertoire that it is arranged.
MMR server 8302 is any one in the server, and preceding text have been described said server, and it comprises any part or all in their function.MMR server 8302 comprises other logic and the function as preceding text have been described, is connected to other system and mutual with said other system.For example, though not shown, can MMR server 8302 be connected to cellular infrastructure 132, Wireless Fidelity (Wi-Fi) technology 134, Bluetooth technology 136 and/or infrared ray (IR) technology 138.In Figure 83 with these or other said such as be depicted as through network 128 such connections other devices, direct line, and usually show the said signal wire 8304 that is connected to.More specifically, in order to be connected MMR server 8302 with the communication of subscriber computer 112, media server 114, ISP 122, STB 126, office inlet 120, first third party (TP) server 8306 and n third-party server 8308.Parts as distributed MMR system 8300, each all comprises MMR software suite 222d to n subscriber computer 112, media server 114, ISP 122, STB 126, office inlet 120, first third party (TP) server 8306 and n third-party server 8308.Moreover each instance of software suite 222d to n can be whole version of the software suite 222 of repertoire that it is arranged, or only is those parts of the software suite 222 relevant with the operation that can on special parts, be performed.Each all has the similar function as preceding text have been described subscriber computer 112, media server 114, ISP 122, STB 126, office inlet 120, first third party (TP) server 8306 and n third-party server 8308.It should be noted another system that 8306 to n third-party server of first third party (TP) server 8308 expression can be connected to MMR system 8300 for interoperability.For example, third-party server 8306,8308 can be the system of bank, wired company and company etc.
Because a plurality of reasons, so distributed MMR system 8300 is particularly advantageous.First; There are many, distributed architecture in the parts that comprise software suite 222 to allow MMR to handle; Come it to be crossed MMR system 8300 and to distribute with many modes, and said mode for such as through MMR is handled unload down to other system 112,114,122,126,120,8306 and 8308 such.Second; Because it is mutual that all device or system 106,3400,8302,112,114,122,126,8306 and 8308 use software suite 222; Therefore no matter when user 110 is mutual with MMR system 8300, all uses one group of single, consistent user interface.The 3rd, distributed MMR system 8300 enables to carry out the use of the performance of other system 114,122,126,120,8306 and 6838, carries out action and order, and not so distributed MMR system 8300 can carry out said action and order individually.The 4th, can the MMR coupling be used as trigger, begin the process of on other system 114,122,126,120,8306 and 6608, carrying out.
An application program that is used for this distributed architecture will be carried out more computing function on acquisition equipment 106.For example, when by duplicating document, but the document dump of being duplicated on acquisition equipment the time, is automatically translated said document such as the such server of multi-function printer.Similarly, any one in the MMR server can also be translated the document that is duplicated, and with text to acoustic application to the document that duplicated.This can be output on duplicating machine, that be used to the technology of disabled person's design, and when they used duplicating machine, said duplicating machine was not asked for help and stood in the front of said duplicating machine.Then, derive user interface, and said user interface is presented on the acquisition equipment 106 for duplicating machine.Disabled user can insert they mobile phone earphone and to use that to come with duplicating machine mutual.It can be replied with voice suggestion or buzzer and vibration.Just standing in the user before the front of duplicating machine, the user can also practise using duplicating machine on camera cell phone.
Figure 84 is used to use like the input control of other third party system 8306 according to the present invention and 8308 and the process flow diagram of an embodiment of the method for the MMR system 8300 exporting.Through as preceding text are described, obtaining 8002 images and carry out 8004MMR identification, method begins.Secondly, method set up for 8402 being connected between MMR system 8300 (particularly action processor 7802) and third-party server 8306.Secondly, method presents data in 8404 to the user.In one embodiment, carry out this through the GUI that data is sent to acquisition equipment 106.In another embodiment, accomplish this through letting third-party server 8306 that data are presented to the user.For example, if third-party server 8306 is a CATV infrastructure, then on electronic console 124 to the appearing of user, can data be sent to STB 126 through third-party server 8306 data are presented to the user.Those skilled in the art will recognize that how this can be presented to data user's a example for third-party server 8306.Multiple other mechanism is arranged, and the performance that can use said mechanism to depend on to provide by third-party server 8306 and function and data are presented to the user.Then, MMR system 8300 receives 8406 inputs at acquisition equipment 106 places.MMR system 8300 receiving courses, and input is sent to third-party server 8306.Third party system 8306 receives input from MMR system 8300, and carry out 8408 with import being associated of task.Secondly, method confirms whether 8410 accomplish mutual with third party system 8306.If like this, then method also finishes for what accomplish.On the other hand, if with the mutual of third party system 8306 be uncompleted, then method is back to step 8404 data is presented to the user, maybe request is presented to the user.The mobile phone that can enable camera with the example of use CATV infrastructure, MMR is the control device to wired TV system.For example, can data be inputed to MMR system 8300, input to third party system 8306 (CATV infrastructure) then from acquisition equipment 106.Then, third party system 8306 can use infrastructure, and STB 126 is communicated by letter with user 110, or communicates by letter with acquisition equipment 106 through MMR system 8300 backward.Thereby; Through the image and the keypad of use on acquisition equipment 106 of catching screen; Product shown in user 100 can order on TV, is paid bill, carries out banking, is ordered perhaps many other operations of film the operation of controller top box 126.This is the use of acquisition equipment 106, MMR system 8300 and third party system 8306.It will be appreciated by those skilled in the art that wherein the possibility of coming together to use a plurality of third party system 8306 and 8308 with MMR system 8300.
With reference now to Figure 85,, describes and be used for the method that system therein 8300 and third party system 8306 mutual working environments are used MMR system 8300.Method begins with the request that just is transfused to 8502 to MMR systems 8300.Described request is the visit of the working environment function that is used for being provided by MMR system 8300.Because MMR system 8300 is connected to MMR computing machine 112 and office inlet 120, therefore said MMR system 8300 can such as by its scene such relevant with given document, other information and document be provided.In addition, can start and relevant, the other action of working environment of wherein using the document that is appeared.Secondly, MMR system 8300 will present 8504 to user 110 with system's 8300 available MMR document process options.Said option includes, but are not limited to: the form that compiles (like, Word file) of 1) coming the original text of search file with MMR; 2) path of the form of retrieval to editable document; 3) retrieval is for the edit history of document; 4) with the identic catalogue of editable document in other documents of retrieval; 5) sound annotations is added into the form of the editable document of original text; 6) check and " incident " that be associated such as such document of being caught by the office inlet; 7) physically locate other versions of given document; 8) listen the session that occurs when time on the desktop of said document at you; 9) confirm to print other people of given document or its version; 10) confirm whose given chapters and sections online or the off-line reading documents; 11) with recently other people of the given chapters and sections of reading documents communicate by letter; 12) carrying out collaborative document consults; And 13) other special operations.
Secondly, method receives 8506 inputs from user 110.Through confirming whether 8508 users 110 have imported the history that request obtains document, and method continues.If user 110 has asked the history of document, then method continues to look for 8510 revision histories, and it is presented to user 110.For example, through pointing to the chapters and sections of document, when, where user 110 can see on acquisition equipment 106 or other system 8300 displays and whose establishment or change document.User 110 can also retrieve the form (like, Word file) of the editable document of original text with MMR.Other historical informations comprise with operation: at retrieval other documents as the form of editable document in the identical catalogue; Check " incident " that be associated with document (as, when document printing, appear near user's the PC 112 session---can catch these incidents by another device, like office inlet 120, said device is different from that of document printing); Check such as such establishments such as web page browsing, video viewing and comprise when document being the history of the document of the PC incident that occurs when opening; Check other documents physically adjacent in some time with present document, and some other devices of its hint make image or the identification document that is used for catching desktop---it can use RFID and with the MMR recognition application in those images.Searching system provides image, the incident of those documents and is attached to their history.Under said mode, can produce the figure of the physics life cycle of document.Then, method proceeds to step 8524.
If user 110 does not also import and obtains historical requests, then whether the request that note is added into document is arranged through 8512 users 110, method continues.If user 110 has imported request note is added into document, then method is caught 8514 notes, and it is added into MMR document 500.For example, user 100 can be with sound annotations, or picture or video clipping are added into the form of the editable document of original text.Then, method proceeds to step 8524.
If user 100 is the not request of input interpolation note also, then request through confirming whether 8516 users 110 have selected to get in touch other people, method continues.If user 110 has imported other people request of contact, then method confirm 8518 visited MMR document 500 other people also get in touch them so that user 110 can communicate by letter with other people of the given chapters and sections that read document recently.For example, through acquisition equipment 106 is pointed to Duan Jie, user 110 is read people's the tabulation of the chapters and sections of MMR document 500 recently, and can start immediately and their session.This can converse through the IM that triggers honeycomb telephone call or beginning and the selected people in tabulation and carry out.Alternately, the MMR document can comprise such as when reading or visiting special document, notifying other people such action in real time.Then, method proceeds to step 8524.
If user 110 is not other people request of input contact also, then whether selected operation that will be performed, special through 8520 users 110, method continues.If user 110 has selected the special operation that will be performed, then 8522 said special operations are proceeded and carried out to method.MMR system 8300 can carry out and in working environment, be useful, many special operations.For example, the performance that few is illustrated MMR system 8300 is described here.For example, other versions of given document are physically located in special being operating as.Like preceding text, said manipulate such as ptz camera such, the separated images capture systems watches other users' desktop, and the image that the MMR recognition application is caught in it.The existence of document is remembered in the daily record of shared data bank system.The session that occurs when time on the desk of said document at you is listened in second special being operating as.The said audio capturing system that is attached to the document that camera catches based on the document tracing system of camera with recording that manipulates.The 3rd specialized operations is other people of definite given document of printing or its version, and when they print it and use what printer.This utilize shared data bank with have definite document and another have heterogeneous like the version match program of logic.Whom the 4th special operation confirm is read the given chapters and sections of document online.Said manipulating with screen monitoring software in text that writes on the PC screen and the database matching on ongoing basis.Matching process uses MMR recognizer or text based coupling.Alternately, because the GDI engine usually is created in the text that illustrates on the screen, therefore can obtain screen text from our the current GDI engine that is used for catching document printing.Whom the 5th special operation confirm is read the given chapters and sections of document off-line.Said those the desktop video surveillance of image of catching that user 110 reads of manipulating.MMR recognizer identification document and the chapters and sections within them, and with information stores in shared data bank.The 6th specialized operations is for being added into web browser with the reading history of document as metadata (online or off-line ground).The method that does not rely on window application that this provides storage and representes said data.The 7th special operation is produced as timeline with a people's reading historical (online and off-line ground).Finally, the 8th the special collaborative document that is operating as consulted, and consulting by other people more than a reader of the different editions of wherein identical paper document pointed to the note that any chapters and sections of documents are used through the acquisition equipment 106 with them.Note is shown as the overlayer on the file and picture sketch map.As mentioned above, the example of some in these specialized operations that can carry out for MMR system 8300.
After step 8522 or step 8520, method confirm 8524 with the mutual of third-party server 8306 whether be what accomplish.If be uncompleted alternately, then method is back to step 8506 and comes to receive further input from the user.Yet,, accomplish said method and process and finish if mutual for what accomplish.
Another application program of MMR system 8300 is the mutual unique mode that provides with personal computer 112 or other computing environment.MMR identification is used as the mechanism that triggers or be enabled in activity performed on the personal computer 112.For example, use MMR identification on personal computer 112, to start video playback.When said chapters and sections when being visible, through acquisition equipment 106 being pointed to the chapters and sections of paper documents, the video clipping that change will be shown on the display of personal computer 112 illustrates to user 110.This index with the text of catching with on screen, showing of desktop video is demarcated and is combined.In second example, MMR identification is used as webpage reminds.Camera lens through waving video camera presents user 110 with the trace of the webpage when the corresponding chapters and sections of document are shown, checked on the display of personal computer 112 to document.Can be in browser, on the screen of camera cell phone mulched ground wait webpage be shown.In the 3rd example, use MMR to discern to consult how to accomplish form.Through pointing to blank form, they are illustrated to the user with whole history that said form is imported in the past.Can use this with form that shown, online in web browser.Can be captured in the history of the data of being imported on those documents with our screen capture system.In last example, the retrieval version of the document that do not rely on the time is used MMR identification.For the past and following (after document is in hand, printing), the version of document, use MMR to retrieve source file and the expression of being printed.These examples are only for using MMR to come the minority mode mutual with personal computer 112, and those skilled in the art will recognize that for mutual with personal computer 112, the instrument that MMR is can for how powerful.
In one embodiment of the invention, the MMR system comprises action processor and method and the MMR document of associated action is arranged.Because being used for the different abilities MMR document, different actions, appointment allow the MMR architecture to be used as general-purpose flip-flop or the start-up routine that is used for other processing; So the MMR file structure is particularly advantageous, and said ability is combined with many MMR documents of ability create to(for) position on any medium, special.In other words, can discern based on MMR and trigger or start other processing or action.Action processor receives the output of MMR identifying that output comprises the MMR document of an action at least.Action processor is carried out actions other system, various orders that are included in the MMR system or are connected to the MMR system.Because can carry out action, and carry out action, so the MMR system architecture is favourable through the piece that acquisition equipment is pointed to text.For example, action can be for will be with the text retrieval of electronic form to acquisition equipment, and retrieval is inserted into the MMR document for the standard of action with data, is delivered in the data between the document, and bought item is authorized action or consulted the historical information of relevant action.Action processor is stored in the historical information of relevant action in the database.With the piece of text and the x-y position in the document and the page related in, the historical information of the relevant action of storage.Then, can in catching the piece incident of identical text, retrieve, search for and analyze said information after a while.
The real border (MMR) of blending agent system and the technology that is associated be provided for forming the medium that comprises at least two types (as, as the print paper of first medium with as the digital content and/or the network linking of second medium) the technology of mixed media document.The MMR system comprises action processor and method and the MMR document of associated action is arranged.For different MMR documents, the MMR document is specified different actions, and creates many MMR documents for position on any medium, special, thereby allows the MMR architecture to be used as general-purpose flip-flop or the start-up routine that is used for other processing.Action processor receives the output of MMR identifying that output comprises the MMR document of an action at least.Action processor is carried out said actions other system, various orders that are included in the MMR system or are connected to the MMR system.The MMR system architecture can carry out such as the electronic form of document is retrieved the standard that is used to move to acquisition equipment, retrieval, with data be inserted into the MMR document, be delivered in data between the document, bought item, mandate action or consult the such action of historical information of relevant action.
In one embodiment of the invention, the MMR system comprises action processor and method and the MMR document of associated action is arranged.Because being used for the different abilities MMR document, different actions, appointment allow the MMR architecture to be used as general-purpose flip-flop or the start-up routine that is used for other processing; So the MMR file structure is particularly advantageous, and said ability is combined with many MMR documents of ability create to(for) position on any medium, special.In other words, can discern based on MMR and trigger or start other processing or action.Action processor receives the output of MMR identifying that output comprises the MMR document of an action at least.Action processor is carried out actions other system, various orders that are included in the MMR system or are connected to the MMR system.Because can carry out action, and carry out action, so the MMR system architecture is favourable through the piece that acquisition equipment is pointed to text.Exemplary action comprise with the text retrieval of electronic form to acquisition equipment, retrieval is inserted into the MMR document for the standard of action with data, is delivered in the data between the document, bought item is authorized action or is consulted the historical information of relevant action.The MMR system comprises that the MMR by the text fragment discerns the multiple user application (one or more action) that starts, and for example, is used for the information retrieval of guide-book, stock tabulation or advertisement; , the multimedia that record and storage are associated with document, catch for the information of calendar and also authorize such information capture in real time from the content of meeting such as record; Purchase is used for the medium file of the storage on any part of MMR document.
In a particular embodiment, the MMR system comprises action processor and method and the MMR document of associated action is arranged.Because being used for the different abilities MMR document, different actions, appointment allow the MMR architecture to be used as general-purpose flip-flop or the start-up routine that is used for other processing; So the MMR file structure is particularly advantageous, and said ability is combined with many MMR documents of ability create to(for) position on any medium, special.In other words, can discern based on MMR and trigger or start other processing or action.Action processor receives the output of MMR identifying that output comprises the MMR document of an action at least.Action processor is carried out actions other system, various orders that are included in the MMR system or are connected to the MMR system.Because can carry out action, and carry out action, so the MMR system architecture is favourable through the piece that acquisition equipment is pointed to text.Exemplary action comprise with the text retrieval of electronic form to acquisition equipment, retrieval is inserted into the MMR document for the standard of action with data, is delivered in the data between the document, bought item is authorized action or is consulted the historical information of relevant action.The MMR system comprises that the MMR by the text fragment discerns the multiple user application (one or more action) that starts, and for example, is used for the information retrieval of guide-book, stock tabulation or advertisement; , the multimedia that record and storage are associated with document, catch for the information of calendar and also authorize such information capture in real time from the content of meeting such as record; Purchase is used for the medium file of the storage on any part of MMR document.
In one embodiment of the invention, the MMR system comprises action processor and method and the MMR document of associated action is arranged.Because being used for the different abilities MMR document, different actions, appointment allow the MMR architecture to come with the general-purpose flip-flop or the start-up routine that act on other processing; So the MMR file structure is particularly advantageous, and said ability is combined with many MMR documents of ability create to(for) position on any medium, special.In other words, can discern based on MMR and trigger or start other processing or action.Action processor receives the output of MMR identifying that output comprises the MMR document of an action at least.Action processor is carried out actions other system, various orders that are included in the MMR system or are connected to the MMR system.Because can carry out action through acquisition equipment is pointed to the document that is appeared, and carry out action, so the MMR system architecture is favourable.Exemplary action comprises the electronic form of the document that retrieval appears, and retrieval is inserted into the MMR document for the standard of action with data, is delivered in the data between the document, and bought item is authorized action or consulted the historical information about action.The MMR system uses other system to carry out action, and with the MMR software dispatch among MMR system and these other systems.In another embodiment, with the MMR system, the I/O performance of MMR identification and acquisition equipment is used for the controlling mechanism for the third party system especially.
In one embodiment of the invention, the MMR system comprises action processor and method and the MMR document of associated action is arranged.The MMR file structure specify combine with the ability for many MMR documents of the special position on any medium created, for the different actions of different MMR documents, and said ability allows the MMR architecture to use general-purpose flip-flop or the start-up routine that acts on other processing.Action processor receives the output of MMR identifying that output comprises the MMR document of an action at least.Action processor is carried out actions other system, various orders that are included in the MMR system or are connected to the MMR system.The MMR system uses other system to carry out action, and with the MMR software dispatch among MMR system and these other systems.Because can carry out action through acquisition equipment is pointed to the document that is appeared, and carry out action, so the MMR system architecture is favourable.Exemplary action comprises the electronic form of the document that retrieval appears, and retrieval is inserted into the MMR document for the standard of action with data, is delivered in the data between the document, and bought item is authorized action or consulted the historical information about action.In another embodiment, with the MMR system, the I/O performance of MMR identification and acquisition equipment is used for the controlling mechanism for the third party system especially.
In this algorithm that appears for relating to any special computing machine or other apparatus inherently.According to embodiments of the invention, can be with the system programization of various general purpose and/or special purposes, or dispose said system.As will using numerous programming languages and/or structure to realize multiple said system for so significantly according to said open invention.In addition, embodiments of the invention can be operated on infosystem or network or operate with said infosystem or network.For example, the present invention can be at multi-function printer independently or the configuration of depending on is arranged and the network printing hands-operation of the function that changes.The present invention can operate with any infosystem, and said system from minimum function arranged those to being provided in those of this disclosed repertoire.
For explaining and purpose of description, presented the aforementioned description of embodiments of the invention.But do not meaning detailed or limiting the invention to disclosed precise forms.According to above-mentioned religious doctrine, many changes and change all are possible.Mean scope of the present invention and can't help this detailed description restriction, but by the claim restriction of this application.As be familiar with that those skilled in the art will understand, can embody the present invention with other specific forms, and do not deviate from its spirit or intrinsic propesties.Equally, the special name of module, routine, characteristic, attribute, method and others and part are not enforceable or important, and the mechanism of the present invention or its characteristic that realizes can have different titles, part and/or form.In addition, as inciting somebody to action clearly for the association area those of ordinary skill, module of the present invention, routine, characteristic, attribute, method and others can realize as software, hardware, firmware or this any combination of three.Equally; At an one of which example is that the parts of the present invention of module are realized Anywhere as software; These parts also can be as independent program, as the part of bigger program, as the program of a plurality of separation, as static state or dynamic link library, as the core loadable module, as device driver and/or for the those of ordinary skill in the field of computer programming; Present or in the future known each and any alternate manner, and realize.In addition, the present invention is in no way limited in any specific programming language or about any specific operating system or the realization of environment.In addition, open invention of the present invention meaning illustrative but be not the restriction of scope of the present invention, it is set forth in equivalent structures.
The present invention is based on the S.N.11/461 that U.S.'s priority requisition was submitted on July 31st, 2006, the S.N.11/461 that on July 31st, 017,2006 was submitted to, the S.N.11/461 that on July 31st, 024,2006 was submitted to; 032, the S.N.11/461 that was submitted on July 31st, 2006, the S.N.11/461 that on July 31st, 049,2006 was submitted to, the S.N.11/461 that on July 31st, 064,2006 was submitted to; 075, the S.N.11/461 that was submitted on July 31st, 2006, the S.N.11/461 that on July 31st, 090,2006 was submitted to, the S.N.11/461 that on July 31st, 126,2006 was submitted to; 143, the S.N.11/461 that was submitted on July 31st, 2006; 268, the S.N.11/461 that was submitted on July 31st, 2006, the S.N.60/710 that on August 23rd, 272,2005 was submitted to, the S.N.60/792 that on April 17th, 767,2006 was submitted to; The S.N.60/807 that on July 18th, 912 and 2006 was submitted to; 654, therefore its full content is incorporated into this, for your guidance.
Claims (50)
1. the medium of the medium that utilizes the first kind and second type is created the system of mixed media environment, and said system comprises:
Be equipped with the acquisition equipment of display, it catches first expression of the medium of the said first kind;
The blending agent processor, it interrelates said first expression and second expression and the mixed media document of index with medium of the said first kind; With
The medium memory device, it stores said mixed media document,
Wherein said blending agent processor is suitable for representing to compare with said second expression and confirm said first representes whether be complementary with said second expression, said first expression and said mixed media document are interrelated with said first, and
If in response to the relation between said first expression to the said mixed media document; Said first expression is complementary with said second expression of the mixed media document of coupling, and then said blending agent processor shows the set that is associated with said second expression and carries out the action that the user selects in this group action on the display of said acquisition equipment.
2. the system of claim 1 further comprises:
Communication mechanism, it is connected to said blending agent processor with said acquisition equipment.
3. the system of claim 1, wherein said communication mechanism is one from the group that comprises network, wired connection, infrared link, linking bluetooth, wireless communication link, cellular communication link, data communication links, mobile data service, the global system that is used for mobile communication and public switch telephone network.
4. the system of claim 1; The medium of the wherein said first kind is a paper document, and the medium of said second type is from one in the group that comprises data structure, order, text, audio frequency, video, image, digital photo, network linking text, application file, updated information and service.
5. the system of claim 1, wherein said action for from comprise retrieving information, place an order, in the group of document, document printing or image, display document or image, search information or presentation information that retrieve video, retrieval sound, canned data, establishment are new one.
6. the system of claim 1, wherein said acquisition equipment comprise numerical data or the portable input media of image of the part of the medium that can catch the said first kind of expression.
7. the system of claim 1, wherein said acquisition equipment comprises the portable output unit of the medium that can export said second type.
8. the system of claim 1, wherein said acquisition equipment is for from comprising camera cell phone; Personal digital assistant device; Digital camera; Bar code reader; RF identification (RFID) reader; Computer peripheral; Web camera; Or in the group of video card one.
9. the system of claim 1; Further comprise the storer that is connected to said blending agent processor; And instruction and/or data that said memory stores can be carried out by said blending agent processor, said memory stores from text or image fragment retrieve content-based information, the content-based retrieval parts.
10. the system of claim 1 further comprises the media base storer, and it stores the medium of said second type with primitive form, and wherein said media base storer is connected to said blending agent processor.
11. the system of claim 1 comprises that further each all has the blending agent storer of a plurality of mixed media documents of second expression and index, wherein said blending agent storer is connected to said blending agent processor.
12. the system of claim 1 further comprises output unit, it is in response to the medium of exporting said second type from the signal of said blending agent processor, and wherein said output unit is connected to said blending agent processor.
13. the system of claim 1, wherein said first be expressed as its can be used for pattern match, have a text fingerprints of document printing of two row at least.
14. the system of claim 1, wherein said first expression is identified in the position in the document printing uniquely, and is the part of mating of document printing.
15. system as claimed in claim 14, wherein said first is expressed as the text fingerprints of document printing of the part of a part with first row and adjacent second row.
16. the system of claim 1, wherein said mixed media document further comprise the file or the data structure of the medium of said second type, and wherein said index is the link between said file and said second expression.
17. said system as claimed in claim 1, wherein said mixed media document further comprises the file of the medium of a plurality of said second types.
18. the system of claim 1; Wherein said mixed media document further comprises the text or the one or more order that will be performed that are appeared, and wherein said index is linking between representing with said second at the file of the said text that will be appeared or one or more orders that will be performed.
19. the system of claim 1; Wherein said mixed media document further is included in the electronic representation corresponding to the document in the medium of the said first kind of said first expression, and wherein said index is the link between said electronic representation and said second expression.
20. the system of claim 1, the medium of related said second expression of wherein said index, said mixed media document and said second type.
21. the system of claim 1, wherein said mixed media document have each and all have second expression corresponding index, a plurality of.
22. the acquisition equipment of a blending agent system, and said acquisition equipment comprises:
Display;
Input media, it catches first expression of first media type;
Communication transceiver is connected to information and order is sent to the blending agent system and receives the input media and the output unit of information and order from the blending agent system; With
The blending agent processor; It all has a plurality of mixed media document comparisons of corresponding second expression with said first expression and each and confirms whether said first expression is complementary with said second expression; If corresponding second expression of the mixed media document of the said coupling of the said first expression coupling then shows the set that is associated with said second expression and carries out the action that the user selects in this group action on said display.
23. acquisition equipment as claimed in claim 22, wherein said first media type is a document printing.
24. acquisition equipment as claimed in claim 22, wherein said action for from comprise retrieving information, place an order, in the group of document, document printing or image, display document or image, search information or presentation information that retrieve video, retrieval sound, canned data, establishment are new one.
25. acquisition equipment as claimed in claim 22 further comprises the output unit that presents second media type, wherein said output unit is connected to that said blending agent processor receives will be by the instruction and data of said output unit output.
26. acquisition equipment as claimed in claim 25; Wherein said action is said second media type of output, and said second media type is from one in the group that comprises data structure, order, text, audio frequency, video, image, digital photo, network linking text, application file, updated information and service.
27. acquisition equipment as claimed in claim 22; Wherein said blending agent processor comprise from text or image fragment retrieve content-based information, the content-based retrieval parts, and the expression of wherein said first media type at first is the fragment of text.
28. acquisition equipment as claimed in claim 22, wherein said acquisition equipment comprises the memory storage of storing a plurality of mixed media documents.
29. acquisition equipment as claimed in claim 22; The text fingerprints of the document printing of the part of a part and second row of adjacent of wherein said blending agent processor through relatively having first row is with the expression and the corresponding expression comparison of said first media type.
30. the device that first media type is associated with second media type, said device comprises:
Blending agent processor with input end and output terminal; Be used for first expression of first media type is compared to confirm the mixed media document of coupling and confirmed whether said first expression is complementary with said second expression with a plurality of mixed media documents that each all has second expression; And if second expression of the mixed media document of the said coupling of the said first expression coupling; Show on the display of the acquisition equipment that obtains first expression that then depending on the user selects executable set and carry out the action that the user selects in this group action, and the input that connects said blending agent processor is brought in and is received said first and represent; With
Memory storage, it stores said a plurality of mixed media document, and wherein said memory storage is connected to said blending agent processor.
31. device as claimed in claim 30 further comprises communication mechanism, it is connected to said blending agent processor with acquisition equipment.
32. device as claimed in claim 30; Wherein said first media type is a paper document, and at least one mixed media document is included as from one second media type in the group that comprises data structure, order, text, audio frequency, video, image, digital photo, network linking text, application file, updated information and service.
33. device as claimed in claim 30, wherein said action for from comprise retrieving information, place an order, in the group of document, document printing or image, display document or image, search information or presentation information that retrieve video, retrieval sound, canned data, establishment are new one.
34. device as claimed in claim 30; Further comprise the storer that is connected to said blending agent processor; Instruction and/or data that said memory stores can be carried out by said blending agent processor, said memory stores from the content-based information of text or image fragment searching, the content-based retrieval parts.
35. device as claimed in claim 30, wherein said memory storage is stored second media type with primitive form.
36. device as claimed in claim 30, wherein said mixed media document further comprise index and file or data structure, and wherein said index is the link between said file and said second expression.
37. device as claimed in claim 30, wherein said mixed media document further comprise corresponding to the electronic representation of the document of said first media type of said first expression and link said electronic representation and the index of said second expression.
38. device as claimed in claim 30, wherein said first be expressed as can be used for pattern match, have a text fingerprints of document printing of at least two row.
39. one kind is used to provide the method with the mutual mixed media environment of the document of first media type and second media type, said method comprises:
Produce the expression of the document of first media type;
The expression index of the said document that is produced is demarcated to the mixed media document with input expression;
Obtaining said input representes;
Said input is represented to represent whether mate the expression that is produced with the expression comparison that is produced and definite said input; And
If the said input of the mixed media document of coupling is represented to be complementary with the expression that is produced, then show the set that is associated with the expression that is produced and carry out the action that the user selects.
40. method as claimed in claim 39, the expression of the document of said first media type of wherein said generation automatically comprises the text layout that catches document printing.
41. method as claimed in claim 40 is wherein by carrying out saidly catching automatically from one in the group that comprises document printing trapping module, operating system, print driver, scanner and file transfers mechanism.
42. method as claimed in claim 39, in the group of any other expression of the expression of the expression that is expressed as expression from the fragment of the expression that comprises whole documents, text, word that is wherein produced, the part of image and the part of mating of said document one.
43. method as claimed in claim 39 is wherein demarcated the expression index that produced and is comprised from one in the group of the electronic representation that comprises index, said document, second media type and action related with the expression that is produced.
44. like the said method of claim 39, wherein with the expression index that produced demarcate comprise from the x-y position that comprises the character of specifying document printing, specify the word of said document printing the x-y position, specify in the group of x-y position of x-y position, the x-y position of carrying out OCR operation and designated character and/or word, execution feature extraction and designated character and/or word of a part of image or image in the document printing.
45., wherein obtain input expression and comprise the possibility of confirming to extract exactly the feature description of expectation from the fragment and the pre-service of the fragment that comprises scan text, scan text, the data that scan machine-readable identifier, the said document of input identification, Email that reception has document attachment, receive the part of said document, scan image and import in the group of said document through file transfers like the said method of claim 39.
46. method as claimed in claim 39, wherein said input is represented and the expression that produced more further comprise execution said input represent and the expression that produced between pattern match.
47. method as claimed in claim 46, wherein the document fingerprint through document fingerprint that said input is represented and the expression that is produced compares, and carries out said pattern match.
Extract characteristic 48. method as claimed in claim 47, wherein more said document fingerprint comprise from the image of being caught, synthesize descriptor from said characteristic, and search the document and the fragment of the said descriptor that comprises certain ratio.
49. method as claimed in claim 39, wherein carry out by the specified action of said mixed media document comprise the retrieval be associated with the said mixed media document of the expression that is produced with said coupling, predetermined actions.
50. method as claimed in claim 39, wherein said action for from comprise retrieving information, with information write to the position, carry out to the search of information, on client apparatus display message, guiding interactive sessions, postpone the purchase of the said action that is used for execution after a while, change document, product and to of the group of the input of the order of another system.
Applications Claiming Priority (25)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US71076705P | 2005-08-23 | 2005-08-23 | |
| US60/710,767 | 2005-08-23 | ||
| US79291206P | 2006-04-17 | 2006-04-17 | |
| US60/792,912 | 2006-04-17 | ||
| US80765406P | 2006-07-18 | 2006-07-18 | |
| US60/807,654 | 2006-07-18 | ||
| US11/461,075 US7812986B2 (en) | 2005-08-23 | 2006-07-31 | System and methods for use of voice mail and email in a mixed media environment |
| US11/461,032 | 2006-07-31 | ||
| US11/461,090 US7551780B2 (en) | 2005-08-23 | 2006-07-31 | System and method for using individualized mixed document |
| US11/461,143 | 2006-07-31 | ||
| US11/461,272 US8005831B2 (en) | 2005-08-23 | 2006-07-31 | System and methods for creation and use of a mixed media environment with geographic location information |
| US11/461,024 | 2006-07-31 | ||
| US11/461,090 | 2006-07-31 | ||
| US11/461,049 US7920759B2 (en) | 2005-08-23 | 2006-07-31 | Triggering applications for distributed action execution and use of mixed media recognition as a control input |
| US11/461,075 | 2006-07-31 | ||
| US11/461,272 | 2006-07-31 | ||
| US11/461,126 US8195659B2 (en) | 2005-08-23 | 2006-07-31 | Integration and use of mixed media documents |
| US11/461,268 US7639387B2 (en) | 2005-08-23 | 2006-07-31 | Authoring tools using a mixed media environment |
| US11/461,064 | 2006-07-31 | ||
| US11/461,268 | 2006-07-31 | ||
| US11/461,049 | 2006-07-31 | ||
| US11/461,017 | 2006-07-31 | ||
| US11/461,143 US8156427B2 (en) | 2005-08-23 | 2006-07-31 | User interface for mixed media reality |
| US11/461,126 | 2006-07-31 | ||
| PCT/JP2006/316814 WO2007023994A1 (en) | 2005-08-23 | 2006-08-22 | System and methods for creation and use of a mixed media environment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101292258A CN101292258A (en) | 2008-10-22 |
| CN101292258B true CN101292258B (en) | 2012-11-21 |
Family
ID=40035652
Family Applications (4)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200680039477.4A Expired - Fee Related CN101297318B (en) | 2005-08-23 | 2006-08-22 | Data organization and access for mixed media document system |
| CN2006800393767A Active CN101292258B (en) | 2005-08-23 | 2006-08-22 | System and methods for creation and use of a mixed media environment |
| CN200680039532.XA Expired - Fee Related CN101297319B (en) | 2005-08-23 | 2006-08-22 | Embedding hot spots in electronic documents |
| CN2006800393983A Expired - Fee Related CN101292259B (en) | 2005-08-23 | 2006-08-22 | Method and system for image matching in a mixed media environment |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200680039477.4A Expired - Fee Related CN101297318B (en) | 2005-08-23 | 2006-08-22 | Data organization and access for mixed media document system |
Family Applications After (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200680039532.XA Expired - Fee Related CN101297319B (en) | 2005-08-23 | 2006-08-22 | Embedding hot spots in electronic documents |
| CN2006800393983A Expired - Fee Related CN101292259B (en) | 2005-08-23 | 2006-08-22 | Method and system for image matching in a mixed media environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (4) | CN101297318B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI496016B (en) * | 2013-01-02 | 2015-08-11 | 104 Corp | Method and system for managing hibrid database |
Families Citing this family (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102356405A (en) * | 2009-03-20 | 2012-02-15 | 谷歌公司 | Associating rendered advertisements with digital content |
| EP2275916A3 (en) * | 2009-06-29 | 2013-01-23 | Kabushiki Kaisha Toshiba | Print job managing apparatus, print job managing system, and print job managing method |
| US9245043B2 (en) * | 2009-12-23 | 2016-01-26 | Fuji Xerox Co., Ltd. | Embedded media markers and systems and methods for generating and using them |
| US8332424B2 (en) * | 2011-05-13 | 2012-12-11 | Google Inc. | Method and apparatus for enabling virtual tags |
| WO2013115788A1 (en) * | 2012-01-31 | 2013-08-08 | Hewlett-Packard Development Company, L.P. | Print sample feature set |
| CN104603833B (en) * | 2012-08-09 | 2018-12-14 | 温克应用程序有限公司 | Method and system for linking printing object with digital content |
| US9374517B2 (en) | 2012-10-12 | 2016-06-21 | Ebay Inc. | Guided photography and video on a mobile device |
| JP5998952B2 (en) * | 2013-01-25 | 2016-09-28 | 富士ゼロックス株式会社 | Sign image placement support apparatus and program |
| JP5967036B2 (en) * | 2013-08-22 | 2016-08-10 | 富士ゼロックス株式会社 | Image search system, information processing apparatus, and program |
| CN104699707A (en) * | 2013-12-06 | 2015-06-10 | 深圳先进技术研究院 | Data clustering method and device |
| US10043070B2 (en) * | 2016-01-29 | 2018-08-07 | Microsoft Technology Licensing, Llc | Image-based quality control |
| US11599833B2 (en) * | 2016-08-03 | 2023-03-07 | Ford Global Technologies, Llc | Vehicle ride sharing system and method using smart modules |
| US10558817B2 (en) * | 2017-01-30 | 2020-02-11 | Foley & Lardner LLP | Establishing a link between identifiers without disclosing specific identifying information |
| CN110020108B (en) * | 2017-09-12 | 2023-04-28 | 腾讯科技（深圳）有限公司 | Network resource recommendation method, device, computer equipment and storage medium |
| CN108446737B (en) * | 2018-03-21 | 2022-07-05 | 百度在线网络技术（北京）有限公司 | Method and device for identifying objects |
| CN109034267B (en) * | 2018-08-20 | 2019-07-12 | 南京乐象网络科技有限公司 | Piece caudal flexure intelligent selection device |
| CN110888993A (en) * | 2018-08-20 | 2020-03-17 | 珠海金山办公软件有限公司 | Composite document retrieval method and device and electronic equipment |
| CN111291167B (en) * | 2018-12-07 | 2023-05-05 | 宁波方太厨具有限公司 | Automatic product paper specification checking method based on image recognition |
| CN111339387B (en) * | 2018-12-18 | 2023-06-09 | 阿里巴巴集团控股有限公司 | Click feedback acquisition method and device based on information template and electronic equipment |
| US10846553B2 (en) * | 2019-03-20 | 2020-11-24 | Sap Se | Recognizing typewritten and handwritten characters using end-to-end deep learning |
| CN110210470B (en) * | 2019-06-05 | 2023-06-23 | 复旦大学 | Commodity information image recognition system |
| CN110909726B (en) * | 2019-11-15 | 2022-04-05 | 杨宏伟 | Written document interaction system and method based on image recognition |
| CN111275043B (en) * | 2020-01-22 | 2021-08-20 | 西北师范大学 | Paper numbered musical notation electronization play device based on PCNN handles |
| CN112597345B (en) * | 2020-10-30 | 2023-05-12 | 深圳市检验检疫科学研究院 | Automatic acquisition and matching method for laboratory data |
| CN114511058B (en) * | 2022-01-27 | 2023-06-02 | 国网江苏省电力有限公司泰州供电分公司 | Load element construction method and device for electric power user portrait |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6537324B1 (en) * | 1997-02-17 | 2003-03-25 | Ricoh Company, Ltd. | Generating and storing a link correlation table in hypertext documents at the time of storage |
| CN1641646A (en) * | 2004-01-15 | 2005-07-20 | 微软公司 | Image-based document indexing and retrieval |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6411953B1 (en) * | 1999-01-25 | 2002-06-25 | Lucent Technologies Inc. | Retrieval and matching of color patterns based on a predetermined vocabulary and grammar |
-
2006
- 2006-08-22 CN CN200680039477.4A patent/CN101297318B/en not_active Expired - Fee Related
- 2006-08-22 CN CN2006800393767A patent/CN101292258B/en active Active
- 2006-08-22 CN CN200680039532.XA patent/CN101297319B/en not_active Expired - Fee Related
- 2006-08-22 CN CN2006800393983A patent/CN101292259B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6537324B1 (en) * | 1997-02-17 | 2003-03-25 | Ricoh Company, Ltd. | Generating and storing a link correlation table in hypertext documents at the time of storage |
| CN1641646A (en) * | 2004-01-15 | 2005-07-20 | 微软公司 | Image-based document indexing and retrieval |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI496016B (en) * | 2013-01-02 | 2015-08-11 | 104 Corp | Method and system for managing hibrid database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101297319B (en) | 2013-02-27 |
| CN101292259B (en) | 2012-07-11 |
| CN101292258A (en) | 2008-10-22 |
| CN101292259A (en) | 2008-10-22 |
| CN101297318B (en) | 2013-01-23 |
| CN101297318A (en) | 2008-10-29 |
| CN101297319A (en) | 2008-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101292258B (en) | System and methods for creation and use of a mixed media environment | |
| KR100980748B1 (en) | System and methods for creation and use of a mixed media environment | |
| US7672543B2 (en) | Triggering applications based on a captured text in a mixed media environment | |
| US7920759B2 (en) | Triggering applications for distributed action execution and use of mixed media recognition as a control input | |
| CN102369724B (en) | Automatically capturing information, for example, use document awareness apparatus capturing information | |
| CN102822817B (en) | For the Search Results of the action taked of virtual query | |
| US7639387B2 (en) | Authoring tools using a mixed media environment | |
| US8005831B2 (en) | System and methods for creation and use of a mixed media environment with geographic location information | |
| US7769772B2 (en) | Mixed media reality brokerage network with layout-independent recognition | |
| US7551780B2 (en) | System and method for using individualized mixed document | |
| US7812986B2 (en) | System and methods for use of voice mail and email in a mixed media environment | |
| US20070046982A1 (en) | Triggering actions with captured input in a mixed media environment | |
| US20070047816A1 (en) | User Interface for Mixed Media Reality | |
| US20070050419A1 (en) | Mixed media reality brokerage network and methods of use | |
| KR100979457B1 (en) | Method and system for image matching in a mixed media environment | |
| CN102356405A (en) | Associating rendered advertisements with digital content | |
| EP2482210A2 (en) | System and methods for creation and use of a mixed media environment | |
| JP2009245440A (en) | System, method, and computer program that annotates unindexed documents | |
| CN102770862A (en) | Hybrid use of location sensor data and visual query to return local listings for visual query | |
| JP4897795B2 (en) | Processing apparatus, index table creation method, and computer program | |
| KR100960640B1 (en) | Method, system and computer readable recording medium for embedding a hotspot in a document |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |