WO2013173802A1

WO2013173802A1 - Systems and methods for crawling and indexing content

Info

Publication number: WO2013173802A1
Application number: PCT/US2013/041717
Authority: WO
Inventors: Chung Tin KWOK; Lei Zhong; Zhihuan QIU; Hyung J. KIM
Original assignee: Google Inc.
Priority date: 2012-05-17
Filing date: 2013-05-17
Publication date: 2013-11-21

Abstract

A system, computer-readable storage medium storing at least one program, and a computer-implemented method for crawling and indexing content is presented. A message is received, where the message includes (i) submitted content, wherein the submitted content is identified as being published by an entity, and (ii) a link to a location on a resource hosting the submitted content. An evaluation is made as to whether the submitted content is represented in an index of known content to determine whether the submitted content is new relative to the known content. In response to the evaluation, a request is issued to a search engine to crawl and index the submitted content hosted by the resource associated with the link when the submitted content is new relative to the known content.

Description

SYSTEMS AND METHODS FOR CRAWLING AND INDEXING

CONTENT

TECHNICAL FIELD

[0001] The disclosed embodiments relate generally to crawling and indexing content.

BACKGROUND

[0002] A search engine periodically crawls links to identify and index new (or previously un-indexed) content. When time-sensitive content such as news is published on a website at a time that occurs between times when the search engine performs two successive crawl operations for the website, the time-sensitive content may not be indexed and may not be available to the search engine until after the usefulness of the time-sensitive content has passed. For example, a news article on the financial status of a publicly-traded company may be published on a website during market trading hours and at a time that occurs after a prior crawl operation for the website and before a subsequent crawl operation for the website. If the times when the search engine performs the two successive crawl operations for the website are sufficiently long (e.g., every six hours) and the news article is only useful for the first hour after posting to the website, the news article is not available to the search engine before the usefulness of the news article has passed.

[0003] Another problem faced by search engines is that crawled content may contain duplicates or substantial duplicates of existing content. For instance, multiple content providers may provide the exact same content. The content may be syndicated across multiple websites. Alternatively or additionally, the content may be copied (e.g., without permission) onto a collection of websites that are used to increase the ranking of

predetermined web pages and/or websites (e.g., search engine optimization link farms). This poses a problem to indexers of network content because it could degrade their ability to use resulting indexes, which become overloaded with duplicate or near duplicate content, to provide unique relevant hits in response to user queries.

[0004] Given the above background there is a need in the art for improved systems and methods for crawling and indexing content. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The embodiments disclosed herein are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. Like reference numerals refer to corresponding parts throughout the drawings.

[0006] Figure 1 is a block diagram illustrating a network system, according to some embodiments.

[0007] Figure 2A is a block diagram illustrating an example process for crawling and indexing submitted content, according to some embodiments.

[0008] Figure 2B continues the process illustrated in Figure 2A, according to some embodiments.

[0009] Figure 3 A is a block diagram illustrating an example process for determining a likelihood that an entity is an author of original content, according to some embodiments.

[0010] Figure 3B continues the process illustrated in Figure 3A, according to some embodiments.

[0011] Figure 4A is a block diagram illustrating an example process for re-ranking ranked search results, according to some embodiments.

[0012] Figure 4B continues the process illustrated in Figure 4A, according to some embodiments.

[0013] Figure 5A is a block diagram illustrating an example process for registering an entity as a registered author of content, according to some embodiments.

[0014] Figure 5B continues the process illustrated in Figure 5A, according to some embodiments.

[0015] Figure 6 is a block diagram illustrating a server, according to some embodiments.

[0016] Figure 7 is a block diagram illustrating a resource server, according to some embodiments.

[0017] Figure 8 is a block diagram illustrating a search engine, according to some embodiments. [0018] Figure 9 is a block diagram illustrating a client device, according to some embodiments.

[0019] Figure 10 is a flowchart of a method for crawling and indexing content, according to some embodiments.

[0020] Figure 11 A is a flowchart of a method for determining whether submitted content is new relative to known content, according to some embodiments.

[0021] Figure 1 IB continues the process illustrated in Figure 11 A, according to some embodiments.

[0022] Figure 12 is a flowchart of a method for determining that an entity is an author of original content, according to some embodiments.

[0023] Figure 13 is a flowchart of a method for determining that shingles representing other submitted content identified as being published by an entity are first instances with respect to the corresponding shingles in the first plurality of shingles representing known content, according to some embodiments.

[0024] Figure 14 is a flowchart of a method for storing information for submitted content, according to some embodiments.

[0025] Figure 15 is a flowchart of a method for registering an entity as a registered author of content, according to some embodiments.

[0026] Figure 16A is an example screenshot illustrating a web page that allows an entity to notify a server that content has been published on a resource server, according to some embodiments.

[0027] Figure 16B is an example screenshot illustrating another web page that allows an entity to notify a server that content has been published on a resource server, according to some embodiments.

[0028] Figure 16C is an example screenshot illustrating another web page that allows an entity to notify a server that content has been published on a resource server, according to some embodiments. DETAILED DESCRIPTION

[0029] The embodiments described herein provide techniques for crawling and indexing content.

[0030] Figure 1 is a block diagram illustrating a network system 100, according to some embodiments. The network system 100 includes resource servers 102 (e.g., the resource servers 102-1 to 102-N), client devices 104 (e.g., the client devices 104-1 to 104-M) of entities 105 (e.g., entities 105-1 to 105-P), a server 110, and a search engine 112 coupled to network 120. Network 120 can generally include any type of wired or wireless

communication channel capable of coupling together computing nodes. This includes, but is not limited to, a local area network, a wide area network, or a combination of networks. In some embodiments, network 120 includes the Internet.

[0031] In some embodiments, a respective resource server 102 is associated with a respective entity 105. For example, in some embodiments the respective entity 105 is an operator of the respective resource server 102. In other embodiments, the respective entity 105 is a subscriber of a publication service (e.g., a news wire service) hosted by the respective resource server 102, and/or a subscriber of a web hosting service of the respective resource server 102.

[0032] In some implementations, the respective entity 105 includes at least one author of content. For example, the respective entity 105 may include an individual author or one of a plurality of co-authors for (or contributors to) content. In some embodiments, an entity 105 is a business organization that produces original, or partially original content. In some embodiments, an entity 105 is a news organization. In some implementations, the entity 105 includes at least one publisher of content. For example, the respective entity 105 may be a publisher of books, a publisher of periodicals, a publisher of online content and/or the like. In some implementations, the respective entity 105 is the author of content on at least one website. For example, the respective entity 105 may contribute original content to a blogging website, a website for a publisher (e.g., news, magazine, etc.) and/or the like. Note that such a website may include a subset of the content within a particular domain. For example, the website may include content in a particular domain (e.g., a top-level domain example.com). In another example, the website includes content in a sub-domain of the particular domain (e.g., a sub-domain blogs.example.com). In another example, the website includes content in a directory of the domain (e.g., www.example.com/~johndoe/). In some embodiments, the website includes content in: a plurality of domains (e.g., a network of affiliated websites), a plurality of sub-domains of at least one domain, and/or a plurality of subdirectories of at least one domain. In some embodiments, the content authored by an entity 105 is a blog post, a social network post, or a post in an on-line discussion thread. In some embodiments, the content authored by the entity 105 is any content that has been posted to a location accessible on the Internet such that it is readily ascertainable that the entity 105 posted the content.

[0033] Thus, network system 100 encompasses a broad array of systems 100. For example, in some embodiments (i) a plurality of entities 105 author or publish content (e.g., books, periodicals, websites, etc.) that is hosted on a single resource server 102, (ii) a plurality of entities 105 author content that is stored on a plurality of resource servers 102, and/or (iii) a single entity 105 authors content that is stored on a plurality of resource servers 102. Any combination of such arrangements is also found in the network systems 100 within the scope of the present disclosure. For instance, an example network system 100 provides for one entity 105 that authors content that is stored on two resource servers 102 while two or more other entities 105 author content that is stored on a single resource server 102.

[0034] The search engine 112 periodically crawls links to identify and index new (or previously un-indexed) content hosted by resource servers 102. The frequency at which the search engine 112 crawls links for a particular domain may depend on factors including, but not limited to, a frequency at which new (or previously un-indexed content) is available on the particular domain, an amount of traffic the domain receives from users (e.g., entities), an importance of web pages in the particular domain, whether the domain includes original content, and the like. The search engine 112 also identifies ranked search results that satisfy search queries received from client devices 104.

[0035] The client devices 104 include any computing device that the entities 105 may use to access the resource servers 102 (e.g., a website or content hosted on the resource servers 102, etc.), the server 1 10, and/or the search engine 112. For example, the client devices 104 may include a laptop computer, a desktop computer, a smartphone, a mobile phone, a personal digital assistant, a tablet device, a set top box, a game console, and/or the like.

[0036] In some embodiments, the entity 105-1 uses the client device 104-1 to publish content onto the resource server 102-1. For example, the resource server 102-1 may host content for a newspaper website and the entity 105-1 may use the client device 104-1 to upload and to publish a news article onto the newspaper website hosted on the resource server 102-1. Note that the entity 105-1 may upload the news article to the resource server 102-1 without publishing the news article. For example, the entity 105-1 may upload the news article to the resource server 102-1 and publish the news article at a later date and/or time. The act of publishing content makes the content available to users (e.g., to the public, to subscribers, etc.).

[0037] As discussed above, one problem faced by search engines is that content uploaded to the resource server 102-1 may not be indexed and may not be available to the search engine until after the usefulness of the content has passed. In order to address this problem, in some embodiments, the resource server 102-1 sends a message to the server 110 to notify the server 110 that the content has been published on the resource server 102-1. Alternatively, in some embodiments, the entity 105-1 uses the client device 104-1 to send a message to the server 110 to notify the server 110 that the content has been published on the resource server 102-1. For example, the entity 105-1 may use the client device 104-1 to send a message to the server 110 using a web page provided by the server 110 that allows the entity 105-1 to notify the server 110 that the content has been published on the resource server 102-1. In either of these alternative embodiments, the message includes all or a portion of the content, termed the submitted content 231 , which was published on the resource server 102-1 and/or a link to the published content on the resource server 102-1. The term "submitted content" is used in this specification to refer to the content included in the message or the content that is a target of a link that is included in the message. The server 110 may then conditionally issue a request to the search engine 112 to crawl and index the content (i.e., the submitted content 231) that was uploaded to the resource server 102-1. In some embodiments, the request is a high-priority request that is directed to a high priority crawler of the search engine 112. In these embodiments, a time when the high priority crawler of the search engine 112 is scheduled to crawl the content occurs before a time when a standard crawler of the search engine 112 is scheduled to crawl the submitted content 231. These embodiments are described in more detail below with reference to Figures 2A, 2B, and 10-15.

[0038] Some embodiments of the disclosed systems and methods further address the problem incurred by crawling and indexing when faced with duplicate or substantially duplicate existing content. Such duplicate content poses a challenge to the identification of authors of original content so that such original content may be attributed with authorship of content and accordingly returned in response to user queries. Accordingly, in some embodiments, the server 110 determines a likelihood that an entity is an author of original content. These embodiments are described in more detail below with reference to Figures 3 A and 3B. Furthermore, in some embodiments, search results including the content that are associated with authors of original content may be ranked higher in ranked search results than search results including the content that are not associated with authors of original content. These embodiments are described in more detail below with reference to, for example, Figures 4 A and 4B.

[0039] Note that although Figure 1 shows one instance for each of the server 110 and the search engine 112, multiple servers and search engines may be present in the network system 100. For example, each of the server 110 and the search engine 112 may include a plurality of distributed servers. The plurality of distributed servers may provide load balancing and/or may provide low-latency points of access to nearby computer systems. The distributed servers may be located within a single location (e.g., a data center, a building, etc.) or may be geographically distributed across multiple locations (e.g., data centers at various geographical locations, etc.).

[0040] Also note that although the embodiments described herein refer to the server

110 and the search engine 112, the embodiments may be applied to multiple servers and search engines. Furthermore, the functionality of any of the server 110 and the search engine 112 may be implemented within a single server (or a set of distributed servers). For example, the server 110 and the search engine 112 may be located on the same server (or the same set of distributed servers).

[0041] Figures 2 A and 2B are block diagrams 200 illustrating an example process for crawling and indexing submitted content 231, according to some embodiments. Note that although the following discussion refers to the resource server 102-1, the client device 104-1, and the entity 105-1, the process illustrated in Figures 2 A and 2B may be applied to any resource server, client device, and entity. As illustrated in Figure 2A, in response to content being published on the resource server 102-1, the resource server 102-1 sends a message 230 to the server 110 indicating that the content was published by the entity 105-1 on the resource server 102-1. [0042] In an alternative embodiment not depicted in Figure 2A, in response to content being published on the resource server 102-1, the client device 104-1 sends the message 230 to the server 110 indicating that the content was published by the entity 105-1 on the resource server 102-1. For example, the entity 105-1 may use the client device 104-1 to send the message 230 to the server 110 by accessing a web page provided by the server 110 that allows the entity 105-1 to notify the server 110 that the content has been published by the entity 105-1 on the resource server 102-1. In another example, the client device 104-1 may send the message 230 to the server 110 indicating that the content was published by the entity 105-1 on the resource server 102-1.

[0043] In some embodiments, the message 230 includes all of the content that was published on the resource server 102-1 and a link to the content that was published on the resource server 102-1. In some embodiments, the message 230 includes a portion of the content that was published on the resource server 102-1 and a link to the content that was published on the resource server 102-1. For example, the message 230 may contain the first kilobyte, first 10 kilobytes, first 100 kilobytes or first megabyte of the content that was published on the resource server 102-1. In some embodiments, the message 230 includes a digest of the content that was published on the resource server 102-1 and a link to the content that was published on the resource server 102-1. In some embodiments, the message 230 only includes a link to the content that was published on the resource server 102-1. In these embodiments, the server 110 obtains all or a portion of the content published on the resource server 102-1 prior to proceeding with the other operations described below. For example, in some embodiments the server 110 obtains the first kilobyte, first 10 kilobytes, first 100 kilobytes or first megabyte of the content that was published on the resource server 102-1. In another example, in some embodiments the server 110 obtains a digest of the content that was published on the resource server 102-1. This digest may be the result of, for example, a predetermined hash function on the known content. As discussed above, the term "submitted content" is used in this specification to refer to the content included in the message 230 or the content that is a target of a link that is included in the message 230.

[0044] Returning to the embodiment illustrated in Figure 2A, regardless of whether the message 230 originated from resource server 102 or from client device 104, the front end module 202 receives the message 230 and provides the message 230 to the content module 204. The content module 204 evaluates whether the submitted content 231 is represented in an index of known content 220 to determine whether the submitted content 231 is new relative to the known content.

[0045] In some embodiments, the index of known content 220 includes content obtained through crawl operations of the search engine 112. In some embodiments, the index of known content 220 includes a representation of content obtained through messages (e.g., the message 230) received from the resource server 102-1 (or the client device 104-1). The content is obtained either directly from a message 230 when the message 230 includes the submitted content 231 or indirectly when the message 230 includes a link to the submitted content 231.

[0046] As illustrated in Figure 2B, when the submitted content 231 is new relative to the known content in the index of known content 220, the content module 204 issues a request 232 to the search engine 112 (via the front end module 202) to crawl and index the submitted content 231 that is hosted by the resource server 102-1. The request 232 includes the link (e.g., address, URL, pointer) to the content that was published on the resource server 102-1 (i.e., the submitted content 231).

[0047] In some embodiments, the content module 204 stores a representation of the submitted content 231 in the index of known content 220. For example, in some instances, the content module 204 generates a set of shingles for the submitted content 231 , and for each respective shingle in the set of shingles, the content module 204 (i) applies a hash function to the shingle to produce a corresponding hash value for the respective shingle and (ii) stores information relating to the submitted content 231 (e.g., a link to a location on the resource server 102-1 hosting the submitted content 231, a timestamp of the message 230, etc.) in a record in the index of known content 220 that is indexed by the respective hash value.

[0048] The embodiments described in Figures 2A and 2B are described in more detail below with reference to Figures 10-15.

[0049] Figure 3A and 3B are block diagrams 300 illustrating an example process for determining a likelihood that an entity is an author of original content, according to some embodiments. Note that although the following discussion refers to the resource server 102-

1, the client device 104-1, and the entity 105-1, this process may be applied to any resource server, client device, and entity (e.g., any resource server 102, any client device 104, and any entity 105). As illustrated in Figure 3 A, in response to content being published on the resource server 102-1, the resource server 102-1 sends a message 330 to the server 110 indicating that the content was published by the entity 105-1 on the resource server 102-1. Alternatively, although not depicted in Figure 3 A, in response to content being published on the resource server 102-1 (or in response to an indication that content will be published at a future time on the resource server 102-1), the client device 104-1 associated with the entity 105-1 sends the message 330 to the server 110 indicating that the content was published by the entity 105-1 on the resource server 102-1 (or will be published on the resource server 102-1 at a future time). For example, the entity 105-1 may use the client device 104-1 to send the message 330 to the server 110 by accessing a web page provided by the server 110 that allows the entity 105-1 to notify the server 110 that the content has been published by the entity 105-1 on the resource server 102-1. Figures 16A-16C are example screenshots of web pages provided by the server 110 that allow the entity 105-1 to notify the server 110 that content has been published by the entity 105-1 on the resource server 102-1. In Figure 16 A, a web page 1600 allows the entity 105-1 to view content associated the entity 105-1. In this example, the entity 105-1 is associated with the domain "www.foo.com" and the path "bar," as illustrated by a location selector 1604. However, in other examples, the entity 105-1 may be associated with a subdomain offoo.com" (e.g., "blog.foo.com") and/or any path of the domain (or subdomain) offoo.com" (e.g., "blog.foo.com/bar/", "www.foo.com"). When the entity 105-1 presses a "register content" button 1602, the entity 105-1 is redirected to a content registration web page 1610, which is illustrated in Figure 16B. The content registration web page 1610 allows the entity 105-1 to provide content in a content field 1616, provide labels for the content in an optional label field 1612, provide a link to the content (e.g., URL) in an optional link field, and format the content using formatting tools 1614. The entity 105-1 submits the content to the server 110 by pressing a "submit content" button 1618. After submitting the content using the content registration web page 1610, the entity 105-1 is redirected to the web page 1600, which now indicates that the content is registered and the date (and/or time) at which the content was registered (e.g., see reference numbers 1622 and 1624). In some embodiments, full access to the functionally associated with the web pages illustrated in Figure 16 require that a user log in to their account by, for example, selecting the "My Account" tab 1650 and providing log in information that identifies the user. Returning to Figure 3 A, in another example, the client device 104-1 may send the message 330 to the server 110 indicating that the content was published by the entity 105-1 on the resource server 102-1. [0050] Returning to Figure 3 A, in some embodiments, the message 330 includes only the content that was published on the resource server 102-1 or that will be published on the resource server 102-1 at a future time. In some embodiments, the message 330 includes the content that was published on the resource server 102-1 and a link to the content that was published on the resource server 102-1.

[0051] In some embodiments, the message 330 only includes a link to the content that was published on the resource server 102-1. In some embodiments, the message 330 includes a portion of the content that was published on the resource server 102-1 and a link to the content that was published on the resource server 102-1. For example, the message 330 may contain the first kilobyte, first 10 kilobytes, first 100 kilobytes or first megabyte of the content that was published on the resource server 102-1. In some embodiments, the message 330 may contain a digest of the content that was published on the resource server 102-1. In embodiments where the message 330 does not include any portion of the content but does contain a link to the content, the server 110 obtains the content published on the resource server 102-1 prior to proceeding with the other operations described below. In some embodiments where the message 330 includes a portion of the content but not all of the content, the server 110 obtains the full set of content published on the resource server 102-1 prior to proceeding with the other operations described below. In some alternative embodiments where the message 330 includes a portion of the content but not all of the content, the server 110 does not obtain the full set of content published on the resource server 102-1 prior to proceeding with the other operations described below.

[0052] The front end module 202 receives the message 330 and provides the message

330 to the content module 204. The content module 204 stores a representation of the submitted content 331 in the index of known content 220. In some embodiments, the content module 204 stores a representation of the submitted content 331 in the index of known content 220. For example, in some embodiments, the content module 204 generates a set of shingles for the submitted content 331 and, for each respective shingle in the set of shingles, the content module 204 (i) applies a hash function to the respective shingle to produce a corresponding hash value for the respective shingle and (ii) stores information relating to the submitted content 331 (e.g., a link to a location on the resource server 102-1 hosting the submitted content 331, a timestamp of the message 330, etc.) in a record in the index of known content 220 that is indexed by the respective hash value. [0053] As illustrated in Figure 3B, the author module 206 calculates an original author score 332 for the entity 105-1 based on an evaluation of the collective representations of submitted content (e.g., all submitted content associated with the entity 105-1) in the index of known content 220 identified as being associated with the entity 105-1. The original author score is a metric representing a probability or likelihood that an entity is an author of original content. In some embodiments, the original author score for the entity 105-1 is calculated based at least in part on a percentage of the representations of submitted content in the index of known content 220 identified as being associated with the entity 105-1 that are first instances of representations of submitted content in the index of known content 220.

[0054] In some embodiments, the author module 206 calculates the original author score for the entity 105-1 in response to receiving the message 330. In some embodiments, the author module 206 periodically calculates the original author score for the entity 105-1 (e.g., every hour, every day, every week, etc.). In some embodiments, all content associated with the entity 105-1 in the index of known content 220 is considered in computing the original author score for the entity 105-1. In some embodiments, only more recent content associated with the entity 105-1 is considered in computing the original author score for the entity 105-1. For example, in various embodiments, only content within a threshold time period is used to calculate the original author score of the entity 105-1 (e.g., content with a timestamp within the past week, the past month, the past year, or the past two years, etc.). In some embodiments, only a discrete set of content associated with the entity 105-1 is considered in computing the original author score for the entity 105-1. For example, in various embodiments, only five or less, ten or less, fifty or less, or 100 or less different documents identified as being associated with the entity 105-1 are used to calculate the original author score of the entity 105-1. In some embodiments, recent content associated with the entity 105-1 in the index of known content provides a more significant contribution to the original author score for the entity 105-1 than less recent content. To illustrate one such embodiment, in some original authorship calculations, more recent content is up- weighted and less recent content is down- weighted in the original author score calculations.

[0055] In some embodiments, only content that is unique with respect to all other content that (i) has already been selected to compute an original author score for the entity 105-1 and (ii) is associated with the entity 105-1 is considered when computing the original author score for the entity. For example, in some embodiments, only content that (i) satisfies a dissimilarity criterion (e.g., at least 80 percent different, at least 90 percent different, etc.) with respect to all other content that has been selected for computation of an original author score for the entity 105-1 and (ii) is associated with the entity 105-1 is used in computing the original author score for the entity 105-1. In this way, original author scores are not biased by multiple submissions of the original same original work by the entity 105-1.

[0056] Dissimilarity between a first document and second document associated with the entity 105-1 can be computed using any measure of similarity or dissimilarity in order to determine with the dissimilarity criterion has been satisfied. In some embodiments, a first document and second document associated with the entity 105-1 satisfy the dissimilarity criterion when a Levenshtein edit distance from the first document to the second document is greater than a predetermined number of character edits. In some embodiments, the first and second documents satisfy the dissimilarity criterion when a Jaro-Winkler distance metric from the first document to the second document is more than a predetermined distance. In some embodiments, the first and second documents satisfy the dissimilarity criterion when a Jaccard index is less than a predetermined value. In some embodiments, the first and second documents satisfy the dissimilarity criterion when a Masi distance is greater than a predetermined distance.

[0057] In some embodiments, the first and second documents document satisfy the dissimilarity criterion when a character count of the first document and a character count of the second document deviate by more than a character count threshold (and/or a word count threshold). For example, if the character count threshold is five characters, the first document and the second document satisfy the dissimilarity criterion when the character count of the first document and the character count of the second document deviate by more than five characters.

[0058] In some embodiments, the first and second documents document satisfy the dissimilarity criterion when the first document contains a predetermined quantity of shingles that are not found in the second document, or vice versa, where such shingles are computed using the techniques disclosed below in conjunction with step 1202 of Figure 12.

[0059] In some embodiments, the author module 206 stores the original author score

332 in a record of an author database 222 that is associated with the entity 105-1.

[0060] The embodiments described in Figures 3A and 3B are described in more detail below with reference to Figures 10-16 in U.S. Provisional Patent Application No. 61/648,562 filed on May 17, 2012, entitled "Systems and Methods for Determining a Likelihood that an Entity is an Author of Original Content", which is hereby incorporated by reference herein in its entirety.

[0061] Figures 4 A and 4B are block diagrams 400 illustrating an example process for re-ranking ranked search results, according to some embodiments. As illustrated in Figure 4 A, ranked search results 430 are obtained from the search engine 112. The ranked search results 430 may be identified by the search engine 112 in response to the search engine 112 receiving a search query from a client device (e.g., the client device 104-1). The front end module 202 provides the ranked search results 430 to the re-ranking module 208.

[0062] When the ranked search results 430 includes a first search result

corresponding to a first document associated with a first entity (e.g., entity 105-1) and a second search result corresponding to a second document associated with a second entity (e.g., entity 105-2) where (1) the first search result is ranked higher than the second search result, (2) the first document and the second document satisfy a similarity criterion (e.g., based on information obtained from the index of known content 220), and (3) the second entity satisfies a predefined authorship differential with respect to the first entity (e.g., based on original author scores obtained from the author database 222), the re-ranking module 208 swaps the first ranked search result and the second ranked search result to produce re-ranked search results 431.

[0063] As illustrated in Figure 4B, the re-ranking module 208 provides the re-ranked search results 431 to the search engine 112 via the front end module 202. The search engine 112 then provides the re-ranked search results 431 to the client device for presentation to a user of the client device.

[0064] The embodiments described in Figures 4A and 4B are described in more detail below with reference to Figures 10-19 in U.S. Provisional Patent Application No. 61/648,553 filed on May 17, 2012, entitled "System and Method Re-ranking Ranked Search Results" which is hereby incorporated by reference herein in its entirety.

[0065] Figures 5A and 5B are block diagrams 500 illustrating an example process for registering an entity as a registered author of content, according to some embodiments. The process of registering an entity as a registered author of content is an optional process that is performed prior to processes described above with reference to Figures 2-4. Note that although the following discussion refers to the entity 105-1, this process may be used to register any entity as a registered author of content. Also note that the process of registering an entity as a registered author of content merely identifies the entity as an author of content, but does not verify whether the entity is an author of original content (e.g., an entity that generates original content versus an entity that copies content).

[0066] In accordance with the example embodiments illustrated in Figures 5A and

5B, the server 110 receives, from the client device 104-1, a registration request 530 to register the entity 105-1 as a registered author of content. The front end module 202 provides the registration request 530 to the registration module 210. When the registration module 210 determines that the entity 105-1 is not registered as a registered author of content, the registration module 210 provides, to the client device 104-1 associated with the entity 105-1 via the front end module 202, a verification mechanism 531 to verify that the entity 105-1 is an author of content. The verification mechanism 531 includes one or more requests and one or more targets.. For example, the verification mechanism 531 may include a request for the entity 105-1 to upload a predetermined file to a specified target such as a predetermined location on a resource server associated with the entity 105-1 (e.g., the resource server 102-

1). In another example, the verification mechanism 531 includes a request for the entity 105-

1 to add predetermined text to a target such as a predetermined webpage on a resource server associated with the entity 105-1 (e.g., the resource server 102-1). In another example, the verification mechanism 531 includes a request for the entity 105-1 to add a predetermined

HTML tag in a target such as a predetermined webpage hosted on a resource server associated with the entity 105-1 (e.g., the resource server 102-1). In still another example, the verification mechanism 531 includes a request for the entity 105-1 to add a predetermined script to the target such as a predetermined webpage hosted on the resource server 102 associated with the entity 105-1. In yet another example, the verification mechanism 531 includes a request for the entity 105-1 to add a DNS TXT record to a DNS record for a website operated by the entity 105-1 and hosted on a resource server associated with the entity 105-1 (e.g., a website of the entity 105-1 that is hosted on the resource server 102-1).

In some embodiments, in addition to performing any one of, or any combination of, the above-identified verification mechanisms, the registration module 210 may perform additional verification mechanisms such as, for example, ensuring that the applicant entity

105 is not on a list of known bad actors. For example, in some embodiments, when the applicant entity 105 is on a list of known bad actors, the applicant is denied registration even if the applicant successfully enacted the request at the target specified by the at least one verification mechanism 531. Example verification mechanisms 531 have now been described. One of skill in the art, based on the present disclosure, will recognize and appreciate that there are numerous other verification mechanisms 531 and all such verification mechanisms are within the scope of the present disclosure.

[0067] As illustrated in Figure 5B, in some embodiments, responsive to the providing the at least one verification mechanism 531, the server 110 receives a verification request 532 from the client device 104-1 to verify that the entity 105-1 is an author of content based upon the verification mechanism 531. The registration module 210 verifies that the entity 105-1 is an author of content by checking to see if the one or more requests included in the verification mechanism 531 have been enacted at the one or more targets specified by the verification mechanism 531. If each of the respective requests in the one or more requests has been enacted at corresponding targets in the one or more targets, the registration module 210 registers the entity 105-1 as a registered author of content. For example, the registration module 210 may store entity information 533 for the entity 105-1 in a record of an author database 222. In some embodiments in accordance with this example, the entity information 533 for the entity 105-1 includes a name of the entity 105-1, an original author score for the entity 105-1, links to websites associated with the entity 105-1, links to content authored by the entity 105-1 , links to profile pages for the entity 105-1 (e.g., on social networks or other websites), biographic information for the entity 105-1, and/or bibliographic information for the entity 105-1.

[0068] In an alternative to the embodiment depicted in Figure 5B, after providing the at least one verification mechanism 531 to the client device 104-1, the registration module 210 polls the target specified by the verification mechanism 531 to determine whether the request set forth in the verification mechanism 531 has been enacted. In some embodiments, the registration module 210 polls the target a number of times (e.g., every minute, every five minutes, every hour, each day) within a predetermined cut off time (e.g., within one hour, one day, one week after providing the verification mechanism 531 to the client device 104-1) to determine whether the request set forth in the verification mechanism 531 has been enacted. If this polling successfully determines that the request specified by the verification mechanism 531 has been enacted at the target, the registration module 210 registers the entity 105-1 as a registered author of content. [0069] Figure 6 is a block diagram illustrating the server 1 10, according to some embodiments. The server 1 10 typically includes one or more processing units (CPU's, sometimes called processors) 602 for executing programs (e.g., programs stored in memory 610), one or more network or other communications interfaces 604, memory 610, and one or more communication buses 609 for interconnecting these components. The communication buses 609 may include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. The server 1 10 optionally includes (but typically does not include) a user interface 605 comprising a display device 606 and input devices 608 (e.g., keyboard, mouse, touch screen, keypads, etc.). Memory 610 includes highspeed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and typically includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non- volatile solid state storage devices. Memory 610 optionally includes one or more storage devices remotely located from the CPU(s) 602. Memory 610, or alternately the non-volatile memory device(s) within memory 610, comprises a non-transitory computer readable storage medium. In some embodiments, memory 610 or the computer readable storage medium of memory 610 stores the following programs, modules and data structures, or a subset thereof:

• an operating system 612 that includes procedures for handling various basic system services and for performing hardware dependent tasks;

• a communication module 614 that is used for connecting the server 1 10 to other

computers via the one or more communication interfaces 604 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;

• an optional user interface module 616 that receives commands from the user via the input devices 608 and generates user interface objects in the display device 606;

• the front end module 202 that provides an interface between the server 1 10 and other computer systems, as described herein;

• the content module 204 that evaluates submitted content to determine whether the submitted content is new relative to known content in the index of known content 220, issues requests to the search engine 1 12 to crawl and index the submitted content, and/or stores a representation of the submitted content in the index of known content 220, as described herein; • the author module 206 that calculates an original author score for an entity based at least in part on an evaluation of representations of submitted content in the index of known content 220 identified as being associated with the entity, as described herein;

• the re -ranking module 208 that re -ranks ranked search results based at least in part on an original author score for the entities associated with the ranked search results, as described herein;

• the registration module 210 that registers an entity as a registered author of content, as described herein;

• the index of known content 220 that includes representations of submitted content, as described herein; and

• the author database 222 that includes entity information for entities (e.g., the entity information 533).

[0070] In some embodiments, the programs or modules identified above correspond to sets of instructions for performing a function described above. The sets of instructions can be executed by one or more processors (e.g., the CPUs 602). The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these programs or modules may be combined or otherwise re-arranged in various embodiments. In some embodiments, memory 610 stores a subset of the modules and data structures identified above.

Furthermore, memory 610 may store additional modules and data structures not described above.

[0071] Although Figure 6 shows a "server," Figure 6 is intended more as functional description of the various features which may be present in a set of servers than as a structural schematic of the embodiments described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some items shown separately in Figure 6 could be implemented on single servers and single items could be implemented by one or more servers. The actual number of servers used to implement a server and how features are allocated among them will vary from one implementation to another, and may depend in part on the amount of data traffic that the system must handle during peak usage periods as well as during average usage periods. [0072] Figure 7 is a block diagram illustrating the resource server 102-1 , according to some embodiments. Note that discussion below is not limited to the resource server 102-1 and may apply to any resource server. The resource server 102-1 typically includes one or more processing units (CPU's, sometimes called processors) 702 for executing programs (e.g., programs stored in memory 710), one or more network or other communications interfaces 704, memory 710, and one or more communication buses 709 for interconnecting these components. The communication buses 709 may include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. The resource server 102-1 optionally includes (but typically does not include) a user interface 705 comprising a display device 706 and input devices 708 (e.g., keyboard, mouse, touch screen, keypads, etc.). Memory 710 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and typically includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 710 optionally includes one or more storage devices remotely located from the CPU(s) 702. Memory 710, or alternately the non- volatile memory device(s) within memory 710, comprises a non-transitory computer readable storage medium. In some embodiments, memory 710 or the computer readable storage medium of memory 710 stores the following programs, modules and data structures, or a subset thereof:

• an operating system 712 that includes procedures for handling various basic system services and for performing hardware dependent tasks;

• a communication module 714 that is used for connecting the resource server 102-1 to other computers via the one or more communication interfaces 704 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;

• an optional user interface module 716 that receives commands from the user via the input devices 708 and generates user interface objects in the display device 706;

• a web module 718 that responds to requests to access web pages or other resources (e.g., scripts, programs, etc.) hosted on the resource server 102-1 ; and

• a publication module 720 that publishes content for entities and sends messages (e.g., the messages 230 and 330) to the server 1 10 indicating that content has been published on the resource server 102-1. [0073] In some embodiments, the programs or modules identified above correspond to sets of instructions for performing a function described above. The sets of instructions can be executed by one or more processors (e.g., the CPUs 702). The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these programs or modules may be combined or otherwise re-arranged in various embodiments. In some embodiments, memory 710 stores a subset of the modules and data structures identified above.

Furthermore, memory 710 may store additional modules and data structures not described above.

[0074] Although Figure 7 shows a "resource server," Figure 7 is intended more as functional description of the various features which may be present in a set of resource servers than as a structural schematic of the embodiments described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some items shown separately in Figure 7 could be implemented on single servers and single items could be implemented by one or more servers. The actual number of servers used to implement a resource server and how features are allocated among them will vary from one implementation to another, and may depend in part on the amount of data traffic that the system must handle during peak usage periods as well as during average usage periods.

[0075] Figure 8 is a block diagram illustrating the search engine 112, according to some embodiments. The search engine 112 typically includes one or more processing units (CPU's, sometimes called processors) 802 for executing programs (e.g., programs stored in memory 810), one or more network or other communications interfaces 804, memory 810, and one or more communication buses 809 for interconnecting these components. The communication buses 809 may include circuitry (sometimes called a chipset) that

interconnects and controls communications between system components. The search engine 112 optionally includes (but typically does not include) a user interface 805 comprising a display device 806 and input devices 808 (e.g., keyboard, mouse, touch screen, keypads, etc.). Memory 810 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and typically includes nonvolatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non- volatile solid state storage devices. Memory 810 optionally includes one or more storage devices remotely located from the CPU(s) 802. Memory 810, or alternately the non- volatile memory device(s) within memory 810, comprises a non-transitory computer readable storage medium. In some embodiments, memory 810 or the computer readable storage medium of memory 810 stores the following programs, modules and data structures, or a subset thereof:

• an operating system 812 that includes procedures for handling various basic system services and for performing hardware dependent tasks;

• a communication module 814 that is used for connecting the search engine 112 to other computers via the one or more communication interfaces 804 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;

• an optional user interface module 816 that receives commands from the user via the input devices 808 and generates user interface objects in the display device 806;

• a search module 818 that identifies ranked search results that satisfy a search query;

• a crawler module 820 that periodically performs normal-priority crawl operations on links to identify new content and performs high-priority crawl operations to identify new content in response to receiving a request from the content module 204 of the server 110; and

• an indexer module 822 that indexes new content identified by the crawler module 820 into a search index 824.

[0076] In some embodiments, the programs or modules identified above correspond to sets of instructions for performing a function described above. The sets of instructions can be executed by one or more processors (e.g., the CPUs 802). The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these programs or modules may be combined or otherwise re-arranged in various embodiments. In some embodiments, memory 810 stores a subset of the modules and data structures identified above.

Furthermore, memory 810 may store additional modules and data structures not described above.

[0077] Although Figure 8 shows a "search engine," Figure 8 is intended more as functional description of the various features which may be present in a set of servers implementing a search engine than as a structural schematic of the embodiments described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some items shown separately in Figure 8 could be implemented on single servers and single items could be implemented by one or more servers. The actual number of servers used to implement a search engine and how features are allocated among them will vary from one implementation to another, and may depend in part on the amount of data traffic that the system must handle during peak usage periods as well as during average usage periods.

[0078] Figure 9 is a block diagram illustrating the client device 104-1, according to some embodiments. Note that discussion below may apply to any client device. The client device 104-1 typically includes one or more processing units (CPU's, sometimes called processors) 902 for executing programs (e.g., programs stored in memory 910), one or more network or other communications interfaces 904, memory 910, and one or more

communication buses 909 for interconnecting these components. The communication buses 909 may include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. The client device 104-1 includes a user interface 905 comprising a display device 906 and input devices 908 (e.g., keyboard, mouse, touch screen, keypads, etc.). Memory 910 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and typically includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 910 optionally includes one or more storage devices remotely located from the CPU(s) 902. Memory 910, or alternately the non-volatile memory device(s) within memory 910, comprises a non-transitory computer readable storage medium. In some embodiments, memory 910 or the computer readable storage medium of memory 910 stores the following programs, modules and data structures, or a subset thereof:

• an operating system 912 that includes procedures for handling various basic system services and for performing hardware dependent tasks;

• a communication module 914 that is used for connecting the client device 104-1 to other computers via the one or more communication interfaces 904 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on; • a user interface module 916 that receives commands from the user via the input devices 908 and generates user interface objects in the display device 906; and

• a browser module 918 that provides a user interface for users to access and obtain documents and/or services hosted on servers (e.g., the resource servers 102), renders the documents and/or services in the user interface, and/or executes scripts (or other code).

[0079] In some embodiments, the programs or modules identified above correspond to sets of instructions for performing a function described above. The sets of instructions can be executed by one or more processors (e.g., the CPUs 902). The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these programs or modules may be combined or otherwise re-arranged in various embodiments. In some embodiments, memory 910 stores a subset of the modules and data structures identified above.

Furthermore, memory 910 may store additional modules and data structures not described above.

[0080] Although Figure 9 shows a "client device," Figure 9 is intended more as functional description of the various features which may be present in a client device than as a structural schematic of the embodiments described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated.

Crawling and Indexing Content

[0081] The embodiments described below with reference to Figures 10-15 address the problems faced by search engines in which content uploaded to a resource server may not be indexed and may not be available to the search engine until after the usefulness of the content has passed.

[0082] The following discussion refers to the resource server 102-1, the client device

104-1, the entity 105-1, and the search engine 112. However, it should be noted that the following discussion may be applied to any resource server, client device, entity, and search engine. Furthermore, the following discussion refers to particular modules of the server 110 performing particular operations illustrated in the Figures 10-15. However, the operations discussed below may be performed by other modules of the server 110. [0083] Figure 10 is a flowchart of a method 1000 for crawling and indexing submitted content, according to some embodiments. The content module 204 receives (1002) a message (e.g., the message 230, the message 330, etc.), where the message includes (i) submitted content (e.g., the submitted content 231, the submitted content 331 , etc.), where the submitted content is identified as being published by an entity (e.g., the entity 105-1), and (ii) a link to location on a resource (e.g., the resource server 102-1) hosting the submitted content.

[0084] In some embodiments, the message is received in response to the entity publishing the submitted content on a website. For example, the message may be generated automatically when the entity 105-1 publishes content on a website hosted on the resource server 102-1.

[0085] In some embodiments, the message is received in response to the entity submitting the submitted content and the link to the location on the resource hosting the submitted content using a user interface provided by the server 110. For example, the user interface provided by the server 110 may be a web page that allows an entity to notify the server 110 that content has been published by the entity on a resource server.

[0086] In some embodiments, the message is received in response to the entity submitting a link to the published location of the submitted content (e.g., an address associated with a resource server 102) using a user interface provided by the server 110.

[0087] In some embodiments, the message is received in response to the entity submitting the submitted content using a user interface provided by the server 110, where the submitted content is marked as being submitted by the entity prior to publication of the submitted content by the entity. For example, the user interface provided by the server 110 may be a web page that allows an entity to notify the server 110 that content will be published at a future time by the entity on a resource server. In some embodiments, the server infers that the content will be published at a future time by the entity on a resource server if a URL of the resource server on which the content will be published is not provided by the entity.

[0088] The content module 204 evaluates (1004) whether the submitted content is represented in the index of known content 220 to determine whether the submitted content is new relative to the known content. Operation 1004 is described in more detail below with reference to Figures 11 A and 1 IB. [0089] In response to the evaluating, the content module 204 issues (1008) a request to the search engine 112 to crawl and index the submitted content hosted by the resource associated with the link when the submitted content is new relative to the known content. In some embodiments, the request is a high priority request that is directed to a high priority crawler of the search engine 112. In these embodiments, a time when the high priority crawler of the search engine 112 is scheduled to crawl the submitted content occurs before a time when a standard crawler of the search engine 112 is scheduled to crawl the submitted content.

[0090] In some embodiments, the request to the search engine 112 includes the link to the location on the resource and a timestamp associated with the message. In these embodiments, the timestamp associated with the message is included in a search index entry for the submitted content (e.g., in an entry of the search index 824).

[0091] In some embodiments, a respective link to a respective location on a respective resource hosting a respective known content and a timestamp associated with the respective known content are included in a respective record of the index of known content 220. In some such embodiments, the index of known content 220 is indexed by a respective hash value for a respective shingle of the respective known content.

[0092] Attention is now directed to Figures 11 A and 1 IB, which are flowcharts of a method for determining (1004) whether submitted content is new relative to known content, according to some embodiments. The content module 204 identifies (1102) a set of shingles representing the submitted content, where a respective shingle in the set of shingles includes a predetermined number of consecutive tokens for the submitted content. In some embodiments, the shingles in the set of shingles include at least one overlapping token (e.g., word or character). For example, if the content is "a big dog crosses the street" and 3- shingling is used (i.e., the set of all unique combinations of three words in the content is used as the basis for shingling), the set of shingles includes the following shingles: "a big dog," "big dog crosses," "dog crosses the," and "crosses the street." In some embodiments, the shingles in the set of shingles do not include overlapping tokens. For example, using the example content from the previous example, the set of shingles for a 3 -shingling of the content includes the following shingles: "a big dog," and "crosses the street." It will be appreciated that w-Shingling, where w is a positive integer greater than 1 , is used to perform the shingling disclosed herein. In some embodiments w is 2, 3, 4, 5, 6, or an integer greater than 6.

[0093] Note that the discussion with reference to Figure 11 refers to operations performed with respect to shingles (e.g., the respective tokens of the content included in the respective shingles). However, in some embodiments, hash values of the tokens in the shingles are used instead of the actual tokens in the shingles. The hash values for the tokens of the content may be generated using any appropriate hash function (e.g., MD5, SHAl, etc.). For example, for the overlapping token 3 -shingling of the content from the example above, the hash values for the shingles "a big dog," "big dog crosses," "dog crosses the," and "crosses the street" may be used instead of the tokens themselves. Thus, when the terms "shingles" is used, it should be understood that hash values of the tokens in the shingles may be used instead.

[0094] The content module 204 compares (1104) the set of shingles representing the submitted content with a first plurality of shingles representing the known content. For example, the first plurality of shingles may include all or a subset of all of the shingles representing the known content that are included in the index of known content 220.

[0095] Operation 1106 specifies the conditions under which the submitted content is deemed to be a first instance (e.g., in time) of content based on a single criterion. The criterion specifies that, when less than a first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content (e.g., all or a subset of the shingles included in the index of known content 220), the content module 204 determines (1106) that the submitted content is new relative to the known content. For example, if the first predetermined threshold number of the set of shingles representing the submitted content is 51% of the total number of shingles in the set of shingles representing the submitted content, the content module 204 determines that the submitted content is new relative to the known content when less than 51% of the shingles in the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content (e.g., all or a subset of the shingles included in the index of known content 220).

[0096] Operations 1108-1112 specify the disposition of the submitted content when the criterion specified in operation 1106 fails. That is, when more than a first predetermined threshold number of the shingles in the submitted content shingle set are also found in the known content shingle set. Failure of the criterion specified in operation 1106 suggests that the index of known content 220 includes a sufficient number of shingles that match the shingles of the submitted content such that the submitted content is deemed to not be a first instance of the content. However, operations 1108-1112 include three possible condition sets which collectively determine whether the submitted content may be deemed new relative to the known content despite the failure of the criterion of operation 1106.

[0097] When (i) at least the first predetermined threshold number of the set of shingles representing the submitted content (e.g., 51% of the total number of shingles in the set of shingles representing the submitted content) is included in the first plurality of shingles representing the known content, (ii) at least a second predetermined threshold number of the set of shingles representing the submitted content (e.g., 50% of the total number of shingles in the set of shingles representing the submitted content) is included in a second plurality of shingles representing other submitted content identified as being published by the entity, and (iii) at least a third predetermined threshold of corresponding shingles in the second plurality of shingles representing the other submitted content (e.g., 51% of the total number of corresponding shingles in the second plurality of shingles representing the other submitted content) are first instances with respect to corresponding shingles in the first plurality of shingles representing the known content, the content module 204 determines (1108) that the submitted content is new relative to the known content. Stated another way, when condition (i) is satisfied, the submitted content is deemed to not be a first instance of the content in the index of known content 220. In other words, the index of known content 220 includes a sufficient number of shingles that match the shingles of the submitted content so that the submitted content is deemed to not be a first instance of the content in the index of known content 220. When condition (ii) is satisfied, the submitted content is deemed to be similar to (or a duplicate of) other submitted content associated with the entity. For example, the entity may have submitted the content prior to publication of the content (e.g., as discussed above with reference to Figures 2 A and 2B). When condition (iii) is satisfied, the shingles for the other submitted content that correspond to the submitted content are deemed to be first instances of the shingles in the index of known content 220. Operation 1108 serves to identify instances where operation 1106 failed to identify submitted content as new because the known content already included an appreciable amount of the entities' 105 own work and such work was causing the criterion of operation 1106 to fail. On this basis, when conditions (i), (ii), and (iii) of 1108 are satisfied, the submitted content is nevertheless deemed to be new relative to the known content in the index of known content 220.

[0098] When (i) at least the first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content, (ii) at least a second predetermined threshold number of the set of shingles representing the submitted content is included in a second plurality of shingles representing other submitted content identified as being published by the entity, and (iii) less than a third predetermined threshold of corresponding shingles in the second plurality of shingles representing the other submitted content are first instances with respect to corresponding shingles in the first plurality of shingles representing the known content, the content module 204 determines (1110) that the submitted content is not new relative to the known content. Conditions (i) and (ii) are the same as conditions (i) and (ii), respectively, in operation 1108. Condition (iii) in operation 1110 contemplates the other case of condition (iii) in operation 1 108, where the shingles for the other submitted content that correspond to the submitted content are not first instances of the shingles in the index of known content 220. When conditions (i), (ii), and (iii) are satisfied, the submitted content is deemed to not be new relative to the known content in the index of known content 220.

[0099] When (i) at least the first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content, and (ii) less than a second predetermined threshold number of the set of shingles representing the submitted content is included in a second plurality of shingles representing other submitted content identified as being published by the entity, the content module 204 determines (1112) that the submitted content is not new relative to the known content. Condition (i) is the same condition (i) in operation 1108. Condition (ii) in operation 1 112 contemplates the other case of condition (iii) in operation 1108 where the submitted content is deemed to not be similar (or not be a duplicate) to other submitted content associated with the entity. In other words, the submitted content is similar to (or a duplicate of) known content in the index of known content 220, but was not previously submitted by (or associated with) the entity (e.g., the submitted content was not previously submitted prior to publication of the submitted content). In some embodiments, the first predetermined threshold number, the second predetermined threshold number, and the third predetermined threshold number are each different numbers. In some embodiments one or more of the first predetermined threshold number, the second predetermined threshold number, and the third predetermined threshold number are the same. When conditions (i), (ii), and (iii) are satisfied, the submitted content is deemed to not be new relative to the known content in the index of known content 220.

[00100] In some embodiments, prior to issuing (1008) the request to the search engine

112 to crawl and index the submitted content hosted by the resource associated with the link, the content module 204 determines (1006) that the entity is an author of original content based on an evaluation of other submitted content identified as being published by the entity, where the other submitted content identified as being published by the entity is included in the known content. In these embodiments, the request to the search engine 112 to crawl and index the submitted content hosted by the resource associated with the link is issued responsive to determining that the submitted content is deemed to not be represented in the index of known content (i.e., new with respect to the index of known content) and

determining that the entity is an author of original content.

[00101] Attention is now directed to Figure 12, which is a flowchart of a method for determining (1006) that an entity is an author of original content based on the other submitted content identified as being published by the entity, according to some embodiments. The content module 204 identifies (1202) a second plurality of shingles representing the other submitted content identified as being published by the entity.

[00102] The content module 204 determines (1204) that at least a third predetermined threshold number of the second plurality of shingles representing the other submitted content identified as being published by the entity are first instances with respect to corresponding shingles in a first plurality of shingles representing the known content. Responsive to the determining, the content module 204 determines (1206) that the entity is an author of original content. In other words, when at least the third predetermined threshold number of shingles (e.g., 51% of the shingles) representing other submitted content identified as being published by (or otherwise associated with) the entity, the entity is deemed to be an author of original content. In some embodiments, an original author score is calculated for the entity. In these embodiments, when the original author score exceeds a predetermined threshold, the entity is deemed to be an author of original content.

[00103] Figure 13 is a flowchart of a method for determining (1204) that shingles representing the other submitted content identified as being published by an entity are first instances with respect to the corresponding shingles in the first plurality of shingles representing known content, according to some embodiments. The content module 204 determines (1302) a first number of the second plurality of shingles representing the other submitted content identified as being published by the entity that are unique with respect to the first plurality of shingles representing the known content.

[00104] The content module 204 determines (1304) a second number of the second plurality of shingles representing the other submitted content identified as being published by the entity that are not unique with respect to the shingles for the known content but that are associated with timestamps that precede timestamps for corresponding shingles in the first plurality of shingles representing the known content.

[00105] The content module 204 determines (1306) that a sum of the first number and the second number is at least the third predetermined threshold number.

[00106] Note that although the discussion above refers to shingles, any technique for partitioning content to generate portions of the content may be used. For example, the content may be partitioned into portions based on a predetermined number of characters.

[00107] In some embodiments, the content module 204 stores information for the submitted content in the index of known content 220. Figure 14 is a flowchart of a method 1400 for storing information for submitted content, according to some embodiments. The content module 204 identifies (1402) a set of shingles for the submitted content. For each respective shingle in the set of shingles, the content module applies (1404) a hash function to the shingle to produce a respective hash value for the respective shingle, stores (1406) the link to a location on the resource hosting the submitted content and a timestamp of the message in a record in the index of known content indexed by the respective hash value, and optionally stores (1408) an identifier for the entity in the record of the index of known content.

[00108] In some embodiments, prior to receiving the message, the server 110 registers the entity as a registered author of content. Figure 15 is a flowchart of a method 1500 for registering an entity as a registered author of content, according to some embodiments. The registration module 210 receives (1502), from a client device of the entity (e.g., the client device 104-1 of the entity 105-1), a registration request to register the entity as a registered author of content (e.g., the registration request 530). The registration module 210 provides

(1504), to the client device of the entity, at least one verification mechanism to verify that the entity is an author of content (e.g., the verification mechanism 531). Responsive to the providing, the registration module 210 receives (1506), from the client device of the entity, a verification request to verify that the entity is an author of content using the at least one verification mechanism (e.g., the verification request 532). The registration module 210 verifies (1508) that the entity is an author of content using the at least one verification mechanism and registers (1510) the entity as a registered author of content.

[00109] The methods illustrated in Figures 10-15 may be governed by instructions that are stored in a computer readable storage medium and that are executed by at least one processor of at least one server. Each of the operations shown in Figures 10-15 may correspond to instructions stored in a non-transitory computer memory or computer readable storage medium. In various implementations, the non-transitory computer readable storage medium includes a magnetic or optical disk storage device, solid state storage devices such as Flash memory, or other non-volatile memory device or devices. The computer readable instructions stored on the non-transitory computer readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted and/or executable by one or more processors.

[00110] Plural instances may be provided for components, operations or structures described herein as a single instance. Finally, boundaries between various components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the implementation(s). In general, structures and functionality presented as separate components in the example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the implementation(s).

[00111] It will also be understood that, although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, which changing the meaning of the description, so long as all occurrences of the "first contact" are renamed consistently and all occurrences of the second contact are renamed consistently. The first contact and the second contact are both contacts, but they are not the same contact.

[00112] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of the implementations and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[00113] As used herein, the term "if may be construed to mean "when" or "upon" or

"in response to determining" or "in accordance with a determination" or "in response to detecting," that a stated condition precedent is true, depending on the context. Similarly, the phrase "if it is determined (that a stated condition precedent is true)" or "if (a stated condition precedent is true)" or "when (a stated condition precedent is true)" may be construed to mean "upon determining" or "in response to determining" or "in accordance with a determination" or "upon detecting" or "in response to detecting" that the stated condition precedent is true, depending on the context.

[00114] The foregoing description included example systems, methods, techniques, instruction sequences, and computing machine program products that embody illustrative implementations. For purposes of explanation, numerous specific details were set forth in order to provide an understanding of various implementations of the inventive subject matter. It will be evident, however, to those skilled in the art that implementations of the inventive subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures and techniques have not been shown in detail.

[00115] The foregoing description, for purpose of explanation, has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The implementations were chosen and described in order to best explain the principles and their practical applications, to thereby enable others skilled in the art to best utilize the implementations and various implementations with various modifications as are suited to the particular use contemplated.

Claims

What is claimed is:

1. A computer-implemented method for crawling and indexing content, performed on a server system having at least one processor and memory storing at least one program for execution by the at least one processor to perform the method, comprising:

receiving a message, the message including (i) submitted content, wherein the submitted content is identified as being published by an entity, and (ii) a link to location on a resource hosting the submitted content;

evaluating whether the submitted content is represented in an index of known content to determine whether the submitted content is new relative to the known content; and

in response to the evaluating, issuing a request to a search engine to crawl and index the submitted content hosted by the resource associated with the link when the submitted content is new relative to the known content, wherein the request to the search engine to crawl and index the submitted content hosted by the resource associated with the link is issued responsive to determining that the submitted content is deemed to not be represented in the index of known content.

2. The computer-implemented method of claim 1, wherein the request is a high priority request that is directed to a high priority crawler of the search engine, and wherein a time when the high priority crawler of the search engine is scheduled to crawl the submitted content occurs before a time when a standard crawler of the search engine is scheduled to crawl the submitted content.

3. The computer-implemented method of any of claims 1-2, wherein evaluating whether the submitted content is represented in the index of known content to determine whether the submitted content is new relative to the known content includes:

identifying a set of shingles representing the submitted content, a respective shingle in the set of shingles including a predetermined number of consecutive tokens for the submitted content;

comparing the set of shingles representing the submitted content with a first plurality of shingles representing the known content; and

when less than a first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content, determining that the submitted content is new relative to the known content.

4. The computer-implemented method of claim 3, wherein when

(i) at least the first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content,

(ii) at least a second predetermined threshold number of the set of shingles representing the submitted content is included in a second plurality of shingles representing other submitted content identified as being published by the entity, and

(iii) at least a third predetermined threshold of corresponding shingles in the second plurality of shingles representing the other submitted content are first instances with respect to corresponding shingles in the first plurality of shingles representing the known content, the method includes:

determining that the submitted content is new relative to the known content.

5. The computer-implemented method of any of claims 3-4, wherein when

(iii) less than a third predetermined threshold of corresponding shingles in the second plurality of shingles representing the other submitted content are first instances with respect to corresponding shingles in the first plurality of shingles representing the known content, the method includes:

determining that the submitted content is not new relative to the known content.

6. The computer-implemented method of any of claims 3-5, wherein when

(i) at least the first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content, and (ii) less than a second predetermined threshold number of the set of shingles representing the submitted content is included in a second plurality of shingles representing other submitted content identified as being published by the entity, the method includes: determining that the submitted content is not new relative to the known content.

7. The computer-implemented method of any of claims 1-6,

wherein prior to issuing the request to the search engine to crawl and index the submitted content hosted by the resource associated with the link, the method includes determining that the entity is an author of original content based on an evaluation of other submitted content identified as being published by the entity, wherein the other submitted content identified as being published by the entity is included in the known content; and wherein the request to the search engine to crawl and index the submitted content hosted by the resource associated with the link is issued responsive to determining that the submitted content is deemed to not be represented in the index of known content and determining that the entity is an author of original content.

8. The computer-implemented method of claim 7, wherein determining that the entity is an author of original content based on the other submitted content identified as being published by the entity includes:

identifying a second plurality of shingles representing the other submitted content identified as being published by the entity;

determining that at least a third predetermined threshold number of the second plurality of shingles representing the other submitted content identified as being published by the entity are first instances with respect to corresponding shingles in a first plurality of shingles representing the known content; and

responsive to the determining, determining that the entity is an author of original content.

9. The computer-implemented method of claim 8, wherein determining that at least the third predetermined threshold number of the second plurality of shingles representing the other submitted content identified as being published by the entity are first instances with respect to the corresponding shingles in the first plurality of shingles representing the known content includes: determining a first number of the second plurality of shingles representing the other submitted content identified as being published by the entity that are unique with respect to the first plurality of shingles representing the known content;

determining a second number of the second plurality of shingles representing the other submitted content identified as being published by the entity that are not unique with respect to the shingles for the known content but that are associated with timestamps that precede timestamps for corresponding shingles in the first plurality of shingles representing the known content; and

determining that a sum of the first number and the second number is at least the third predetermined threshold number.

10. The computer-implemented method of any of claims 1-9, further comprising:

identifying a set of shingles for the submitted content;

for each respective shingle in the set of shingles,

applying a hash function to the shingle to produce a respective hash value for the respective shingle; and

storing the link to a location on the resource hosting the submitted content and a timestamp of the message in a record in the index of known content indexed by the respective hash value.

11. The computer-implemented method of claim 10, further comprising for each respective shingle in the set of shingles, storing an identifier for the entity in the record of the index of known content.

12. The computer-implemented method of any of claims 1-11, wherein the request to the search engine includes the link to the location on the resource and a timestamp associated with the message, and wherein the timestamp associated with the message is included in a search index entry for the submitted content.

13. The computer-implemented method of any of claims 1-12, wherein the message is received in response to the entity publishing the submitted content on a website.

14. The computer-implemented method of any of claims 1-12, wherein the message is received in response to the entity submitting the submitted content and the link to the location on the resource hosting the submitted content using a user interface provided by the server.

15. The computer-implemented method of any of claims 1-12, wherein the message is received in response to the entity submitting the submitted content using a user interface provided by the server, and wherein the submitted content is marked as being submitted by the entity prior to publication of the submitted content by the entity.

16. The computer-implemented method of any of claims 1-15, wherein the entity includes at least one author of content.

17. The computer-implemented method of any of claims 1-15, wherein the entity includes at least one publisher of content.

18. The computer-implemented method of any of claims 1-15, wherein the entity includes at least one website.

19. The computer-implemented method of any of claims 1-18, wherein (i) a respective link to a respective location on a respective resource hosting a respective known content and (ii) a timestamp associated with the respective known content are included in a respective record of the index of known content indexed by a respective hash value for a respective shingle of the respective known content.

20. The computer-implemented method of any of claims 1-19, wherein, prior to receiving the message, the method includes:

receiving, from a client device of the entity, a registration request to register the entity as a registered author of content;

providing, to the client device of the entity, at least one verification mechanism to verify that the entity is an author of content;

responsive to the providing, receiving, from the client device of the entity, a verification request to verify that the entity is an author of content using the at least one verification mechanism;

verifying that the entity is an author of content using the at least one verification mechanism; and

registering the entity as a registered author of content.

21. A system to crawl and index content, comprising:

at least one processor;

memory; and at least one program stored in the memory and executable by the at least one processor, the at least one program comprising instructions to:

receive a message, the message including (i) submitted content, wherein the submitted content is identified as being published by an entity, and (ii) a link to a location on a resource hosting the submitted content;

evaluate whether the submitted content is represented in an index of known content to determine whether the submitted content is new relative to the known content; and

in response to the evaluating, issue a request to a search engine to crawl and index the submitted content hosted by the resource associated with the link when the submitted content is new relative to the known content, wherein the request to the search engine to crawl and index the submitted content hosted by the resource associated with the link is issued responsive to determining that the submitted content is deemed to not be represented in the index of known content.

22. The system of claim 21, wherein the request is a high priority request that is directed to a high priority crawler of the search engine, and wherein a time when the high priority crawler of the search engine is scheduled to crawl the submitted content occurs before a time when a standard crawler of the search engine is scheduled to crawl the submitted content.

23. The system of any of claims 21-22, wherein the instructions to evaluate whether the submitted content is represented in the index of known content to determine whether the submitted content is new relative to the known content include instructions to:

identify a set of shingles representing the submitted content, a respective shingle in the set of shingles including a predetermined number of consecutive tokens for the submitted content;

compare the set of shingles representing the submitted content with a first plurality of shingles representing the known content; and

when less than a first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content, determine that the submitted content is new relative to the known content.

24. The system of claim 23, wherein when (i) at least the first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content,

(iii) at least a third predetermined threshold of corresponding shingles in the second plurality of shingles representing the other submitted content are first instances with respect to corresponding shingles in the first plurality of shingles representing the known content, the at least one program includes instructions to:

determine that the submitted content is new relative to the known content.

25. The system of any of claims 23-24, wherein when

(iii) less than a third predetermined threshold of corresponding shingles in the second plurality of shingles representing the other submitted content are first instances with respect to corresponding shingles in the first plurality of shingles representing the known content, the at least one program includes instructions to:

determine that the submitted content is not new relative to the known content.

26. The system of any of claims 23-25, wherein when

(i) at least the first predetermined threshold number of the set of shingles representing the submitted content is included in the first plurality of shingles representing the known content, and

(ii) less than a second predetermined threshold number of the set of shingles representing the submitted content is included in a second plurality of shingles representing other submitted content identified as being published by the entity, the at least one program includes instructions to:

determine that the submitted content is not new relative to the known content.

27. The system of any of claims 21-26,

wherein prior to issuing the request to the search engine to crawl and index the submitted content hosted by the resource associated with the link, the at least one program includes instructions to determine that the entity is an author of original content based on an evaluation of other submitted content identified as being published by the entity, wherein the other submitted content identified as being published by the entity is included in the known content; and

wherein the request to the search engine to crawl and index the submitted content hosted by the resource associated with the link is issued responsive to determining that the submitted content is deemed to not be represented in the index of known content and determining that the entity is an author of original content.

28. The system of claim 27, wherein the instructions to determine that the entity is an author of original content based on the other submitted content identified as being published by the entity include instructions to:

identify a second plurality of shingles representing the other submitted content identified as being published by the entity;

determine that at least a third predetermined threshold number of the second plurality of shingles representing the other submitted content identified as being published by the entity are first instances with respect to corresponding shingles in a first plurality of shingles representing the known content; and

responsive to the determining, determine that the entity is an author of original content.

29. The system of claim 28, wherein the instructions to determine that at least the third predetermined threshold number of the second plurality of shingles representing the other submitted content identified as being published by the entity are first instances with respect to the corresponding shingles in the first plurality of shingles representing the known content include instructions to:

determine a first number of the second plurality of shingles representing the other submitted content identified as being published by the entity that are unique with respect to the first plurality of shingles representing the known content;

determine a second number of the second plurality of shingles representing the other submitted content identified as being published by the entity that are not unique with respect to the shingles for the known content but that are associated with timestamps that precede timestamps for corresponding shingles in the first plurality of shingles representing the known content; and

determine that a sum of the first number and the second number is at least the third predetermined threshold number.

30. The system of any of claims 21-29, further comprising instructions to:

identify a set of shingles for the submitted content;

for each respective shingle in the set of shingles,

apply a hash function to the shingle to produce a respective hash value for the respective shingle; and

store the link to the location on the resource hosting the submitted content and a timestamp of the message in a record in the index of known content indexed by the respective hash value.

31. The system of claim 30, further comprising instructions to for each respective shingle in the set of shingles, store an identifier for the entity in the record of the index of known content.

32. The system of any of claims 21-31, wherein the request to the search engine includes the link to the location on the resource and a timestamp associated with the message, and wherein the timestamp associated with the message is included in a search index entry for the submitted content.

33. The system of any of claims 21-32, wherein the message is received in response to the entity publishing the submitted content on a website.

34. The system of any of claims 21-32, wherein the message is received in response to the entity submitting the submitted content and the link to the location on the resource hosting the submitted content using a user interface provided by the server.

35. The system of any of claims 21-32, wherein the message is received in response to the entity submitting the submitted content using a user interface provided by the server, and wherein the submitted content is marked as being submitted by the entity prior to publication of the submitted content by the entity.

36. The system of any of claims 21-35, wherein the entity includes at least one author of content.

37. The system of any of claims 21-35, wherein the entity includes at least one publisher of content.

38. The system of any of claims 21-35, wherein the entity includes at least one website.

39. The system of any of claims 21-38, wherein (i) a respective link to a respective location on a respective resource hosting a respective known content and (ii) a timestamp associated with the respective known content are included in a respective record of the index of known content indexed by a respective hash value for a respective shingle of the respective known content.

40. The system of any of claims 21-39, wherein, prior to receiving the message, the at least one program includes instructions to:

receive, from a client device of the entity, a registration request to register the entity as a registered author of content;

provide, to the client device of the entity, at least one verification mechanism to verify that the entity is an author of content;

responsive to the providing, receive, from the client device of the entity, a verification request to verify that the entity is an author of content using the at least one verification mechanism;

verify that the entity is an author of content using the at least one verification mechanism; and

register the entity as a registered author of content.

41. A non-transitory computer readable storage medium storing at least one program configured for execution by at least one processor of a computer system, the at least one program comprising instructions to:

evaluate whether the submitted content is represented in an index of known content to determine whether the submitted content is new relative to the known content; and in response to the evaluating, issue a request to a search engine to crawl and index the submitted content hosted by the resource associated with the link when the submitted content is new relative to the known content, wherein the request to the search engine to crawl and index the submitted content hosted by the resource associated with the link is issued responsive to determining that the submitted content is deemed to not be represented in the index of known content.

42. The non-transitory computer readable storage medium of claim 41 , wherein the request is a high priority request that is directed to a high priority crawler of the search engine, and wherein a time when the high priority crawler of the search engine is scheduled to crawl the submitted content occurs before a time when a standard crawler of the search engine is scheduled to crawl the submitted content.

43. The non-transitory computer readable storage medium of any of claims 41-42, wherein the instructions to evaluate whether the submitted content is represented in the index of known content to determine whether the submitted content is new relative to the known content include instructions to:

44. The non-transitory computer readable storage medium of claim 43, wherein when

(ii) at least a second predetermined threshold number of the set of shingles

representing the submitted content is included in a second plurality of shingles representing other submitted content identified as being published by the entity, and (iii) at least a third predetermined threshold of corresponding shingles in the second plurality of shingles representing the other submitted content are first instances with respect to corresponding shingles in the first plurality of shingles representing the known content, the at least one program includes instructions to:

determine that the submitted content is new relative to the known content.

45. The non-transitory computer readable storage medium of any of claims 43-44, wherein when

determine that the submitted content is not new relative to the known content.

46. The non-transitory computer readable storage medium of any of claims 43-45, wherein when

determine that the submitted content is not new relative to the known content.

47. The non-transitory computer readable storage medium of any of claims 41-46,

48. The non-transitory computer readable storage medium of claim 47, wherein the instructions to determine that the entity is an author of original content based on the other submitted content identified as being published by the entity include instructions to:

49. The non-transitory computer readable storage medium of claim 48, wherein the instructions to determine that at least the third predetermined threshold number of the second plurality of shingles representing the other submitted content identified as being published by the entity are first instances with respect to the corresponding shingles in the first plurality of shingles representing the known content include instructions to:

determine a second number of the second plurality of shingles representing the other submitted content identified as being published by the entity that are not unique with respect to the shingles for the known content but that are associated with timestamps that precede timestamps for corresponding shingles in the first plurality of shingles representing the known content; and determine that a sum of the first number and the second number is at least the third predetermined threshold number.

50. The non-transitory computer readable storage medium of any of claims 41-49, further comprising instructions to:

identify a set of shingles for the submitted content;

for each respective shingle in the set of shingles,

51. The non-transitory computer readable storage medium of claim 50, further comprising instructions to for each respective shingle in the set of shingles, store an identifier for the entity in the record of the index of known content.

52. The non-transitory computer readable storage medium of any of claims 41-51, wherein the request to the search engine includes the link to the location on the resource and a timestamp associated with the message, and wherein the timestamp associated with the message is included in a search index entry for the submitted content.

53. The non-transitory computer readable storage medium of any of claims 41-52, wherein the message is received in response to the entity publishing the submitted content on a website.

54. The non-transitory computer readable storage medium of any of claims 41-52, wherein the message is received in response to the entity submitting the submitted content and the link to the location on the resource hosting the submitted content using a user interface provided by the server.

55. The non-transitory computer readable storage medium of any of claims 41-52, wherein the message is received in response to the entity submitting the submitted content using a user interface provided by the server, and wherein the submitted content is marked as being submitted by the entity prior to publication of the submitted content by the entity.

56. The non-transitory computer readable storage medium of any of claims 41-55, wherein the entity includes at least one author of content.

57. The non-transitory computer readable storage medium of any of claims 41-55, wherein the entity includes at least one publisher of content.

58. The non-transitory computer readable storage medium of any of claims 41-55, wherein the entity includes at least one website.

59. The non-transitory computer readable storage medium of any of claims 41-58, wherein (i) a respective link to a respective location on a respective resource hosting a respective known content and (ii) a timestamp associated with the respective known content are included in a respective record of the index of known content indexed by a respective hash value for a respective shingle of the respective known content.

60. The non-transitory computer readable storage medium of any of claims 41-59, wherein, prior to receiving the message, the at least one program includes instructions to: receive, from a client device of the entity, a registration request to register the entity as a registered author of content;

register the entity as a registered author of content.

61. A non-transitory computer readable storage medium storing at least one program configured for execution by at least one processor of a system, the at least one program comprising instructions to be executed by the at least one processor so as to perform the method of any of claims 1-20.

62. A system, comprising:

at least one processor; and

memory storing at least one program for execution by the at least one processor; the system including means for performing the computer-implemented method of any of claims 1-20.