|Publication number||US7734627 B1|
|Application number||US 10/462,690|
|Publication date||Jun 8, 2010|
|Filing date||Jun 17, 2003|
|Priority date||Jun 17, 2003|
|Also published as||US8209339, US8650199|
|Publication number||10462690, 462690, US 7734627 B1, US 7734627B1, US-B1-7734627, US7734627 B1, US7734627B1|
|Original Assignee||Google Inc.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (30), Non-Patent Citations (26), Referenced by (34), Classifications (4), Legal Events (3)|
|External Links: USPTO, USPTO Assignment, Espacenet|
A. Field of the Invention
The present invention relates generally to document processing and, more particularly, to comparing documents to find similar or near duplicate documents.
B. Description of Related Art
There are a number of applications in which it may be desirable to be able to determine whether documents are similar or near duplicates of one another. Detecting spam email is one such application. Spam is unsolicited commercial email that is transmitted to multiple email accounts. To the receiver, spam is generally considered to be “junk email.”
In a typical spam episode, a single message is sent to thousands of email accounts. One known technique for removing spam from a network identifies spam based on its content. Thus, the network may be designed to recognize when many identical emails are being transmitted across the network. These identical emails can then be considered candidates for deletion before they arrive at the user email account.
In an effort to thwart automated spam detection and deletion, spam senders may slightly alter the text of each spam email by adding, removing, or replacing characters or superfluous sentences so as to defeat duplicate matching schemes. Thus, altered spam messages may be highly similar, but not identical, to one another.
Other applications for which similar document detection may be useful include detection of plagiarism and duplicate document detection in search engines.
Thus, there is a need in the art for techniques that can more accurately detect similar or near duplicate documents.
A document similarity detection technique consistent with the principles of the invention compares documents based on a set of relationships that define the relative order of terms within the documents.
One aspect of the invention is directed to a method for determining similarity of a document to a first set of documents. The method includes building a similarity model that defines a relative ordering of terms in the first set of documents, comparing an ordering of terms from the document to the similarity model, and generating similarity metrics that describe a degree of similarity between the document and the documents in the first set of documents based on the comparing of the ordering of terms.
Another aspect of the invention is directed to a similarity detection device. The device includes an inverted index that relates pairs of terms to clusters that contain the pairs of terms. The device further includes an enumeration component that generates pairs of terms for a received document and a pair lookup component that looks up the generated pairs in the inverted index to obtain clusters that contain the generated pairs. Further, the device includes a cluster selection component that selects those of the clusters obtained by the pair lookup component that are similar to the received document.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate the invention and, together with the description, explain the invention. In the drawings,
The following detailed description of the invention refers to the accompanying drawings. The detailed description does not limit the invention. Instead, the scope of the invention is defined by the appended claims and equivalents.
There are many ways in which documents may be determined to be similar, duplicates or near duplicates.
One document similarity detection technique is a “shingle” method. In the shingle method, a small consecutive sequence of words within a document is called a shingle. Two documents are said to be similar if they have several shingles in common. One problem with using the shingle method in adversarial situations, such as spam email, is that an adversary can defeat the matching algorithm by performing local swaps, replacements, or deletions of words within a sentence without changing the meaning of the sentence. As an example of this, consider the sentence: “The quick brown fox jumped over the lazy dog,” which can be transformed to: “The brown quick fox jumped over a lazy dog.” These two sentences do not share a single shingle of length four. Thus, the shingle method may classify these two sentences as not nearly identical when semantically the sentences are near duplicates of one another. Accordingly, automated programs that randomly alter words at the sentence level (such as substituting words with close synonyms, switching consecutive adjectives, and other local transformations) can defeat the shingle method.
Another document similarity detection technique is based on considering a document as a vector of terms. For example, the sentence: “The quick brown fox jumped over the lazy dog,” could be considered as a vector having eight entries—one for each unique word in the sentence. Term vector approaches known in the art, however, throw out ordering information in the document. Throwing out the ordering information can make it easier to get false matches when trying to find near duplicate documents because documents with the same words but entirely different word orders will be considered identical even though the documents may not be similar.
As an example of the possible problems of a term vector approach to detecting similar documents, consider the following three sentences: (1) When the defendant won the plaintiff hit the judge, (2) The judge hit the defendant when the plaintiff won, and (3) When the plaintiff hit the judge the defendant won. In a simple implementation of term vector similarity, all three sentences would have the same weighted term vectors and would be considered exact duplicates. For spam email duplicate detection, false matches are highly undesirable because it is important to users that legitimate emails are not deleted.
The detection of near duplicate or highly similar documents is also useful in many other applications, such as detection of plagiarism, duplicate document detection in search engines, etc. For web search engines, in particular, duplicate documents can often be undesirable. Storing duplicate documents effects both the accuracy and efficiency of the search engine. Further, retrieving duplicate documents in response to a user's query may lower the number of valid responses provided to the user, thus lowering the quality of the response set.
ISP 110 may include a spam filter 115, which may be implemented as a computer program stored on a computer-readable medium. Spam filter 115 may examine incoming email and delete messages that it determines to be spam. Spam filter 115 may make this determination based on results from a similarity detection component 117, which determines similarity between documents. If multiple emails transmitted through ISP 110 are determined by similarity detection component 117 to be highly similar or near duplicates of one another then spam filter 115 may consider these emails to be candidates for deletion. In some implementations, other features of the email messages, such as the transmitting domain name associated with the email messages, may be taken into account by spam filter 115 when determining whether to classify an email message as spam.
Another possible application of similarity detection component 117 is in the area of search engines.
Search engine 240 may be a program stored in a computer-readable medium that locates relevant information in response to search queries from users 220. In particular, users 220 send search queries to search engine 240, which responds by returning a list of relevant information to users 220. Typically, users 220 ask search engine 240 to locate web pages (i.e., documents) relating to a particular topic and stored at other devices or systems connected to network 205 (or another network). Search engine 240 may contain, or be coupled to, a database 245 that includes an index to the set of searchable web pages available though search engine 240.
Search engine 240 may use similarity detection component 117 in performing searches and/or in indexing the set of searchable web pages. Similar web pages detected by similarity detection component 117 may be used by search engine 240 in a number of ways. For example, highly similar web pages may not be separately stored in database 245. Alternatively, when returning results of a search to one of users 220, search engine 240 may use similarity detection component 117 to remove multiple references to nearly duplicate documents in the set of returned documents.
The operation of similarity detection component 117 according to one embodiment of the invention will next be described in detail. In general, similarity detection component 117 may operate in one of two modes. In the first mode, similarity detection component 117 adds new documents to a similarity model. In a second mode, similarity detection component 117 receives a document and determines if the document is similar to any of the documents in the model.
Cluster creation component 301 creates clusters Ci that describe documents. A cluster may be created for each of a number of documents i. Each cluster, Ci, may include one or more pairs of words from document i. Stated more formally
Ci=(u0,v0), (u1,v1), . . . , (un,vn),
where u and v represent terms in document i in which u comes before v, but the terms do not have to be consecutive. Thus, the pair (u0, v0) represents that document i contains the term u0 and the term v0 and that u0 occurs before v0. Generally, another document is said to be similar if it includes pairs that match the pairs in Ci. In other words, the other document is similar if it tends to contain words in the same order as those that appear in document i.
As another example of a cluster, consider the randomly sampled pairs shown in
Cluster creation component 301 may store each created pair for a cluster in inverted index 302. Inverted index 302 lists, for each pair, the clusters for which that pair was created.
In addition to maintaining inverted index 302, cluster creation component 301 may update table 303 when adding a new document to the similarity model.
The creation of a cluster Ci by cluster creation component 301 will now be described in more detail with reference to the flow chart of
Sampling the document to obtain the pairs can be performed using a number of different sampling techniques. The general goal is to create a useful representation of the document for the purpose of later determining similarity of the document to other documents. In one implementation, cluster creation component 301 randomly samples pairs of words from the input document. In one variation to this random sampling approach, the “random” sampling may be biased so that terms closer to each other have a greater chance of being included in a pair.
The number of pairs to sample for each cluster may be based on the length of the documents. Thus, clusters for longer documents may include more pairs.
Terms that have a lower frequency of occurrence in a corpus are often more relevant to the meaning of a document than more common terms. Accordingly, in some implementations, cluster creation component 301 may include a bias that is more likely to sample less frequently occurring terms. On the other hand, terms that are very rare, such as random sequences of symbols used by spammers to thwart similarity detection schemes, may not be included in the pairs of a cluster. Thus, in one embodiment cluster creation component 301 may be biased to sample rare words but to avoid very rare words. One of ordinary skill in the art will recognize that a precise meaning of “rare” and “very rare” may be obtained for a particular application through experimentation and/or observation of the frequency of occurrence of various terms in the corpus.
In addition to avoiding very rare terms, other terms, such as terms within HTML tags, may be ignored when sampling a document.
As another possible variation on document sampling, cluster creation component 301, instead of creating clusters that include entries that are pairs, may create clusters from triple, quadruple, or n-ary cluster entries. Such n-ary cluster entries may be referred to as n-ary vectors. For a cluster made of three term sets, for example, each entry would represent that the first term occurs before the second term, which both occur before the third term.
In another variation on the document sampling, cluster creation component 301 may bias the sampling such that pairs that occur in a pre-selected section of the document such as the upper middle section of the document are preferred. Email spammers may place “junk” terms near the bottom or beginning of a document in an attempt to thwart similarity detection. However, too many “junk” terms placed near the upper middle section of an email, such as in the first few paragraphs of the email, can make the email difficult to read and the reader may lose interest in the email if he/she has to scan past multiple lines of random text before seeing the true message. Accordingly, by sampling pairs from the upper middle section of a document, the clusters generated by cluster creation component 301 may be more resistant to spammer counter-measures. Cluster creation component 301, after sampling the pairs for a cluster, may update inverted index 302 to reflect the new cluster (Act 803). Cluster creation component 301 may also update table 303 by adding an entry in table 303 for the new cluster (Act 804). The entry may indicate the number of pairs that were sampled for the cluster.
The above discussion of similarity detection component 117 described the operation of similarity detection component 117 when adding a new document to the similarity model. In the second mode of operation, similarity detection component 117 determines similarity of an input document based on the similarity model defined by inverted index 302.
Pair enumeration component 901 enumerates the pairs within the input document. In one implementation, pair enumeration component 901 may enumerate all possible pairs for the input document. In other implementations, the pairs may be enumerated within a fixed window size within the input document. In this implementation, for each word u within the input document, pair enumeration component 901 may enumerate all pairs of words that include u and that are within a fixed number of words (the window size) after u. The window may then be moved to the next word after u and the process repeated. For example, if a document includes the five consecutive words a, b, c, d, and e, and the window size is set to three, pair enumeration component 901 may begin by forming the pairs ab, ac, and ad. Pair enumeration component 901 may then move the sliding window to word b and enumerate the pairs bc, bd, and be. The window may then be moved again and this process repeated for each word in the document. Using a fixed window size can be beneficial for both accuracy and efficiency.
For each enumerated pair, pair lookup component 902 may look up the pair in inverted index 302 to determine the previously stored clusters that correspond to the pair. Cluster aggregation component 904 may keep track of each cluster that was looked-up for the input document, as well as the number of occurrences of that cluster. For example, a short input document, after enumeration by pair enumeration component 901, may be determined to contain 10 pairs. The 10 pairs may correspond to 30 different clusters in the similarity model. Some of the 30 different clusters may have been output multiple times from inverted index 302, which indicates that the input document has multiple pairs in common with the document corresponding to the cluster.
Cluster selection component 904 may select the most frequently occurring clusters stored by cluster aggregation component 903. These are the clusters that have the most pairs in common with the input document. The most frequently occurring clusters can be defined as an absolute number (e.g., the input document may contain 15 of the pairs in C10) or on a percentage basis (e.g., the input document may contain 90% of the pairs included in cluster C10) or by using any other measure.
The operation of similarity detection component 117 when determining similarity of an input document will now be described in more detail with reference to the flow chart shown in
Similarity detection component 117 may begin by receiving a document for which a similarity determination is to be made (Act 1001). The received document will be called document B for purposes of this explanation. Pair enumeration component 901 may then enumerate pairs in document B (Act 1002). As previously mentioned, pair enumeration component may enumerate all possible pairs within document B or a subset of the possible pairs within document B. For example, all possible pairs of words (u,v) where u and v are within a fixed distance from each other in document B may be enumerated.
For each pair enumerated in Act 1002, pair lookup component 902 may use inverted index 302 to obtain the clusters that contain the pair (Act 1003). The total set of clusters, C(B), obtained by pair lookup component 902 may be maintained by cluster aggregation component 903 (Act 1004). Pair lookup component 902 may additionally tabulate the number of pairs that document B has in common with each cluster in C(B) (Act 1004).
Cluster selection component 904 may then use the information obtained in Act 1004 to determine a similarity metric that describes the similarity of document B to the documents that correspond to the clusters in C(B) (Act 1005). As mentioned, in one implementation, cluster selection component 904 may divide the number of pairs that document B has in common with a cluster to the number of pairs in that cluster to obtain the percentage of pairs that document B shares with the cluster. In another implementation, cluster selection component 904 may use the absolute number of pairs that document B has in common with the cluster as the similarity metric (e.g., B contains 15 of the pairs in cluster C10). Cluster selection component 904 may then compare the calculated similarity metrics for the clusters to a predetermined threshold value (Act 1006). Values above the threshold may indicate that the document B is a similar or near-duplicate document to the document corresponding to the cluster (Act 1007).
Longer documents are more likely to contain word pairs in the correct ordering. In some implementations, in order to further determine whether a document is to be considered a similar or near-duplicate document, additional factors, such as document length or a comparison of term vectors for the input document and the documents in the similarity model may also be performed.
Although the above discussion of duplicate or near duplicate document detection was primarily concerned with applications in spam email detection, other applications, such as plagiarism detection, are possible. In one plagiarism detection scheme, for example, a document A may be added to the similarity model by sampling pairs (or n-ary cluster entries), as previously described. Additionally, each paragraph or few paragraphs of document A (or other segments of document A) may be independently added to the similarity model as if it was an independent document. When adding paragraphs, the document from which the paragraph was excerpted from is also stored by the similarity model.
When a new document B is to be checked for plagiarism, the new document may be compared as a whole to determine if it is similar, and also compared as segments, such as paragraphs, to determine if any of the segments are similar. A final plagiarism judgment on document B can then be made based on one or more of a number of factors, including: (1) how many matching clusters document B has with other documents in the similarity model, (2) how many paragraphs (or other segments) in document B are similar to other paragraphs inserted in the similarity model, and (3) how many similar paragraphs document B has to each document A in the model. Thus, item (1) can be used to determine whether document B is a near copy of another document. Item (2) can be used to determine whether document B includes paragraphs from several other different documents. Item (3) can be used to determine whether document B includes multiple paragraphs from the same document.
The similarity detection described above can detect similar or near duplicate occurrences of a document and is relatively robust in the face of deliberate attempts to thwart its operation. The similarity detection component determines similarity of documents by characterizing the documents as clusters each made up of a set of term pairs. Another document that has a threshold level of term pairs in common with a cluster may be considered similar to the document characterized by the cluster.
It will be apparent to one of ordinary skill in the art that aspects of the invention, as described above, may be implemented in many different forms of software, firmware, and hardware in the implementations illustrated in the figures. The actual software code or specialized control hardware used to implement aspects consistent with the present invention is not limiting of the present invention. Thus, the operation and behavior of the aspects were described without reference to the specific software code—it being understood that a person of ordinary skill in the art would be able to design software and control hardware to implement the aspects based on the description herein.
The foregoing description of preferred embodiments of the present invention provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention.
No element, act, or instruction used in the description of the present application should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used.
The scope of the invention is defined by the claims and their equivalents.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4358824 *||Dec 28, 1979||Nov 9, 1982||International Business Machines Corporation||Office correspondence storage and retrieval system|
|US4691341 *||Mar 18, 1985||Sep 1, 1987||General Electric Company||Method of transferring digital information and street lighting control system|
|US4823306 *||Aug 14, 1987||Apr 18, 1989||International Business Machines Corporation||Text search system|
|US4839853 *||Sep 15, 1988||Jun 13, 1989||Bell Communications Research, Inc.||Computer information retrieval using latent semantic structure|
|US5297039 *||Jan 27, 1992||Mar 22, 1994||Mitsubishi Denki Kabushiki Kaisha||Text search system for locating on the basis of keyword matching and keyword relationship matching|
|US5321833 *||Aug 29, 1990||Jun 14, 1994||Gte Laboratories Incorporated||Adaptive ranking system for information retrieval|
|US5418951 *||Sep 30, 1994||May 23, 1995||The United States Of America As Represented By The Director Of National Security Agency||Method of retrieving documents that concern the same topic|
|US5442546 *||Nov 30, 1992||Aug 15, 1995||Hitachi, Ltd.||System and method for automatically generating translation templates from a pair of bilingual sentences|
|US5442778 *||Nov 12, 1991||Aug 15, 1995||Xerox Corporation||Scatter-gather: a cluster-based method and apparatus for browsing large document collections|
|US5619709 *||Nov 21, 1995||Apr 8, 1997||Hnc, Inc.||System and method of context vector generation and retrieval|
|US5640553 *||Sep 15, 1995||Jun 17, 1997||Infonautics Corporation||Relevance normalization for documents retrieved from an information retrieval system in response to a query|
|US5652898 *||Aug 31, 1993||Jul 29, 1997||Hitachi, Ltd.||Dictionary memory for text processing using word frequency and word recency occurrence information|
|US5675819 *||Jun 16, 1994||Oct 7, 1997||Xerox Corporation||Document information retrieval using global word co-occurrence patterns|
|US5805771 *||Jun 22, 1994||Sep 8, 1998||Texas Instruments Incorporated||Automatic language identification method and system|
|US5867811 *||Jun 17, 1994||Feb 2, 1999||Canon Research Centre Europe Ltd.||Method, an apparatus, a system, a storage device, and a computer readable medium using a bilingual database including aligned corpora|
|US5909677 *||Jun 18, 1996||Jun 1, 1999||Digital Equipment Corporation||Method for determining the resemblance of documents|
|US5913185 *||Dec 20, 1996||Jun 15, 1999||International Business Machines Corporation||Determining a natural language shift in a computer document|
|US5913208 *||Jul 9, 1996||Jun 15, 1999||International Business Machines Corporation||Identifying duplicate documents from search results without comparing document content|
|US5926812 *||Mar 28, 1997||Jul 20, 1999||Mantra Technologies, Inc.||Document extraction and comparison method with applications to automatic personalized database searching|
|US5963940 *||Aug 14, 1996||Oct 5, 1999||Syracuse University||Natural language information retrieval system and method|
|US6098033 *||Jul 31, 1997||Aug 1, 2000||Microsoft Corporation||Determining similarity between words|
|US6112021 *||Dec 19, 1997||Aug 29, 2000||Mitsubishi Electric Information Technology Center America, Inc, (Ita)||Markov model discriminator using negative examples|
|US6119124||Mar 26, 1998||Sep 12, 2000||Digital Equipment Corporation||Method for clustering closely resembling data objects|
|US6161130 *||Jun 23, 1998||Dec 12, 2000||Microsoft Corporation||Technique which utilizes a probabilistic classifier to detect "junk" e-mail by automatically updating a training and re-training the classifier based on the updated training set|
|US6169999 *||Apr 14, 1998||Jan 2, 2001||Matsushita Electric Industrial Co., Ltd.||Dictionary and index creating system and document retrieval system|
|US6192360 *||Jun 23, 1998||Feb 20, 2001||Microsoft Corporation||Methods and apparatus for classifying text and for building a text classifier|
|US6621930 *||Aug 9, 2000||Sep 16, 2003||Elron Software, Inc.||Automatic categorization of documents based on textual content|
|US6687696 *||Jul 26, 2001||Feb 3, 2004||Recommind Inc.||System and method for personalized search, information filtering, and for generating recommendations utilizing statistical latent class models|
|US6990628 *||Jun 14, 1999||Jan 24, 2006||Yahoo! Inc.||Method and apparatus for measuring similarity among electronic documents|
|US7188106 *||Apr 30, 2002||Mar 6, 2007||International Business Machines Corporation||System and method for aggregating ranking results from various sources to improve the results of web searching|
|1||*||"A Bayesian Approach to Filtering Junk E-Mail" by M. Sahami et al. Published 1998. Accessed Jun. 18, 2006. Available from: http://research.microsoft.com/users/horvitz/junkfilter.htm.|
|2||*||"Inverted index" by P.E. Black in "Dictionary of Algorithms and Data Structures" by U.S. National Institute of Standards and Technology. Dec. 17, 2004. Accessed Jun. 18, 2006. Available from: http://www.nist.gov/dads/HTML/invertedIndex.html.|
|3||*||"Inverted index-Wikipedia" by Wikipedia. Accessed Jun. 18, 2006. Available from: http://en.wikipedia.org/wiki/Inverted-index.|
|4||*||"Levenshtein Distance, in Three Flavors" by Michael Gilleland. Available online at http://www.merriampark.com/Id.htm. Accessed Jan. 8, 2006.|
|5||*||"Inverted index—Wikipedia" by Wikipedia. Accessed Jun. 18, 2006. Available from: http://en.wikipedia.org/wiki/Inverted—index.|
|6||A. Chowdhury et al.: "Collection Statistics for Fast Duplicate Document Detection," pp. 1-30, Apr. 2002.|
|7||Andrei Z. Broder et al.: "Syntactic Clustering of the Web," Proc. 6th International World Wide Web Conference; Apr. 1997 and SRC Technical Note 1997-015; Jul. 25, 1997; pp. 1-14.|
|8||Andrei Z. Broder: "On the resemblance and containment of documents," Proc. Of Compression and Complexity of Sequences 1997; IEEE Computer Society; pp. 1-9.|
|9||Andrei Z. Broder: "Some applications of Rabin's fingerprinting method," Sequences II: Methods in Communications, Security, and Computer Science; (Springer-Verlag, 1993); pp. 1-10.|
|10||*||Baker, L. D. and McCallum, A. K. 1998. Distributional clustering of words for text classification. In Proceedings of the 21st Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Melbourne, Australia, Aug. 24-28, 1998). SIGIR '98. ACM Press, NY, NY, 96-103. DOI= http://doi.acm.org/10.1145/290941.290970.|
|11||*||Cliff. "Ask Slashdot: Seeking Prior Art on Markov-Based SPAM Filters?" Published Nov. 28, 2002. Accessed Sep. 28, 2007. Available online at: http://ask.slashdot.org/article.pl?sid=02/11/27/0841216.|
|12||*||Cohen, W. W. 1996a. Learning rules that classify e-mail. In Papers from the AAAI Spring Symposium on Machine Learning in Information Access, 18-25. http://citeseer.ist.psu.edu/cohen96learning.html.|
|13||Co-pending Application entitled "Detecting Duplicate and Near-Duplicate Files," filed Jan. 24, 2001; William Pugh et al.; 61 page specification; 18 sheets of drawings.|
|14||Co-pending U.S. Appl. No. 10/425,819, filed Apr. 30, 2003 entitled "Systems and Methods for Predicting Lists," 36 page specification, 16 sheets of drawings.|
|15||*||Cutting, D. and Pedersen, J. 1990. Optimization for dynamic inverted index maintenance. In Proceedings of the 13th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Brussels, Belgium, Sep. 5-7, 1990). J. Vidick, Ed. SIGIR '90. ACM, New York, NY, 405-411.|
|16||*||Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391-407, http://citeseer.ist.psu.edu/deerwester90indexing.html.|
|17||*||H. Drucker, W. Donghui, V.N. Vapnik, "Support vector machines for spam categorization," Neural Networks, IEEE Transactions on, vol. 10, No. 5, pp. 1048-1054, Sep. 1999.|
|18||*||H. Drucker, W.Donghui, V.N. Vapnik, "Support vector machines for spam categorization," Neural Networks, IEEE Transactions on, vol. 10, No. 5, pp. 1048-1054, Sep. 1999.|
|19||Lecture Notes, CS276A, "Text Information Retrieval, Mining, and Exploitation," Nov. 19, 2002, http://www.stanford.edu/class/cs276a/handouts/lecture-13-gin1.pdf.|
|20||Lecture Notes, CS276A, "Text Information Retrieval, Mining, and Exploitation," Nov. 19, 2002, http://www.stanford.edu/class/cs276a/handouts/lecture—13-gin1.pdf.|
|21||Min Fang et al.: "Computing Iceberg Queries Efficiently," Proc. 24th International Conference on Very Large Databases; (1998); pp. 1-25.|
|22||*||Paul Graham. "A Plan for Spam" Published Aug. 2002. Accessed Sep. 28, 2007. Available online at http://www.paulgraham.com/spam.html.|
|23||*||Paul Graham. "Better Bayesian Filtering" Published Jan. 2003. Accessed Sep. 28, 2007. Available online at http://www.paulgraham.com/better.html.|
|24||*||Rennie, Jason. "ifile: An Application of Machine Learning to E-Mail Filtering." CMU, Dec. 1998. http://citeseer.ist.psu.edu/article/rennie98ifile.html.|
|25||Sergey Brin et al.; "Copy Detection Mechanisms for Digital Documents," Proc. Of ACM SIGMOD Annual Conference; San Jose, CA 1995; pp. 1-21.|
|26||*||Zhai, C. 1997. Fast statistical parsing of noun phrases for document indexing. In Proceedings of the Fifth Conference on Applied Natural Language Processing (Washington, DC, Mar. 31-Apr. 3, 1997). Applied Natural Language Conferences. Association for Computational Linguistics, Morristown, NJ, 312-319. DOI= http://dx.doi.org/10.3115/974557.9746.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7930306 *||Apr 19, 2011||Msc Intellectual Properties B.V.||System and method for near and exact de-duplication of documents|
|US7953679 *||Sep 9, 2009||May 31, 2011||Xerox Corporation||Scalable indexing for layout based document retrieval and ranking|
|US8001193 *||May 16, 2006||Aug 16, 2011||Ntt Docomo, Inc.||Data communications system and data communications method for detecting unsolicited communications|
|US8010538 *||May 8, 2006||Aug 30, 2011||Black Duck Software, Inc.||Methods and systems for reporting regions of interest in content files|
|US8073818 *||Oct 3, 2008||Dec 6, 2011||Microsoft Corporation||Co-location visual pattern mining for near-duplicate image retrieval|
|US8122032||Jul 20, 2007||Feb 21, 2012||Google Inc.||Identifying and linking similar passages in a digital text corpus|
|US8180773 *||May 27, 2009||May 15, 2012||International Business Machines Corporation||Detecting duplicate documents using classification|
|US8244711 *||Sep 28, 2009||Aug 14, 2012||Chin Lung Fong||System, method and apparatus for information retrieval and data representation|
|US8380697||Oct 21, 2010||Feb 19, 2013||Citizennet Inc.||Search and retrieval methods and systems of short messages utilizing messaging context and keyword frequency|
|US8391614 *||Jan 25, 2007||Mar 5, 2013||Equivio Ltd.||Determining near duplicate “noisy” data objects|
|US8423541||May 19, 2005||Apr 16, 2013||Google Inc.||Using saved search results for quality feedback|
|US8504550||May 17, 2010||Aug 6, 2013||Citizennet Inc.||Social network message categorization systems and methods|
|US8554854 *||Dec 13, 2010||Oct 8, 2013||Citizennet Inc.||Systems and methods for identifying terms relevant to web pages using social network messages|
|US8612293||Oct 19, 2011||Dec 17, 2013||Citizennet Inc.||Generation of advertising targeting information based upon affinity information obtained from an online social network|
|US8615434||Dec 20, 2011||Dec 24, 2013||Citizennet Inc.||Systems and methods for automatically generating campaigns using advertising targeting information based upon affinity information obtained from an online social network|
|US8650195||Mar 25, 2011||Feb 11, 2014||Palle M Pedersen||Region based information retrieval system|
|US8650199||Jun 25, 2012||Feb 11, 2014||Google Inc.||Document similarity detection|
|US8713034||Jun 3, 2011||Apr 29, 2014||Google Inc.||Systems and methods for identifying similar documents|
|US8793201 *||Oct 27, 2011||Jul 29, 2014||Amazon Technologies, Inc.||System and method for seeding rule-based machine learning models|
|US8843493 *||Sep 18, 2012||Sep 23, 2014||Narus, Inc.||Document fingerprint|
|US8874663 *||Aug 28, 2009||Oct 28, 2014||Facebook, Inc.||Comparing similarity between documents for filtering unwanted documents|
|US9002892||Aug 7, 2012||Apr 7, 2015||CitizenNet, Inc.||Systems and methods for trend detection using frequency analysis|
|US9031945||Apr 15, 2013||May 12, 2015||Google Inc.||Sharing and using search results|
|US9053497||Mar 15, 2013||Jun 9, 2015||CitizenNet, Inc.||Systems and methods for targeting advertising to groups with strong ties within an online social network|
|US9063927||Apr 6, 2012||Jun 23, 2015||Citizennet Inc.||Short message age classification|
|US9081852 *||Oct 1, 2008||Jul 14, 2015||Fujitsu Limited||Recommending terms to specify ontology space|
|US20090094020 *||Oct 1, 2008||Apr 9, 2009||Fujitsu Limited||Recommending Terms To Specify Ontology Space|
|US20100150453 *||Jan 25, 2007||Jun 17, 2010||Equivio Ltd.||Determining near duplicate "noisy" data objects|
|US20100306204 *||May 27, 2009||Dec 2, 2010||International Business Machines Corporation||Detecting duplicate documents using classification|
|US20110055332 *||Aug 28, 2009||Mar 3, 2011||Stein Christopher A||Comparing similarity between documents for filtering unwanted documents|
|US20110145348 *||Dec 13, 2010||Jun 16, 2011||CitizenNet, Inc.||Systems and methods for identifying terms relevant to web pages using social network messages|
|US20110202826 *||Aug 18, 2011||Canon Kabushiki Kaisha||Document creation support apparatus and document creation supporting method that create document data by quoting data from other document data, and storage medium|
|US20130018906 *||Jan 17, 2013||Aol Inc.||Systems and Methods for Providing a Spam Database and Identifying Spam Communications|
|US20130073510 *||Mar 21, 2013||Gang Qiu||Method for automatically retrieving and analyzing multiple groups of documents by mining many-to-many relationships|
|Dec 15, 2004||AS||Assignment|
Owner name: GOOGLE TECHNOLOGY INC.,CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TONG, SIMON;REEL/FRAME:016075/0170
Effective date: 20030613
|Dec 17, 2004||AS||Assignment|
Owner name: GOOGLE INC.,CALIFORNIA
Free format text: MERGER;ASSIGNOR:GOOGLE TECHNOLOGY INC.;REEL/FRAME:016081/0053
Effective date: 20030827
Owner name: GOOGLE INC., CALIFORNIA
Free format text: MERGER;ASSIGNOR:GOOGLE TECHNOLOGY INC.;REEL/FRAME:016081/0053
Effective date: 20030827
|Dec 9, 2013||FPAY||Fee payment|
Year of fee payment: 4