|Publication number||US7765215 B2|
|Application number||US 11/466,173|
|Publication date||Jul 27, 2010|
|Filing date||Aug 22, 2006|
|Priority date||Aug 22, 2006|
|Also published as||US20080059420|
|Publication number||11466173, 466173, US 7765215 B2, US 7765215B2, US-B2-7765215, US7765215 B2, US7765215B2|
|Inventors||Windsor Wee Sun Hsu, Soumyadeb Mitra|
|Original Assignee||International Business Machines Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (12), Referenced by (13), Classifications (7), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present invention generally relates to indexing records. In one aspect, the present invention pertains to a fast method of inserting records into an index. When used with Write-Once Read-Many (WORM) storage, the present invention ensures that records that have been inserted into the index cannot be modified or deleted by manipulating the index.
Records such as electronic mail (email), financial statements, meeting memos, experimental logs, and quality assurance documents are valuable assets. Key decisions in business operations and other critical activities are based on information in these records. Consequently, these records require maintenance in a trustworthy fashion that is safe from improper destruction or modification while keeping the records readily accessible. Businesses increasingly store these records electronically, making then relatively easy to delete and modify without leaving much of a trace. Ensuring that records are readily accessible, accurate, credible, and irrefutable is particularly imperative given recent legal and regulatory trends.
As critical data are increasingly stored in electronic form, it is imperative that the critical data be stored reliably in a tamper-proof manner. Furthermore, a growing subset of electronic data (e.g., email, instant messages, drug development logs, medical records, etc.) is subject to regulations governing long-term retention and availability of the data. Recent high-profiled accountability issues at large public companies have further caused regulatory bodies such as the Securities and Exchange Commission (SEC) to tighten their regulations. A requirement in many such regulations is that data must be stored reliably in non-erasable, non-rewritable storage such that the data, once written, cannot be altered or overwritten. Such storage is commonly referred to as WORM (Write-Once Read-Many) storage as opposed to WMRM (Write-Many Read-Many) storage, which can be written many times.
However, storing records in WORM storage is inadequate to ensure that the records are trustworthy, i.e., able to provide irrefutable evidence of past events. The key issue is that critical data requires some form of organization such that all of the data relevant to an enquiry can be promptly discovered and retrieved. Scanning all of the data in a large volume of data to discover entries that are relevant to an enquiry is not practical. Instead, some form of a direct access mechanism, such as an index must be built on the data for supporting efficient access.
If an index through which a record is accessed can be suitably manipulated, the record can, for all practical purposes, be hidden or deleted, even if the record is stored in WORM storage. For example, if the index entry pointing to the record is removed or made to point to a different record, the original record becomes inaccessible. Hence, the index itself must be maintained in a trustworthy fashion.
To address the need for a trustworthy index, fossilized indexes have been developed, that are impervious to such manipulations, when maintained on WORM. One such index is the generalized hash tree that supports exact-match lookups of records based on attribute values and hence is most suitable for use with structured data. Although such indexing schemes have proven to be useful, it would be desirable to present additional improvements. Most business records such as email, memos, meeting minutes, etc., are unstructured or semi-structured. The natural query interface for these records is feature (keyword) search, where the user provides a list of features and receives a list of records that contain some or all of the features. Feature based searches are handled by an inverted index.
An inverted index (or index) comprises a dictionary of features and, for each feature, an associated posting list of record identifiers and additional metadata such as feature frequency, feature type, feature position, etc. A trustworthy inverted index requires the posting list entries for a record and a path to those entries to be durable and immutable. This required immutability may be achieved by keeping each posting list in an append-only object (e.g. block, file) in WORM storage. The index can be updated when a new record is added by appending a record identifier (ID) of the new record to the posting lists of all the features contained in the new record. However, this operation can be prohibitively slow, as each append may require a random I/O. For an exemplary set of records in which a record comprises 500 features on average and an append incurs a two msec random I/O, the index update rate could be 1 doc per second.
Conventional approaches for supporting inverted index updates amortize the cost of random I/O, by buffering the index entries of the new records in memory or disk and committing these index entries to the index in batches. Specifically, the features of newly arriving records are appended to an in-memory or on-disk log comprising <feature, record ID> pairs. This log is periodically sorted on feature to create an inverted index for the new records, which is then merged with the original inverted index. Although this technology has proven to be useful, it would be desirable to present additional improvements. Researchers have found that this strategy is effective primarily when a large number of index entries are buffered. For example, over 100,000 records might have to be buffered to achieve an index update rate of 2 records per second.
Buffering creates a time lag, about half a day for the previous example, between the time a record is created to the time the index is updated to include the record. This time lag is inconsistent with maintaining a trustworthy index. Such a time lag provides a window in which an adversary can modify the index by, for example, deleting an index entry while it is still in the buffer, crashing the indexing system and deleting the recovery logs of uncommitted posting list entries, etc.
Keeping the recovery logs on WORM storage also does not guarantee the trustworthiness of the inverted index. Scanning the entire log on every restart is inefficient, while relying on an end-of-committed-log marker is insecure. An adversary can append markers to fool the application into believing that no recovery is required.
The time lag between when a record is compiled and when an adversary may regret the existence of the record is domain-specific and has no a priori lower bound. Furthermore, any delay in committing index entries introduces unnecessary risk and complexity in the compliance process. For example, the prevailing interpretation of e-mail retention regulations is that a regulated e-mail is required to be committed as a record before it is delivered to a mailbox. Thus, generic compliance indexing should not assume any safe time window for committing index entries after returning to the invoking application. A trustworthy index should be updated online, as new records are added.
Search engines answer multi-keyword conjunctive queries (queries in which more than one of the features are required to be contained in the record) by calculating the intersection of the posting lists of the query keywords. To speed up these intersections, additional index structures such as B+ trees are typically maintained on the posting lists. An adversary can effectively conceal a record if the record can be omitted from these posting list indexes and hence such index structures must also be secured by fossilization. Researchers have shown that index structures like B+ trees cannot be fossilized easily. Hence, although B+ trees have proven to be useful in conventional setting, they cannot be directly used in a trustworthy index.
Conventional secure indexing systems, such as Merkle hash trees, authenticated dictionaries etc, have been developed for a threat model in which the data store is untrusted. Merkle hash tree lets one verify the authenticity of any tree node entry by trusting the signed hash value stored at the root node. Authenticated dictionaries support secure lookup operations for dictionary data structures. These conventional systems rely on the data owner to sign data and index entries appropriately. In our model, the all powerful adversary (for example CEO) can assume the identity of the data owner and modify the data/indexes by re-signing them. Hence, although these technologies have proven to be useful in specific threat models, they are inapplicable here.
What is therefore needed is a system, a computer program product, and an associated method for providing inverted index to enable searching of records. The trustworthy inverted index should prevent hiding or modifying of a record through modification of the inverted index. The trustworthy inverted index should be relatively inexpensive with respect to random I/Os and require no time lag between commit of a record and update of the inverted index to include the record. The need for such a solution has heretofore remained unsatisfied.
The present invention satisfies this need, and presents a system, a service, a computer program product, and an associated method (collectively referred to herein as “the system” or “the present system”) for providing a trustworthy inverted index to enable searching of records.
The present system processes records (e.g. documents, images, audio objects, video objects, etc) to identify features (e.g. terms, keywords, tones, colors, shapes, etc) for indexing in the trustworthy inverted index and generates posting lists such that each of the posting lists corresponds to at least one of the identified features. The present system reduces the random I/Os required to insert records into the index, i.e. update the index, by effectively utilizing a storage cache. For this, is maps one or more features onto each posting list such that the tail block of all the posting lists fit largely in the storage cache. Examples of block size include, but are not limited to, 512 B, 4KB, 8KB, 16KB, 32KB, 64KB.
The mapping strategy is decided based on the record insertion performance, a query performance, and a size of the storage cache. In one embodiment, the present system maps multiple features that occur infrequently in the records and/or the queries onto the same posting list. In another embodiment, the features are randomly mapped onto the posting lists.
The present system searches the posting lists corresponding to a search feature in a query to identify records that contain the search feature.
The present system maintains an index structure over the posting lists for supporting faster join operation. The index structure exploits the fact that the record identifiers (IDs) in a posting list form an increasing sequence. IDs inserted into the index structure are not relocated and the path through the index structure to the ID is immutable. The index structure relies on jump pointer maintained with the index entries for supporting efficient lookup operations. To insert and ID into the index structure, the ID is inserted at a root node of the index structure. This insertion comprises a comparison between the ID and a reference ID at the root node of the index structure. If the insertion is unsuccessful, the present system repeats the insertion process at a target node in the index structure until the ID is successfully inserted into the index structure. If the target node does not exist, the present system generates a new node. In one embodiment, the index has a tree-like structure.
In one embodiment, the target node is identified based on a mathematical difference between the ID and a reference ID of the root node of the index structure. In another embodiment, the identification of the target node comprises using a logarithm of the mathematical difference between the ID and the reference ID of the root node. In yet another embodiment, the reference ID of the root node is the largest ID stored in the root node.
The various features of the present invention and the manner of attaining them will be described in greater detail with reference to the following description, claims, and drawings, wherein reference numerals are reused, where appropriate, to indicate a correspondence between the referenced items, and wherein:
The following definitions and explanations provide background information pertaining to the technical field of the present invention, and are intended to facilitate the understanding of the present invention without limiting its scope:
Record: any type of structured, unstructured, or semi-structured object comprising features that may be indexed such as, for example, e-mails, office documents, financial statements, meeting memos, experimental logs, instant messages, drug development logs, medical records, quality assurance documents, images, audio objects, video objects, multimedia objects, closed captioning, etc.
Inverted Index: An index comprising a dictionary of features and, for each feature, an associated posting list of record IDs and additional metadata such as feature frequency, feature type, feature position in the record, etc.
Trustworthy: incapable of modification or manipulation attempted in order to modify or hide a record.
System 10 can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment containing both hardware and software elements. In one embodiment, system 10 is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.
Furthermore, system 10 can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include a compact disk that can be a read only memory (CD-ROM), or a read/write (CD-R/W) disk, and a DVD.
A data processing system suitable for storing and/or executing program code includes at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories that provide temporary storage of at least some program code to reduce the number of times code is retrieved from bulk storage during execution.
Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers.
Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
Computer 15 comprises the indexing system 10 and is connected to the storage system 20. Storage system 20 comprises a storage cache 25 and storage media such as hard disks 60 and tapes 65. In one embodiment, the storage system 20 includes WORM media. Computer 15 is connected to the storage system 20 through a network 80. Clients such as clients 70 access system 10 through a network 85 or directly run on computer 15. In one embodiment, the storage cache 25 is located in computer system 15. In one embodiment, the inverted index system is located in storage system 20.
Each of the posting lists 245 comprises a list of the record IDs of the records in which the associated feature appears. For example, data posting list 340 comprises ID 3, 360, ID 9, 365, and ID 31, 370. Base posting list 345 comprises ID 3, 375, and ID 19, 380. ID 3, 375, indicates that record 3 contains the associated feature of the index dictionary 240. For example, record 3 contains the terms data 310 and base 315, as indicated by ID 3, 360, and ID 3, 375. Each entry in the posting lists 245 may further contain additional metadata such as feature frequency, feature type, feature position, etc. In one embodiment, each posting list is stored as at least one append-only object (e.g. block, file) in WORM storage.
System 10 reduces random I/Os by mapping multiple features onto the same posting list. With reference to
The encoding can be stored in log(q) bits, where q is the number of posting lists 245 that are merged together. In one embodiment, the overhead is further reduced by using an encoding scheme such as, for example, Huffman encoding. This encoding is added to the metadata for each entry in the merged posting lists 245.
System 10 maps features onto posting lists 245 such that the number of active posting lists 245 is largely equal to the number of blocks 430 in the storage cache 25. As illustrated in
As the blocks 430 are filled and/or evicted from the cache, contents of the evicted blocks are written to the corresponding posting lists 245 on disk. Entries are added to the tails 460 as system 10 processes records in which an associated index feature 330 is identified.
Because the tail blocks are maintained in the storage cache 25, most of the updates to the posting lists 245 hit the storage cache 25. A disk random I/O is incurred only when the contents of the evicted block are written to disk, as compared to requiring a random I/O for each entry written to each of the posting lists 245. Furthermore, by merging posting lists 245 that are smaller than one block into a larger posting list 245, the number of random I/Os is further reduced by decreasing a number of partial blocks written to disk.
System 10 maps features onto posting lists 245 based on frequency of occurrence of the index features 330 in records, queries, or records and queries. In a different embodiment, the index features are mapped onto posting lists randomly. Stated differently, the posting lists are merged randomly.
Queries are answered by scanning the posting lists 245 corresponding to the features in the query. The records in the posting lists 245 are assigned scores based on measures such as, for example, cosine or Okapi BM-25. The scores are used to rank the records.
The total workload cost for answering a set of queries can hence be modeled as follows: Let qi be the number of queries that contain the ith feature (the query frequency of the ith feature), and let ti be the number of occurrences of the ith feature or, alternatively, the number of records containing the ith feature (the feature frequency of the ith feature). The total workload cost without any merging is proportional to
To minimize index update time, system 10 merges the posting lists 245 into M lists, where M is the number of blocks 430 available in the storage cache 25. Merging the posting lists 245 into M lists, A1, . . . AM, yields a workload cost proportional to
In general, a small fraction of the index features 330 account for most of the workload cost. Consequently, system 10 merges posting lists 245 for index features 330 that are infrequently queried or infrequently used, while not merging posting lists 245 for index features 330 that are frequently queried or frequently used. In one embodiment, the statistics manager 225 and the map/merge strategy module 230 are provided with estimates of the feature frequency ti and the query frequency qi.
In one embodiment, system 10 learns the frequencies online as records are processed, and adapts the mapping/merging strategy accordingly. In one embodiment, system 10 divides time into epochs and maintains a separate index for the records inserted in each epoch. The choice of posting lists 245 to merge in any particular epoch can be determined by the statistics collected during the previous epoch. Queries are answered by scanning the index 235 of all epochs. By keeping the epoch interval large enough, system 10 can keep the number of separate versions of index 235 manageable. Furthermore, query interfaces typically support query conditions on record creation time (e.g., to retrieve only records created in 2004). For such queries, system 10 only needs to consider any index 235 associated with an epoch that overlaps with the time interval specified in the query. The exact time range restrictions can be checked in a filtering phase after the feature search.
In a different embodiment, system 10 merges the terms randomly. When the qi and ti follow the Zipfian distribution, very few terms have high ti or qi. As long as the posting lists of the popular terms do not get merged with each other, merging does not slow down query response significantly. If the number of posting lists is substantially larger than the number of popular terms (as with the larger cache sizes), these unlucky merges are unlikely to occur.
In one embodiment, the target posting list is determined based on the value of a hash function applied to the extracted feature. An example hash function is to sum all the characters in a term modulo the target number of posting lists.
In one embodiment, the target posting list 245 for the extracted feature is determined based on the feature frequency ti and the corresponding query frequency qi. In one embodiment, a feature with feature frequency ti and/or query frequency qi above a predetermined threshold is mapped to its own unique posting list, i.e. it is not merged. In one embodiment, a feature with feature frequency ti and/or query frequency qi ranked above some percentage of the corresponding population frequencies is mapped to its own unique posting list. In one embodiment, a remaining feature is mapped to a posting list determined based on the value of a hash function applied to the feature.
In one embodiment, the frequencies are learned online as records are processed. In one embodiment, the method divides time into epochs and maintains a separate index for the records inserted in each epoch. In any particular epoch, the target posting list is determined based on the statistics collected during the previous epoch.
In one embodiment, step 825 includes performing a ZIG-ZAG join over more than one of the identified associated posting lists to answer conjunctive queries, which are queries in which more than one of the search features are required to be present in a record for the record to be returned in response to the query). The ZIG-ZAG algorithm skips over portions of the posting list which cannot appear in the join result, by calling the FindGeq( ) operation. The pseudo-code for ZIG-ZAG join is given below:
1: top1 = list1.Start( );
2: top2 = list2.Start( );
4: if ((top1==list1.End( )) ∥ (top2==list2.End( )))
7: If (*top1 < *top2)
8: top1 = list1.FindGeq (*top2);
11: If (*top2 < *top1)
12: top2 = list2.FindGeq (*top1);
15: If (*top1 == *top2)
16: OUTPUT (*top1);
17: top1 = list1.FindGeq(*top1+1);
18: top2 = list2.FindGeq(*top2+1);
The query runtime module 220 first sets the posting list iterators to start of the two posting lists (steps 1, 2). The query runtime module 220 checks if either of the iterators have reached the end of the posting lists (step 4). If this is the case, there are no further query results. If this is not the case, the query runtime module 220 compares the two posting list entries pointed to by the iterators—If they are equal (step 15), the module outputs the matching entries as the next joined result (step 16). If they are not equal, the module advances the iterator pointing to the smaller entry to an entry greater than or equal to the larger entry (using the FindGeq( ) call) (steps 8, 12).
System 10 supports FindGeq( ) in a maximum number of O(log(N)) operations using a jump index, where N is the largest number in the sequence. This bound is generally weaker than the O(log(n)) bound for B+ tree lookups, where n is the number of entries in the tree. However, in system 10, N is equal to the number of stored records, because record IDs are assigned through an increasing counter. Hence, the bound for the jump index in system 10 is logarithmic in the number of records.
The jump index structure exploits the fact that the record IDs in a posting list form an increasing sequence. The jump index provides a trustworthy lookup mechanism because IDs inserted into the index structure are not relocated and the path through the index structure to the Id is immutable.
The intuition behind jump pointers in system 10 is that one can get to any number k≦N in O(log2(N)) steps by taking jumps in powers of 2. Consider a sequence of numbers 0, . . . N−1, where N=2p. Let b1 . . . bp be the binary representation of an integer 0≦k<N. One can reach k in p steps by starting at the beginning of the sequence then successively jumping b1*2p−1 places, then b2*2p−2 places, and so on until bp*20 jump forward arrives at k.
System 10 applies this approach to posting lists 245. The posting list 245 does not contain every record ID, so system 10 stores explicit jump pointers that indicate how far ahead to jump. The ith jump pointer stored with a list entry points to the smallest list entry I′ such that
I+2i ≦I′<I+2i+1. (3)
Shaded jump pointers that point to valid entries are shown with the origination of an arrow, e.g., the jump pointer 0, 940. Shaded pointers that are not shown with the origination of an arrow (e.g., the jump pointer 1, 945) are set to null. Unshaded pointers such as the jump pointer 4, 60, have never been written.
The jump pointer 0, 940, points to the posting list entry 1, 910, because the record ID entry for the posting list entry 0, 905, is 1 and the record ID for the posting list entry 1, 910, is 2 (using equation 3):
The jump pointer 2, 950, points to posting list entry 2, 915, because the record ID for the posting list entry 0, 905, is 1 and the record ID entry for the posting list entry 2, 950, is 5 (using equation 3):
Additional jump pointers are determined similarly.
More generally, let the entries of the posting lists 245 be n1, . . . , nN. System 10 can look up an entry by following jump pointers from the smallest number in the sequence. To look up an entry, e.g. n, system 10 finds l1 such that n1+2i
In one embodiment, system 10 reduces the overhead of storing the jump pointers and the depth of the index, which impacts performance, by storing posting entries together in block of size L and associating pointers with blocks, rather than with every entry.
n b+2i ≦n<n b+2i+1 (4)
The index insert module 210 determines whether the ith pointer of block b has been set (decision step 1025). If no, the index insert module 210 sets the ith pointer of block b to the block containing the appended record ID, n (step 1030). If the ith pointer of block b has been set (decision step 1025), the index insert module 210 selects a next block b (step 1035) and returns to step 1015. The index insert module 210 repeats steps 1015 through 1035 until the pointer is set.
Pseudocode for FindGeqRec(k, s) (interchangeably referenced herein as FindGeqRec( )), a function implementing FindGeq(k) is as follows:
21: IF (s ≧ k) THEN
22: RETURN s
23: END IF
24: Find i ≧ 0 such that s + 2i 2k2s + 2i + 1
25: IF (ptr[i] ≠ NULL) THEN
26: t ← record ID at ptrs[i]
27: ASSERT s + 2i 2t2s + 2i + 1
28: res ← FindGeqRec(k,t)
COMMENT Recursively call FindGeqRec( ) by following the ptr
29: IF (res ≠ NOT_FOUND) THEN
30: ASSERT s + 2i 2res2s + 2i + 1
31: RETURN res
32: END IF
33: END IF
COMMENT: No number ≧ k is found by following ith ptr. RETURN
the first non-null ptr.
34: i ← i + 1
35: WHILE (i < log2(N)) DO
36: IF (ptrz[i] ≠ NULL) THEN
37: t ← record ID at ptrs[i]
38: ASSERT s + 2i 2t 2s + 2i +
39: RETURN t
40: END IF
41: i ← i + 1
42: END WHILE
43: RETURN NOT_FOUND
Pseudocode for FindGeq(k) (interchangeably referenced herein as FindGeq( )), to find number≧k is as follows:
1: RETURN FindGeqRec (k, the smallest number in the sequence)
The jump index also supports an insert operation Insert(k) (interchangeably referenced herein as Insert( )), for inserting ID k into the index 900 (
1: IF index is empty THEN
2: Create a new index with a node containing k
4: END IF
5: s ← the smallest record ID in the index
6: ASSERT s < k
COMMENT: The index entries are required to be monotonically
8: Find 1 ≧ 0, such that s + 2i 2k < s + 2i +1
9: IF (ptrs[i]==NULL) THEN
10: Create a new index node containing k
11: ptr [i] ← ks location // Set the ith pointer of s
12: RETURN DONE
14: s← the record ID at ptrs[i]
COMMENT: Follow the pointer to a new s
15: ASSERT s< k
16: s ← s
18: END IF
19: END LOOP
1: s ← the smallest record ID in the index
3: IF s > k
4: RETURN NOT_FOUND
5: END IF
6: IF s==k THEN
7: RETURN FOUND
8: END IF
9: Find i ≧ 0 such that s + 2i 2k2s + 2i +1
10: IF ptrs[i]==NULL THEN
11: RETURN NOT_FOUND
13: s← the record ID atptrs[i]
COMMENT: Follow the pointer to a new s
14: ASSERT s + 2i 2k2s + 2i +1
15: s ← s1
17: END IF
18: END LOOP
Proposition 1: Let i1, . . . , ij be the values of i selected in step 9 in successive iterations of the loop in Lookup(k), previously discussed. Then i1>L>ij.
Proof: Let the record IDs whose jump nodes are visited by Lookup(k) be s1, . . . , sj, where s1 is the smallest number in the posting list. From step 9 Lookup(k), s1+2i
From step 9 of Lookup(k), i1≦└log2(k)┘+1. Thus it takes at most └log2(k)┘+1 jumps to find k. It can similarly be argued that Insert ( ) and FindGeq( ) also require O(log2(k)) pointer follows. If there are N records in the index 235, the complexity of the operation is, therefore O(log2(N)).
A straightforward approach to storing jump pointers in a WORM device is to maintain each node of the index in a separate disk block. Because of the monotonicity property of record IDs, the pointers are also set in increasing order; i.e., ptrs[i] is always set after ptrs[i′] if i′<i. Hence the pointer assignment operation can also be implemented as an append operation. Under this approach, indexes and their associated posting lists have the following properties.
Proposition 2: Once and ID has been inserted into an index and the associated posting list, it can always be looked up successfully.
Proof Outline: The pointers set up during Insert ( ) (step 11 in pseudocode for Insert ( ), previously discussed) are written to WORM, so the pointers and the entries cannot be altered afterwards. The values of i(i1, . . . , ij) selected by Insert ( ) are the same as those selected by Lookup( ). Hence an entry that has been inserted is always visible to Lookup( ). QED
Proposition 3: Let v be an ID in the posting list 245. If k≦v, then FindGeq(k) does not return a value greater than v.
Proof Outline: Suppose the path to v in the index is through jump pointers j1,K,jo
To show that i1≦j1, consider the initial call to FindGeqRec( ). In line 4 of FindGeqRec( ), i≦j1 is chosen, as k≦v. If the checks in line 5 and line 9 of FindGeqRec( ) succeed for that i, i1=i and hence i1≦j1. If either of the checks in line 5 or line 9 of FindGeqRec( ) fails, FindGeqRec( ) selects an i on line 15. However, since v is in the index, prt0[j1] is not NULL. Therefore, system 10 does not go beyond j1, hence i1≦j1. Now the following cases arise: (i)i1<j1 and (ii) i1=j1. In the case of i1<j1, I<v (I<s0+2i
Proposition 3 ensures that no record ID can be hidden when joining two posting lists 245. Consider a record ID d present in both posting lists being joined. The join starts from the smallest numbers in the lists and makes successive calls to FindGeq( ). Proposition 3 ensures that no number greater than d can be returned in a FindGeq( ) call, before d is returned. In other words, d is eventually returned in a FindGeq( ) call on both the lists and hence appears in the result.
In one embodiment, system 10 defines jump pointers using powers of B rather than powers of two, where p≧B. System 10 maintains (B−1)logB(N) pointers with every block, with each pointer uniquely identified by a pair (i,j), where 0≦i<logB(N) and 1≦j<B. The pointers are set up as follows: Let I1 be the largest record ID stored in block b. The (i,j) pointer in b points to the block containing the smallest record ID s such that
I 1 +jB i ≦s<I 1+(j+1)B i. (5)
The largest number in block 0 1205 is 7, in posting list entry 1216. The jump pointer (0,1) 1220 points to the posting list block 1, 1204, because the posting list block 1, 1204, contains 8 (using equation 5):
The number 8 in the posting list block 1, 1204, is the smallest number satisfying that constraint. Similarly, the jump pointer (2,2) 1230 of the posting list block 0, 1202, points to the posting list block 2, 1206, because the posting list block 2, 1206, contains 25 (using equation 5):
In this embodiment, the index insert module 210 comprises pseudocode Insert_block(i) (interchangeably referenced herein as Insert_block( )) for inserting record id k in index 235. The query runtime module 220 comprises pseudocode Lookup_block( ) for locating record ID k in a block in posting lists 245.
Pseudocode Insert_block(k) for inserting record id k in the index 235 is as follows:
1: last_block ← last block in the index
2: If last block is full (has p entries), allocate a new block and set
last_block to new block.
3: Append (k) to the last_block.
4: b ← initial_block
6: IF (b==last_block) THEN
7: RETURN DONE
8: END IF
9: n ← the largest ID in block b
10: ASSERT nb< k
11: Find (i,j) such that 02i < logR(N),1 ≦ j < B, and
nb+ j * Bi 2k < nb + (j + 1) * Bi
12: IF ptrb[i,j] ≠ NULL THEN
13: b ← ptrb [i,j]
COMMENT: ptrb[i,j] is the (i,j)th pointer in block b
16: ptrb [i,j] ← last_block
18: END IF
19: END LOOP
Pseudocode for Lookup_block(k) (interchangeably referenced herein as Lookup_block( )) for inserting record id k in index 235 is as follows:
1: ← The number being searched for.
2: b ← the initial block of the index
4: nb ← the largest ID in block b
5: IF (k ≦ nb) THEN
6: Search for k in b, and RETURN FOUND or NOT_FOUND
7: END IF
8: Find (i,j) such that 02 i < logB(N),1 ≦ j < B, and
nb + j * Bi 2k < nb + (j + 1) * Bi
9: IF ptrb[i,j] ≠ NULL THEN
10: b ← ptrb [i,j]
COMMENT: ptrb[i,j] is the (i,j)th pointer in block b
13: RETURN NOT_FOUND
14: END IF
15: END LOOP
As when B=2, the pointer set operation in step 15 of Insertblock( ) can be implemented by an append operation. As with B=2, one can show that if the lookup proceeds by following pointers i1, . . . , ik, then i1< . . . <ik. This gives a bound of logB(N) jumps for Lookup( ).
In one application of system 10, system 10 is used to index time sequences. System 10 may be used to index and application in which the entries are always increasing.
It is to be understood that the specific embodiments of the invention that have been described are merely illustrative of certain applications of the principle of the present invention. Numerous modifications may be made to the system and method for providing a trustworthy inverted index to enable searching of records described herein without departing from the spirit and scope of the present invention. Moreover, while the present invention is described fir illustration purpose only in relation to records and WORM storage, it should be clear that the invention is applicable as well to, for example, any type of record stored in any type of storage medium.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5852822||Dec 9, 1996||Dec 22, 1998||Oracle Corporation||Index-only tables with nested group keys|
|US5913208 *||Jul 9, 1996||Jun 15, 1999||International Business Machines Corporation||Identifying duplicate documents from search results without comparing document content|
|US6349308||Feb 15, 1999||Feb 19, 2002||Korea Advanced Institute Of Science & Technology||Inverted index storage structure using subindexes and large objects for tight coupling of information retrieval with database management systems|
|US7155460 *||Mar 18, 2003||Dec 26, 2006||Network Appliance, Inc.||Write-once-read-many storage system and method for implementing the same|
|US7363284 *||Apr 30, 2004||Apr 22, 2008||Unisys Corporation||System and method for building a balanced B-tree|
|US7487178 *||Oct 5, 2005||Feb 3, 2009||International Business Machines Corporation||System and method for providing an object to support data structures in worm storage|
|US20020174355 *||Mar 11, 2002||Nov 21, 2002||Arcot Systems, Inc.||Techniques for searching encrypted files|
|US20040015909||May 8, 2001||Jan 22, 2004||International Business Machines Corporation||System and method for regular expression matching using index|
|US20050198076||Oct 15, 2004||Sep 8, 2005||Stata Raymond P.||Systems and methods for indexing content for fast and scalable retrieval|
|US20060053157 *||Sep 9, 2005||Mar 9, 2006||Pitts William M||Full text search capabilities integrated into distributed file systems|
|US20070050384 *||Aug 9, 2006||Mar 1, 2007||Korea Advanced Institute Of Science And Technology||Two-level n-gram index structure and methods of index building, query processing and index derivation|
|US20070185836 *||Feb 9, 2006||Aug 9, 2007||Handy-Bosma John H||Method for caching faceted search results|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8478704||Nov 22, 2010||Jul 2, 2013||Microsoft Corporation||Decomposable ranking for efficient precomputing that selects preliminary ranking features comprising static ranking features and dynamic atom-isolated components|
|US8620907||Nov 22, 2010||Dec 31, 2013||Microsoft Corporation||Matching funnel for large document index|
|US8645316 *||Aug 16, 2011||Feb 4, 2014||International Business Machines Corporation||Storing records in databases in a randomized manner to effectively utilize database servers|
|US8650153 *||Feb 25, 2012||Feb 11, 2014||International Business Machines Corporation||Storing records in databases in a randomized manner to effectively utilize database servers|
|US8713024||Nov 22, 2010||Apr 29, 2014||Microsoft Corporation||Efficient forward ranking in a search engine|
|US8762387 *||Jul 31, 2013||Jun 24, 2014||Linkedin Corporation||Inverted indexes for accelerating analytics queries|
|US9104748||Oct 21, 2011||Aug 11, 2015||Microsoft Technology Licensing, Llc||Providing a search service including updating aspects of a document using a configurable schema|
|US9152697||Jul 13, 2011||Oct 6, 2015||International Business Machines Corporation||Real-time search of vertically partitioned, inverted indexes|
|US9171031 *||Feb 25, 2013||Oct 27, 2015||Cleversafe, Inc.||Merging index nodes of a hierarchical dispersed storage index|
|US9171062||Sep 13, 2012||Oct 27, 2015||International Business Machines Corporation||Real-time search of vertically partitioned, inverted indexes|
|US20090112843 *||Oct 29, 2007||Apr 30, 2009||International Business Machines Corporation||System and method for providing differentiated service levels for search index|
|US20130046729 *||Feb 21, 2013||International Business Machines Corporation||Storing records in databases in a randomized manner to effectively utilize database servers|
|US20130232306 *||Feb 25, 2013||Sep 5, 2013||Cleversafe, Inc.||Merging index nodes of a hierarchical dispersed storage index|
|International Classification||G06F17/00, G06F17/30|
|Cooperative Classification||G06F17/30631, G06F21/64|
|European Classification||G06F21/64, G06F17/30T1P9|
|Aug 22, 2006||AS||Assignment|
|Mar 7, 2014||REMI||Maintenance fee reminder mailed|
|Jun 20, 2014||FPAY||Fee payment|
Year of fee payment: 4
|Jun 20, 2014||SULP||Surcharge for late payment|
|Mar 13, 2015||AS||Assignment|
Owner name: LINKEDIN CORPORATION, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INTERNATIONAL BUSINESS MACHINES CORPORATION;REEL/FRAME:035201/0479
Effective date: 20140331