US 6601216 B1 Abstract A method for generating error check bits in a digital system is suited for use in a system in which a previously stored data page is modified. The method comprises retrieving a previously computed set of error check bits, which we denote CRC(old page), for a previously stored page of data; and then, upon receiving an indication that the “old page” has been modified, updating the error check bits by incrementally updating CRC(old page) in accordance with the equation,
CRC(new page)=CRC(old page) XOR CRC(Δ page),where CRC(Δ page) is computed by taking advantage of the fact that Δ page (i.e., a page where all bits are
0 except for those bit positions where the data has been modified) is sparse.Claims(18) 1. A method for generating error check bits in a digital system, comprising:
storing a page of digital information (old page) and associated CRC check bits (CRC(old page));
receiving an indication that a new page has been created from the old page by modification of at least some of the bits making up the page of digital information;
forming a page of data (Δ page) indicative of the changed bits; and
computing CRC check bits (CRC(new page)) for the new page based on CRC(old page) and CRC check bits for Δ page CRC(Δ page), wherein CRC(Δ page) is obtained from a pre-computed lookup table containing CRC check bits for predefined error patterns.
2. A method as recited in
3. A method as recited in
4. A method as recited in
5. A method as recited in
6. A method as recited in
CRC(new page)=CRC(old page) XOR CRC(Δ page). 7. A method for generating error check bits in a digital system, comprising:
storing a page of digital information (old page) and associated CRC check bits (CRC(old page));
receiving an indication that a new page has been created from the old page by modification of at least some of the bits making up the page of digital information;
forming a page of data (Δ page) indicative of the changed bits, wherein Δ page is sparse; and
computing CRC check bits (CRC(new page)) for the new page based on CRC(old page) and Δ page, wherein CRC(new page) is computed in accordance with the equation,
CRC(new page)=CRC(old page) XOR CRC(Δ page); and further comprising a first step involving pre-computing a lookup table, wherein the lookup table is formed by pre-computing and storing in the lookup table CRC bits for prescribed pages (P
1, P2, P3, etc.). 8. A method as recited in
finding intervals containing changed bits;
for each such interval, looking up a table entry and forming a CRC polynomial f(x) based on the table entry and a polynomial Δ(x) based on Δ page;
forming the product f(x)*Δ(x);
computing CRC(Δ page)=CRC(f(x)*Δ(x)); and
computing CRC(new page)=CRC(old page) XOR CRC(Δ page).
9. A computer readable storage medium comprising code for instructing a computer in carrying out the steps recited in
10. A computer readable storage medium comprising code for instructing a computer in carrying out the steps recited in
11. A digital system, comprising:
a computer;
a computer readable storage medium operatively coupled to the computer; and
computer code executable by the computer for carrying a differential algorithm for generating error check bits, wherein said algorithm comprises:
i. storing in the storage medium a page of digital information (old page) and associated CRC check bits (CRC(old page));
ii. receiving an indication that a new page has been created from the old page by modification of at least some of the bits making up the page of digital information;
iii. forming a page of data (Δ page) indicative of the changed bits; and
iv. computing CRC check bits (CRC(new page)) for the new page based on CRC(old page) and CRC check bits for Δ page CRC(Δ page), wherein CRC(Δ page) is obtained from a pre-computed lookup table containing CRC check bits for predefined error patterns.
12. A system as recited in
13. A system as recited in
14. A system as recited in
15. A system as recited in
16. A system as recited in
CRC(new page)=CRC(old page) XOR CRC(Δ page). 17. A digital system, comprising:
a computer;
a computer readable storage medium operatively coupled to the computer; and
computer code executable by the computer for carrying a differential algorithm for generating error check bits, wherein said algorithm comprises:
storing in the storage medium a page of digital information (old page) and associated CRC check bits (CRC(old page));
receiving an indication that a new page has been created from the old page by modification of at least some of the bits making up the page of digital information;
forming a page of data (Δ page) indicative of the changed bits, wherein Δ page is sparse; and
computing CRC check bits (CRC(new page)) for the new page based on CRC(old page) and Δ page, wherein CRC(new page) is computed in accordance with the equation
CRC(new page)=CRC(old page) XOR CRC(Δ page); wherein the algorithm further comprises a first step involving pre-computing a lookup table, wherein the lookup table is formed by pre-computing and storing in the lookup table CRC bits for prescribed pages (P
1, P2, P3, etc.). 18. A system as recited in
finding intervals containing changed bits;
for each such interval, looking up a table entry and forming a CRC polynomial f(x) based on the table entry and a polynomial Δ(x) based on Δ page;
forming the product f(x)*Δ(x);
computing CRC(Δ page)=CRC(f(x)*Δ(x)); and
computing CRC(new page)=CRC(old page) XOR CRC(Δ page).
Description The present invention relates generally to the field of error detection in digital systems. More particularly, the invention relates to an improved error detection algorithm that is particularly useful in, although not limited to, applications involving the detection of errors in database systems and file systems, which are characterized by relatively small modifications made to previously stored pages of data. Digital devices are often required to transmit or store data on unreliable media. For example, bits transmitted on a phone line via modem or written to a disk may not necessarily be identical to those received on the other end of the line or read from the disk at a later time due to electronic interference, physical damage, etc. Since many such devices, computers in particular, are highly sensitive to the accuracy of the data they process, it is desirable to have a mechanism for detecting errors in data storage and transmission. This gives the device an opportunity to deal with the erroneous data gracefully. For example, a program may ignore or attempt recovery of a corrupt page on a disk, or a modem may request a retransmission of a damaged chunk of a transmission. Error detection is often accomplished by including some number of error check bits along with the content bits. The error check bits are some function of the content bits. The check bits are recomputed on the receiving end, and compared with the transmitted check bits. The relationship between the content and check bits is such that, if an error occurred during transmission, with high probability the computed check bits will be inconsistent with the transmitted check bits and the error will be detected. One of these techniques, the cyclic redundancy check (CRC), is probably the most widely used error detection method in the world. For instance, a version is used in the U.S. communication standard ANSI X3.66 and the European CCITT X.25, as well as in countless software products. The X3.66 standard refers to the CRC algorithm as the Frame Check Sequence, or FCS, which is described in Appendix D of ANSI 3.66-1979. Current methods for computing the CRC accept a sequential stream of input data, compute the CRC, and emit the CRC check bits. This arrangement is convenient for many applications. However, suppose that one wishes to maintain a CRC for each page of a database file (a page is some convenient chunk of storage; for example, a typical page size is 4192 bytes). Each time data on the page is modified, the CRC must be recomputed in order to be made consistent with the new data. Current methods for doing this require starting from scratch, i.e., doing a sequential rescan of the entire page. What is needed is a more efficient method for computing the error check bits, so that the complete CRC process does not have to be performed when only a small portion of the original data is modified, as in a database or file system. The present invention provides a method for generating error check bits in a digital system. The invention is especially suited for use in a system in which a previously stored data page is modified. The inventive method comprises retrieving a previously computed set of error check bits, which we denote CRC(old page), for a previously stored page of data; and then, upon receiving an indication that the “old page” has been modified, updating the error check bits by incrementally updating CRC(old page) in accordance with the equation,
where CRC(Δ page) is computed by taking advantage of the fact that Δ page (i.e., a page where all bits are Other aspects of the present invention are described below. The foregoing summary and the following detailed description of presently preferred embodiments are better understood when read in conjunction with the appended drawings, in which: FIG. 1 is a block diagram representing a general purpose computer system in which aspects of the present invention may be incorporated. FIG. 2 schematically depicts a digital system employing the present invention. FIG. 3 is a flowchart of a presently preferred implementation of the present invention. The present invention provides a mechanism for performing a differential update of the CRC. While the following discussion describes an embodiment of the invention in which a desktop computer computes CRC check bits for disk files, the present invention is not limited to that use. The invention may also be employed on any type of computer network or individual computer having data stores for files, e-mail messages, databases and the like. The information from all such stores can be processed together or separately in accordance with the invention. The present invention will now be explained with reference to a presently preferred embodiment thereof. An overview of the invention is provided first. After this overview, a description of an exemplary computer environment is provided. Finally, a detailed description of the inventive differential CRC method is provided. The present invention provides a method for performing a differential update of the CRC. That is, given the set of changes to the page (the set of bytes to be written and the positions on the page) and the old CRC, it computes the resulting changes to the CRC check bits without having to look at any of the unchanged regions of the page. In a typical scenario in which only a relatively small fraction of a page is updated at a time, the differential update may be many multiples faster than re-computing the CRC from scratch. The inventive algorithm recognizes and takes advantage of the identity:
where “Δ page” is a page of “ The invention further involves an algorithm for efficiently computing CRC(Δ page), by taking advantage of the fact that Δ page will be sparse (mostly 0's). This algorithm for computing CRC(Δ page) involves pre-computing and memorizing (in a lookup table) CRC check bits for certain predefined error patterns. The value of CRC(Δ page) is determined from the following equation:
where f(x) and Δ(x) are polynomials, f(x) being the polynomial form of the look-up table entry corresponding to the region in which the page was modified and Δ(x) being the polynomial form of the modified data region. One embodiment of a system employing the present invention is schematically depicted in FIG. 2, discussed below, which illustrates how the inventive Differential CRC algorithm generates new CRC bits (denoted CRC(new page)) based on the old CRC (denoted CRC(old page)) and the sparse Δ page. The present invention may be implemented in a computer program executed on a computer. FIG. As shown in FIG. 1, an exemplary general purpose computing system includes a conventional personal computer A number of program modules may be stored on the hard disk, magnetic disk The personal computer When used in a LAN networking environment, the personal computer The cyclic redundancy check interprets data as coefficients of a large polynomial. The coefficients are drawn from a finite field of the same size as the number of characters in the data alphabet. For instance, when working with binary data, which expresses data using two symbols, 0 and 1, the coefficients are treated as integers modulo 2. The bit at position k is identified with the coefficient of the term x The computation of the CRC itself is simple: Let k be the degree of the check polynomial p(x) and let f(x) be the polynomial corresponding to the data to be transmitted. Compute the polynomial r(x) by taking x Referring to FIG. 2, the Differential CRC algorithm For convenience, we will use underlined type to indicate reduction modulo the check polynomial p(x) i.e. f(x) mod p(x)= Suppose that f(x) is the polynomial corresponding to a data page, and x.^{k}f′(x)Let Δ(x) denote the polynomial corresponding to the differences between the two pages, Δ(x)=f′(x)−f(x). Then, where h(x) is the polynomial corresponding to the CRC check bits of the page formed by taking the difference between the old and new pages. In other words, to update the CRC, it is enough to compute the check bits of the difference page and add it to the old check bits. This is useful because the difference page is often sparse (has very few nonzero entries), so its check bits can be computed quickly. Suppose updates are done in consecutive t-bit chunks. Then the difference of a binary page might look like: which corresponds to a Δ(x) of the form
where the c's may be zero or one, and are the only nonzero terms of the polynomial. To allow rapid computation of
Then the check bits of Δ(x) are
which can be rapidly computed: we need only do a table lookup, multiply two polynomials (of degrees at most k−1 and t−1), and reduce the resulting polynomial (of degree at most k+t−2) modulo the check polynomial. This last reduction is accelerated using another group of lookup tables. We choose a set of partition points a
and let
where z runs from 1 to w and, for each table Q
The reduction of p
+p _{0}(x)Q _{1} [p _{1}(x)]+Q _{2} [p _{2}(x)]+ . . . +Q _{w} [p _{w}(x)]requiring just w table lookups and adds. In the binary case, the polynomial multiplication step is very easy. We store each polynomial in a register, with the degree d term corresponding to bit d, say registers p Res=0 While p If low bit of p Res=Res XOR p Shift the bits of p Shift the bits of p Finally, in case the changes do not fall within a single chunk, we simply perform the updates a single chunk at a time, as above. Our exemplary implementation of the algorithm uses the degree 32 ANSI X3.66 check polynomial, 32-bit chunks, and four reduction tables corresponding to partition points The algorithm will now be summarized with reference to the flowchart of FIG. As shown, the first step involves pre-computing the lookup table entries, i.e., pre-computing pages P
and finally computes,
In sum, the present invention provides an error detection algorithm that is particularly useful in applications involving the detection of errors in database systems and file systems that are characterized by relatively small modifications made to previously stored pages of data. It is understood, however, that the invention is susceptible to various modifications and alternative constructions. It should be understood that there is no intention to limit the invention to the specific constructions described herein. On the contrary, the invention is intended to cover all modifications, alternative constructions, and equivalents falling within the scope and spirit of the invention. It should also be noted that the present invention may be implemented in a variety of database and database management applications, as well as file systems. The various techniques described herein may be implemented in hardware or software, or a combination of both. Preferably, the techniques are implemented in computer programs executing on programmable computers that each include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Program code is applied to data entered using the input device to perform the finctions described above and to generate output information. The output information is applied to one or more output devices. Each program is preferably implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage medium or device (e.g., ROM or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the procedures described above. The system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner.
Patent Citations
Non-Patent Citations
Referenced by
Classifications
Legal Events
Rotate |