|Publication number||US7343046 B2|
|Application number||US 10/776,602|
|Publication date||Mar 11, 2008|
|Filing date||Feb 12, 2004|
|Priority date||Feb 12, 2004|
|Also published as||CN1655584A, CN100446542C, EP1564684A2, EP1564684A3, EP1564684B1, US20050180647|
|Publication number||10776602, 776602, US 7343046 B2, US 7343046B2, US-B2-7343046, US7343046 B2, US7343046B2|
|Inventors||Donald J. Curry, Asghar Nafarieh, Doron Kletter|
|Original Assignee||Xerox Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (28), Non-Patent Citations (32), Referenced by (38), Classifications (29), Legal Events (6)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This invention is related to U.S. patent applications Ser. Nos. 10/776,515, 10/776,514, 10/776,608, 10/776,602, 10/776,620, 10/776,509, 10/776,508, 10/776,516 and 10/776,612, filed on an even date herewith and incorporated by reference in their entireties.
1. Field of Invention
This invention is directed to organizing image data into regions having similar locations and color characteristics.
2. Related Art
Documents scanned at high resolutions typically require very large amounts of storage space. Furthermore, a large volume of image data requires substantially more time and bandwidth to manipulate, such as transferring over a local or wide area network, over an intranet, an extranet or the Internet or other distributed networks.
Documents, upon being scanned using a scanner or the like, are typically defined using an RGB color space, i.e., in raw RGB format. However, rather than being stored in this raw scanned RGB format, the document image data is typically subjected to some form of data compression to reduce its volume, thus avoiding the high costs of storing such scanned RGB document image data.
Lossless run-length compression schemes, such as Lempel-Ziv (LZ) or Lempel-Ziv-Welch (LZW), do not perform particularly well on scanned image data or, in general, image data having smoothly varying low-spatial frequencies such as gradients and/or natural pictorial data, while lossy methods such as JPEG, work fairly well on smoothly varying continuous tone image data. However, lossy methods generally do not work particularly well on binary text and/or line art image data, or, in general, on any high spatial frequency image data containing sharp edges or color transitions, for example.
A new approach to satisfying the compression needs of data, such as the different types of image data described above, is to use an encoder pipeline that uses a mixed raster content (MRC) format to describe the data. The image data, such as for example, image data defining a composite image having text intermingled with color and/or gray-scale information, is segmented into two or more planes. These planes are generally referred to as the background plane and the foreground planes. A selector plane is generated to indicate, for each pixel in the composite image, which of the image planes contains the actual image data that should be used to reconstruct the final output image. Segmenting the image data into planes in this manner tends to improve the overall compression of the image, because the data can be arranged into different planes such that each of the planes are smoother and more readily compressible than is the original image data. Segmentation also allows different compression methods to be applied to the different planes. Thus, the most appropriate compression technique for the type of data in each plane can be applied to compress the data of that plane.
Unfortunately, some image document formats, such as the portable document format (PDF), do not currently fully support such three-layer mixed raster content decompositions of an original document. As a result, when attempting to print or otherwise render a document that has been compressed and stored as a mixed raster content image data file using such image document formats, the document either cannot be rendered at all, or contains objectionable artifacts upon rendering.
Systems and methods are provided for converting a document to a mixed raster content format having a plurality of binary foreground planes. After scanning the document, the image data is analyzed to identify regions, or blobs, using boundary continuity and/or color continuity. Each pixel is assigned an id that is only required to be unique inside a blob's bounding rectangle and a table is constructed that identifies the blobs by a bounding rectangle and pixel id and lists other attributes. The table is analyzed to sort and filter the blobs and cluster together blobs having similar locality and color characteristics. The clusters are then placed into one of a plurality of binary foreground layers. Each of the binary foreground layers is assigned a color value that approximates the average color of all blobs assigned to that layer. In this way, a high quality image can be rendered with acceptable color variation in the output image.
The invention is described in detail, with reference to the following figures, wherein:
Various exemplary embodiments of systems and methods are disclosed that automatically process scanned and/or printed color documents to produce small, highly-compressed image data files that accurately capture the original document content. Data files are generated by scanning in documents, and analyzing the data to determine which pixels of the image can be grouped together because they have similar color and are defined by a closed curve boundary. These areas are then clustered together by combining areas of similar color which are near each other spatially. The combined areas (clustered regions) are then placed in one of a plurality of binary foreground layers, and may be compressed for output as a PDF file, for example.
An image data portion in any of the multiple binary foreground layers 220-270 does not overlap other image data portions in any of the other multiple binary foreground layers 220-270. As a result, each of the binary foreground layers 220-270 can be individually combined with the background layer 210 without regard to order or sequence. When all of the multiple binary foreground layers 220-270 are combined with the background layer 210, the reconstructed image 280 is obtained. Further processing of the binary planes might produce slightly better compression but require at least a partial ordering of the foreground planes.
The pixels in the image data may be identified as belonging to a “blob”, which is a region defined by a closed curve boundary and containing pixels having a similar color value. A map of the image data may be made which assigns a non-unique small id to every pixel such that the id is unique only inside the bounding rectangle of the blob which is assigned the id. An alternative map of the image data may also be made, which assigns a unique id to each blob but the id size would then be required to be large enough to address all blobs. The following descriptions focus on using the small id (8 bit) map since it requires fewer resources at key points in the current implementation. An example of a blob ID image is shown in
Blob #2 is another blob, bounded by min_x2, max_x2, min_y2 and max_y2. These two blobs may be clustered together by a blob cluster processing module. The cluster may then be assigned to one of the N-binary foreground layers, as will be described fully below.
The first module in the N-layer image data generating system 1000 is the color conversion module 500. The color conversion module 500 may convert the RGB data to a YCC color space, for example. As is well known, the YCC color space includes a luminance component (Y) and two chroma components (CC). The chroma components may be subsampled at a 2:1 ratio, at least in one direction if not in both, so that there are half as many data points in the chroma components as there are in the luminance components. Other color spaces, such as Lab may also be used.
The image in the YCC color space may then be input to a blob identifying module 600 which generates a list of the regions, or blobs, appearing in the image. The blobs are identified based edge continuity and color characteristics. Blobs are regions having a continuous boundary, which encloses pixels having nearly the same color value. The blob identifying module 600 assigns a blob ID to each pixel in the image, indicating to which blob, if any, the pixel belongs. Pixels which do not belong to blobs are assigned to a background layer. A map of the blob IDs for each pixel location is a blob identification (blob ID) image, and is generated by the blob identifying module 600.
The blob identifying module 600 also generates a global table of blobs (GTB) which is a list of all blobs found in the document image 280. A single entry in the GTB is shown in
The luminance 350 and chroma values 360 and 370 are color sums assigned to a blob; the total pixel count 380 is the total number of pixels in the blob ID image which have been assigned to a given blob (For example, the total pixel count for blob #1 in
Also generated by the blob identifying module is the enhanced color data (ENH), which is the scanned, YCC-converted image. The data in the ENH is “enhanced” because the edge features of the blobs have been enhanced, based on the contrast of the surrounding pixels. The ENH, along with the GTB and the blob ID map image, are the inputs to the blob cluster processing module 700.
The blob cluster processing module 700 inputs the blob ID map image, the GTB, and the ENH, analyzes the global table of blobs, and combines those blobs which share certain color and spatial characteristics into N clustered regions. The blob cluster processing module 700 assigns the clustered regions to one of N+1 binary foreground layers, depending on the color value of the clustered region. The first layer (layer 0) is the selector layer, which is obtained by OR-ing layers 1 through N. The remaining layers 1 through N contain the N-layer information. The layers are then compressed by the compression module 800 before being output in, for example, a PDF format.
The blob cluster processing module 700 inputs the ENH, the blob ID map data (BID) and the GTB through the input/output port 780 from the blob identifying module 600, and generates the N+1 binary foreground layers. The input data may either be stored in total in memory 770, or as much as needed may be stored for processing the data on the fly.
Upon receiving the GTB from the blob identifying module 600, the cluster sort module 130 sorts the blobs to place them in an order which is convenient for subsequent blob cluster processing stages. The sorting may be accomplished ‘in place’ by swapping table record contents in GTB. The cluster sort module 730 sorts the blobs by vertical position (slow scan) order first, then horizontal position (fast scan). The cluster sort module 730 sorts the blobs based on the top-left corner of the blob bounding-box. The blobs may be sorted sequentially, by line number, according to which line the top-left corner of the bounding box is located on, unless two or more blobs fall on the same line, in which case they may be sorted according to which blob bounding box falls on the leftmost pixel.
The cluster sort module 730 also normalizes the color sums in the GTB, by dividing the color sum by the pixel count, to obtain an average blob color. The normalization is done with rounding to the nearest integer. After obtaining an accurate value for the average blob color, the blob pixel count is adjusted to be less than or equal to a predefined upper limit. This last step prevents arithmetic overflows in subsequent steps in the case of very large blobs. After this point in the processing, the count is only used to determine the weight of a blob for the purpose of computing an average for a cluster of blobs.
The sort module 730 then passes the GTB to the mark blobs module 740. The mark blobs module 740 analyzes each blob in the GTB, and applies a mark to the nark field 430, depending on the results of a number of test conditions. The mark field 430 was initialized to zero by the blob identifying module 600, at the creation of the GTB. The purpose of marking the blobs is to indicate which blobs may be eliminated by downstream processes, in order to reduce the number of blobs, and therefore reduce the processing burden. Thus, the mark field 430 is used by the marking process to achieve the process burden reduction purpose.
The mark blobs module 740 tests the blobs according to five different conditions. If the conditions are met, the bits in the mark field are changed to mark the blob. Each condition leads to a different bit pattern in the mark field. In order to expedite the process, simple tests are used based on the bounding box instead of the exact blob shape. The mark blobs module 740 conditions are applied in the following order:
The mark blobs module 740 does not remove nor change any blobs, but only alters the mark field when one or more of the test conditions is satisfied. Subsequent modules may detect one or more of the above mark codes to eliminate marked blobs. In addition, subsequent steps may change or add other marked codes.
The mark blobs module 740 further tests to determine if the color of a blob has a variability which falls within certain predefined limits. If not, the blob is marked accordingly. Only blobs which have at least a minimum pixel count (of at least, for example, 500 pixels) and which have not yet been marked by the mark graphics module, are inspected for color variability.
While all or a large portion of the pixels in a blob may be used for the color variability test, to increase the speed of the color variability test, only the pixels located on a single line, such as the middle line, may be tested, as shown in
For each pixel on the middle line that belongs to the current blob as indicated by the blob ID image, the pixel chroma values are fetched from the enhanced color data ENH. As illustrated by
where ENH1 and ENH2 are the pixel chroma component values (Cb, Cr), and ave and ave are the blob average chroma values (Cb, Cr), respectively.
At the same time, the number of “X” pixels (belonging to the current blob on the middle scanline) is counted. After the last pixel on the middle line is encountered, the accumulated chroma-square variance calculated according to Eq. (1) is normalized by the pixel count.
The mark blobs module 740 then compares the calculated chroma-square variance against a predefined limit. If the chroma-square variance exceeds this limit, the blob is marked accordingly. The mark blobs module 740 then proceeds to check the next blob in the global table of blobs. The process repeats until the last blob is inspected.
After the blobs are marked, they are clustered to combine those blobs having similar color characteristics and nearby in location. Since each cluster will eventually be assigned to its own binary foreground plane, the build clusters module 740 seeks to consolidate as many blobs as possible into as few clusters as possible. The build clusters module 720 organizes the sorted and marked blobs into clusters, based on their color and spatial relationships.
The build clusters module 720 begins by dividing the image area into adjacent and non-overlapping square tiles. The purpose of the tiles is to reduce the image area being evaluated into a more manageable size. The tile size is programmable (for example, 256×256 pixels). For convenience, the tile size is restricted to be a binary power of two. The image area and tile relationship is illustrated in
The build clusters module 720 groups blobs by color and spatial location, to form color clusters and spatial clusters of blobs. Each color cluster contains one or more blobs which are similar in color, i.e., the regions are different in color by an amount less than a color threshold. The color threshold may be defined in terms of the square of the difference in luminance between the blobs, or the chroma-squared difference, as is well known in the art. Each color cluster contains at least one spatial cluster, and each spatial cluster contains at least one blob. If a color cluster contains multiple spacial clusters, they must all be close in color. If a spatial cluster contains multiple blobs, the blobs must be near each other.
To determine whether a blob should join a spatial cluster, an expanded bounding box is used. The purpose of the expanded bounding box is define an exceptable distance threshold between blobs which are still to be considered to be in the same spatial cluster. Blobs which have overlapping expanded bounding boxes are considered to be close enough to each other to belong to the same spatial cluster. An aggregated bounding box is associated with the spatial clusters, which includes the expanded bounding boxes of all the blobs contained in the cluster. As additional blobs are added to the cluster, the aggregated bounding box of the spatial cluster is adjusted to include expanded bounding box of the additional blob.
An example of an expanded bounding box is shown in
For example, returning to
The size of the expansion of the bounding box is chosen based on a tradeoff between the resulting number of binary foreground layers, and the compression efficiency of the binary layers. As larger expanded bounding boxes are used, blobs which are separated by a larger extent will be joined together by the blob cluster processing module, so that fewer foreground layers are needed to reproduce the image. However, the efficiency of the compression of the binary layer having widely spaced blobs, will suffer. Although the expanded bounding box 754 spans all four tiles on this segment of the image, the position of the top-left corner 751 of the expanded bounding box 754 determines the tile to which the initial clusters belong. Thus the blob of
To keep track of the contents of the clusters as they are assembled from the blobs, the build cluster module 720 maintains two levels of cluster data rooted at each tile. For instance, if the blob of
The next field points to the next record in a list. The next field of a blob points to a blob. The next field of a spatial cluster points to a spatial cluster. And the next field of a color cluster points to a color cluster. The next field is NULL at the end of a list. An exemplary data structure used by the build cluster module 720 is shown in
The bottom level 737 contains the original blobs linked to the spatial clusters by a cont pointer and to blobs within the same cluster by next pointers. Thus, each cluster or blob is linked to other clusters or blobs in the same level using a “next” pointer, and the clusters are linked to other clusters on a lower level by the “cont” pointer. Therefore, the “next” pointer links a chain of clusters or blobs, and the “cont” pointer links contents of a cluster or blob. All cluster lists may use the same basic data structure.
The build clusters module 720 processes the blobs from the GTB, one blob at a time, and each blob from the list in turn becomes the “current” blob for the purposes of the following discussion. The coordinates of the top-left bounding-box corner for the current blob is retrieved from the GTB entry, and used to determine the corresponding tile number by dividing the coordinates by the tile size.
The first blob found in each tile initiates new cluster chains, to which subsequent blobs will be linked. The tile entry identifies the tile to which the cluster belongs. The blob data from the blob entry in the GTB, including number of pixels, luminance and chroma, may be copied into new color and spatial cluster records. The new clusters bounding boxes are computed by expanding the blob bounding-box. The color cluster cont field will point to the spatial cluster and the spatial cluster cont field will point to the blob.
The next blob found in the tile can either be assigned to an existing spatial cluster, or a new cluster may be created to contain it. For example, if the current blob 872 is determined not to match any already created color cluster, a new color cluster 846 is created in the cluster list, and inserted at the beginning of the color cluster chain.
Since newly added cluster 846 is a top-level 735 color cluster, the next two levels 736 and 737 also need to be established at this time. The color cluster 846 cont field is set to point to new spatial cluster 858 created at level 736.
Finally, the blob level 737 of the cluster array is updated by linking the newly established spatial blob 858 with the current blob 872. As indicated above, when the new spatial and color clusters 858 and 846 are created, there is only one associated blob. However, later on additional blobs or clusters may be added if they are closely matching in color and they meet the spatial overlap.
As can be seen from
If two color clusters have color values which fall within the color threshold, or two spatial clusters have overlapping extended bounding boxes, they may be merged. The merging includes the updating of average color and pixel count. The equations used to update these values are:
if (cnt1>MaxCount), then new cnt1=MaxCount (6)
where Y1, Y2, Cb1, Cb2, Cr1, Cr2 are the luminance and chrominance values, and cnt1 and cnt2 are the pixel counts. The two extended bounding boxes are combined producing a bounding box which bounds both. Finally, any content lists pointed to by the cont fields of the two clusters are combined by appending one to the other and this combined list will be pointed to by the new cont field. If the two clusters being merged are already in the same list, the one must be removed from the list and the other can be used to contain the merged data.
Since the cluster merging method above is optimized, it is also used during the initial insertion of blobs into the tile cluster structure. A candidate temporary spatial cluster is created which has the extended bounding box computed for the current blob and whose cont field points to the blob. Its next field is empty since it is not yet part of a list. The color cluster chain is searched for a color match. If a matching color cluster is not found a new color cluster is initialized using the color and entended bounding box fields of the current cluster. Then the spatial cluster list of the matching (or new) color cluster is searched for any spatial clusters with extended bounding boxes which intersect the current spatial cluster. If a spatial match is found the current cluster is merged into the matching spatial cluster. If no match is found the current cluster is added to the beginning of any existing content of the color cluster. The testing for a spatial match is illustrated in
If such an overlapping cluster is found, the current cluster data is merged into that spatial cluster. A new aggregated bounding box is established, which includes the extent of the merged bounding boxes. Therefore, the aggregated bounding box has an extent in the x-dimension that includes the widest expanse of the blobs in the newly merged cluster, in the x-direction. Similarly, the aggregated bounding box has an extent in the y-dimension, that includes the widest expanse of the blobs in the newly merged cluster in the y-direction.
As discussed above, if no spatially overlapping cluster is found, the current spatial cluster is added to the beginning of the content list of (a possibly new) color cluster.
The build clusters module 720 next proceeds to inspect the next blob in the GTB. This process repeats until all of the blobs have been inserted into the 3-level structure 735-737 based on their color match and spatial proximity.
Once the entire three-level structure 735-737 has been completed, the build clusters module 720 begins to merge the clusters to reduce the number of remaining clusters. The color clusters are merged only if their average color is close enough, as defined by a programmable parameter. The spatial clusters are merged only if their expanded bounding boxes overlap (see
After cluster merging, the boundary between each 2×2 set of adjacent tiles is removed, by merging clusters which overlap across the tile boundary. The removal of the tile boundaries and the merger of clusters is illustrated by
At this point a new cluster array is allocated with half the number of tiles in each direction, and the entries are filled with the combined data. The merge operation for one 2×2 set of tiles, is illustrated in
The final cluster structure is passed to the merge clusters module 705. The merge clusters module 705 first removes the top level 735 of color clusters from the cluster structure. This leaves just the spatial color clusters and their associated blobs. The operation of the merge cluster module 705 is illustrated in
The merge clusters module 705 searches the chain of spatial clusters again for the possibility to merge clusters based on color match and spatial overlap. If a match is found, the data is merged into the first cluster as described above and the second cluster is deleted. The links are reconnected to maintain the list without the cluster that was eliminated. The operation of the merge clusters module 705 is illustrated in
As illustrated in
The spatial clusters are re-calculated by the merge clusters module 705, directly from the associated blob data content. The merge clusters module 705 re-calculates by evaluating the chain of spatial clusters. For each spatial cluster, the pixel count, the color average, and aggregated bounding box (no expansion) are re-calculated based on the blobs contained in the cluster. The accumulated YCC color sums may be monitored to prevent overflow when necessary, by, for example, dividing both the sums and the pixel count by two. The spatial cluster data is replaced with the re-calculated data. The re-calculation process is similar to that performed by the cluster sort, module 730 described above, except for using the exact bounding box instead of the expanded ones.
The reason for re-calculation is that blob clustering by the build clusters module 720 relied on expanded bounding boxes to connect clusters. But having completed the clustering, the accurate bounding boxes are needed to calculate the area size prior to the separation into N layers. In addition, the repeated cluster merging operation may suffer from accumulated errors. Lastly, the merge clusters module 705 checks the actual cluster pixel counts, without expansion, in order to eliminate clusters that are too small.
The merge clusters module 705 eliminates clusters that may be bad, and re-adjusts the blob mark fields. A spatial cluster is considered marked if it is either too small (pixel count below a certain threshold) or the fraction of blobs it contains that are marked exceeds a certain level. In that case, the mark fields of all the blobs linked to the cluster are set to a non-zero value. Otherwise, the mark fields of all the blobs linked to the cluster are cleared (no mark). Spatial clusters that are marked as indicated above are eliminated and the next links are updated to maintain the list of spatial clusters without the eliminated clusters. This filtering allows very small blobs such as periods in text which normally might be eliminated as being too small, to be included since they will have been clustered with mostly unmarked text blobs. On the other hand, undesirable blobs that might accidentally be found in images are likely to be marked and cause their clusters to be removed.
A natural result of the blob identification process is that every blob is either completely within or completely outside every other blob. After the merge clusters module 705 completes the merge process, the inner blob module 760 checks each blob to see if it can be fully contained in another unmarked blob, i.e., if it is, it is an “inner”. Any blobs determined to be inners are removed from the cluster. A typical example of inner blobs would be the inner holes contained in characters such as “b”, “e”, and “o”, etc. The inner holes are normally classified as blobs by the blob identify module 600.
The operation of the inner blob module 760 requires checking for possible inners between all blob pairs. Because of the computational load that this situation presents, the inner blob module is applied after the merge cluster module 705, where many bad or small blobs are eliminated (marked). This leaves fewer blobs to test for inners by the inner blob module 760.
In order to further expedite performance, simple checks may be used. The simple checks reduce the number of calculations by checking only the bounding-box conditions and just one scan line of blob ID information, for example.
The inner blob module 760 examines all possible combinations of blob pairs. A first blob, blob A, is picked sequentially from the GTB which has been previously sorted by minimum y of the bounding box. A second blob, blob B, is then picked from the GTB beginning just after A. For each blob, the bounding box and blob ID are retrieved from the global table of blobs. The second blob, blob B is tested as inner with respect to blob A. The test procedure is designed to take advantage of shortcuts when possible. The module applies the following tests:
The process for determining whether blob B is an inner of blob A is illustrated in
The state machine 890 shown in
Once the first blob ID A is seen, the state is changed to 1. Now the state machine 890 begins to look for the occurrence of blob ID B. The state remains 1 as long as blob ID A continues to be observed. However, if any ID other than A or B comes after blob ID A, and before blob ID B, the state is reset to zero to start over. Only when blob ID B comes right after blob ID A the state is changed to 2 indicating that B is and inner. The test may be aborted as soon as this state is reached.
If state 2 is found, the B blob is marked as an inner blob by turning on the inner bit in the mark field 430 of the B blob. In all other cases the B blob is not marked as an inner.
It should be noted that the A to B transition (state 2) is sufficient. There is no need to check for B to A transition on the other side, as this is guaranteed by the blob identification module 600.
After checking if B is an inner of A, as described above, the process moves on to check the next B blob in the GTB. This continues until the last blob in the GTB has been evaluated or the minimum y of B is greater than the maximum y of A. At this point, all possible inners of A have been identified and marked. The process then moves on to pick the next A blob in the GTB, and again checks for inners between it and all subsequent B blobs. There is no need to check blobs prior to A since such combinations have already been tested and it is not possible for A to be outside of a blob that has a minimum y value less than or equal to A. The process continues until blob A reaches the last blob in the global table of blobs. At this point, all valid pairs of inner combinations have been examined.
The inner blob module 760 checks the blob content of each spatial cluster to see if there are any blobs that were marked as inners, that is, fully contained in another blob, by the inner blob module 760. The inner blob module 760 then follows the list of blobs associated with each cluster and recalculates the cluster information while excluding the inner blobs. This includes the re-calculation of the bounding box union (excluding inner blobs) and new weighted average color, based on the number of pixels in each blob. Again, the accumulated YCC color sums are monitored to prevent overflow by, for example, dividing both sums and count by two when necessary.
Next, the inner blob module 760 examines the resulting cluster pixel count, excluding inners, and compares it against a predetermined threshold. If the resulting cluster size is too small, the entire cluster is removed and all of its associated blobs are marked.
However, if the resulting cluster size is big enough, the updated YCC color sums are normalized by the pixel count, and the new average color, excluding inners, is loaded into the cluster data replacing the previous sums. Likewise, the new pixel count and updated bounding box information, excluding inner blobs, are loaded into the cluster data.
The cluster sort module 730 sorts the spatial clusters by size, from the largest to smallest pixel count. The purpose for sorting is to provide fast PDF viewing by drawing the larger layers first. Since, at this point, the number of layers is relatively small, the sorting method may be based on the bubble sort algorithm, as is described below in relation to
The spatial cluster chain 991, 992, and 993 is followed one cluster at a time. At each step, the size of the current cluster 991 is compared with the size of the subsequent cluster 992. The subsequent cluster 992 is identified by following the next link from the current cluster 991. The cluster size is given by the count field, which contains the aggregated number of pixels in the cluster. If the number of pixels in 991 is smaller than the number of pixels in the count field of 992, the cluster positions in the chains are switched, and their next links are updated to reflect the new positions.
For example, cluster 991 in
It should be noted that at each sorting step, the cluster content is not changed. Only the relative position of clusters in the chain is modified. The new position is given by the updated linking order, via the next field. It should also be noted that the cont entry for each cluster is kept unchanged. This means that each cluster retains its own associated blob chain as is.
The generate layers module 750 follows the cluster sort module 730. The generate layers module 750 converts the spatial clusters into binary mask planes. Each cluster is sequentially assigned a layer number beginning with 1. As described previously, the spatial clusters, and therefore the binary planes, are sorted by descending pixel count values. The selector plane, an additional binary mask plane number 0, will be the union of layers 1 through N and has the same size as the background layer. Thus, the number of layers is the same as the number of surviving spatial clusters+1.
The generate layers module 750 may also remove the white border around each plane, and byte-align each plane to an even byte boundary. Byte alignment of the individual planes speeds up the process of creating layer 0, the union of all layers. To produce a byte aligned layer, the generate layers module 750 reduces the left coordinate (minimum x) of the bounding box so that it becomes a multiple of 8. This can slightly increase the size of the layers. Only the bounding box left coordinate is modified by the generate layers module 750.
The generate layers module 750 then creates the binary (N+1)-planes. Planes 1 through N contain the binary data for the corresponding layers. The size of each plane is determined based on the (byte aligned) bounding box calculated by the generate layers module 750. Plane 0 contains the binary selector layer that is equivalent to the combined OR-ed data from layers 1 to N. The size of plane 0 is the same as the background plane.
The generate layers module 750 creates one plane at a time. As described previously, the cluster sort module 730 sorts the planes by decreasing pixel count value. For each plane, the byte aligned bounding box is used to calculate the plane size and allocate the right amount of memory storage for the binary image. The plane is initially set to zero. Then each blob in the plane's cluster content list is processed. For each blob, all the locations the ID map image corresponding to the blob's bounding box are examined. For every map pixel in this bounding box which has an ID equal to the blob's ID the corresponding bit in the layer plane is set to one. After each plane is created, the bits are also OR-ed into plane 0, the selector plane. The OR process is greatly simplified when all layers are previously byte-aligned by the generate layers module 750, such that no data shifting on sub-byte boundaries is necessary. The number of planes is identical to the number of surviving layers. Each plane is in the packed binary format (1 bit per pixel, 8 pixels per byte).
Although the previous description was given with reference to a hardware embodiment, the blob clustering process can also be implemented as software executing on a microprocessor. In this case, each of the steps of the method is carried out by a suitably programmed microprocessor, or by a combination of hardware and software.
In step S900, the clusters are built from the GTB, where the clusters are made from blobs which are similar in color and which near each other. In step S1000, the top level (the color level) of the cluster table created in step S900, is removed. In step S110, the clusters which are sufficiently similar in color and which overlap spatially, are merged to reduce the number of surviving clusters. In step S1200, the remaining clusters colors are re-calculated and the cluster counts are updated after the merge operation of step S1100. In step S1300, clusters which do not include a minimum threshold number of pixels are eliminated from the cluster list. In step S1400, blobs which are fully contained in another blob are marked as “inners”. In step S1500, the blobs previously marked as inners are removed. In step S600, the remaining clusters are re-calculated after the removal of the inner blobs in step S500. In step S1700, the blobs are sorted by size.
In step S1800, the remaining clusters are assigned sequentially to binary foreground planes. In step S1900, the different foreground planes are byte-aligned to an even byte boundary. Next, in step S2000, the N+1 binary planes are generated. Plane 0, the selector plane, is used to mask the image data within the grayscale background plane. The grayscale background plane is adjusted to improve compressibility. Then, in step S2100, each of the separate binary foreground planes determined in step S1900 are compressed using a compression technique that is appropriate to such binary foreground planes. Operation then continues to step S2200.
In step S2200, the background plane is compressed using a compression technique that is appropriate for such data. Then, in step S2300, a portable document format (PDF) document file is generated from the compressed binary foreground planes and the compressed grayscale background plane. Next, in step S2400, the generated portable document format (PDF) document file is output to a downstream processor and/or stored in a memory. Operation then ends.
It should also be appreciated that, in various exemplary embodiments, any known or later-developed document format, in place of the portable document format (PDF) can be used in steps S2300 and S2400.
If a determination is made in step S610 that the blob area is not smaller than the certain predefined threshold, operation proceeds to step S630, wherein a determination is made whether the blob pixel density is smaller than a predefined threshold. If so, the blob is marked in step S635 with a code indicative of the pixel density being too small. In step S640, a determination is made whether the last blob in the global table of blobs has been reached, and if not, control returns to step S605 wherein the next blob in the table is selected. If the last blob in the table has been reached, the process ends and control returns to step S800 of
If the determination is made in step S630 that the pixel density is not smaller than the predefined threshold, operation proceeds to step S650, wherein a determination is made whether the aspect ratio of the blob is larger or smaller than certain predefined thresholds. If the aspect ratio is smaller or larger than the predefined threshold, the blob is marked with a code indicative of the aspect ratio being smaller or larger than a certain predefined threshold. A determination is made in step S660 whether the last blob in the table has been reached. If not, control returns to step S605, wherein the next blob in the table is selected. If the last blob in the table has been reached, control returns to step S800 of
If the determination is made in step S650 that the aspect ratio of the blob is not smaller or larger than a certain predefined threshold, operations proceeds to step S675, wherein a determination is made whether the blob boundary box height or width is smaller than a predefined threshold. If so, control proceeds to step S680, wherein the blob is marked with a code indicative of the boundary box height or width being smaller than a predefined threshold. In step S685, a determination is made whether the last blob in the table has been reached. If not, control returns to step S605 wherein the next blob in the table is selected. If the last blob in the table has been reached in step S685, control returns to step S800 of
If the determination is made that the blob bounding box height and width are not smaller than a certain predefined threshold, control proceeds to step S695, wherein a determination is made whether the blob is sufficiently larger than a certain predefined threshold, and whether the luminance is close to white. If so, the blob is marked in step S700 with a code indicative of the blob being larger than a certain predefined threshold and the luminance being close to white. A determination is then made in step S705, whether the last blob in the table has been reached. If not, control returns to step S605, wherein the next blob in the table is selected. If the last blob in the table has been reached, control returns to step S800 of
If the determination in step S715 that the blob is not sufficiently large and the luminance is not close to white, then a determination is made in step S715 whether the last blob in the table has been reached. If not, control returns to step S605, wherein the next blob is selected. If so, control returns to step S800 of
In step S950, clusters in the cluster array are merged if the distance between the colors is close enough. In step S955, clusters having the same color and overlapping spatially are also merged. In step S960, the cluster lists for sets of 2×2 tiles are combined. As a result of the combination, clusters having the same color are merged in step S965. Cluster which have the same color and overlap spatially as a result of the combination are also merged in step S970. In step S975, a determination is made whether the size of the combined tiles is the same as the page image size. If not, control returns to step S960, where the cluster lists of another 2×2 set of tiles is combined. If the size of the combined tiles is the same as the page image size, the procedure ends at step S980, and returns to step S1000 of
If blob B is not marked, control proceeds to step S1425, wherein a determination is made whether blob B overlaps blob A in the vertical direction. If not, control returns to step S1447, wherein a determination is made whether the last B blob in the table has been reached. If blob B does overlap blob A in the vertical direction, a determination is made in step S1430, whether the bounding box of blob A surrounds the bounding box of blob B on all sides. If not, control returns to step S1447, wherein a determination is made whether the last B blob in the table has been reached. If the bounding box of blob A does surround the bounding box of blob B on all sides, control proceeds to step S1440, wherein a determination is made whether there are two adjacent pixels on the middle line segment where the first belongs to A and the second belongs to B. If not, control returns to step S1447, wherein the next blob B is selected from the global table of blobs. If the A to B pixel pair is found on the line segment spanning the width of A, then blob B is marked as an inner of blob A in step S1450. Then, control proceeds to step S1447.
In step S1447, a determination is made whether the last B blob in the global table of blobs has been reached. If not, control returns to step S1415, wherein the next blob B is selected from the global table of blobs. If the last B blob in the global table of blobs has been reached in step S1447, then control proceeds to step S1455, wherein a determination is made whether the last A blob in the table has been reached. If so, then control proceeds to step S1460, wherein the process ends, by returning to step S1500 of
While the invention has been described in conjunction with various exemplary embodiments, these embodiments should be viewed as illustrative, not limiting. Various modifications, substitutes, or the like are possible within the spirit and scope of the invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4849914||Sep 22, 1987||Jul 18, 1989||Opti-Copy, Inc.||Method and apparatus for registering color separation film|
|US5515452||Dec 31, 1992||May 7, 1996||Electroglas, Inc.||Optical character recognition illumination method and system|
|US5544256 *||Oct 22, 1993||Aug 6, 1996||International Business Machines Corporation||Automated defect classification system|
|US5577135 *||Mar 1, 1994||Nov 19, 1996||Apple Computer, Inc.||Handwriting signal processing front-end for handwriting recognizers|
|US5583659||Nov 10, 1994||Dec 10, 1996||Eastman Kodak Company||Multi-windowing technique for thresholding an image using local image properties|
|US5745596||May 1, 1995||Apr 28, 1998||Xerox Corporation||Method and apparatus for performing text/image segmentation|
|US5900953||Jun 17, 1997||May 4, 1999||At&T Corp||Method and apparatus for extracting a foreground image and a background image from a color document image|
|US6024018 *||Apr 3, 1997||Feb 15, 2000||Intex Israel Technologies Corp., Ltd||On press color control system|
|US6058214||Jan 19, 1999||May 2, 2000||At&T Corp.||Compression of partially masked still images|
|US6115031 *||Apr 23, 1999||Sep 5, 2000||Lexmark International, Inc.||Method of converting color values|
|US6215893 *||Aug 20, 1998||Apr 10, 2001||Romedix Ltd.||Apparatus and method for measurement and temporal comparison of skin surface images|
|US6236764 *||Nov 30, 1998||May 22, 2001||Equator Technologies, Inc.||Image processing circuit and method for reducing a difference between pixel values across an image boundary|
|US6324305 *||Dec 22, 1998||Nov 27, 2001||Xerox Corporation||Method and apparatus for segmenting a composite image into mixed raster content planes|
|US6343154||Mar 24, 2000||Jan 29, 2002||At&T Corp.||Compression of partially-masked image data|
|US6400844 *||Dec 2, 1998||Jun 4, 2002||Xerox Corporation||Method and apparatus for segmenting data to create mixed raster content planes|
|US6529202 *||Oct 5, 1998||Mar 4, 2003||Canon Kabushiki Kaisha||Color adviser|
|US6633670 *||Mar 31, 2000||Oct 14, 2003||Sharp Laboratories Of America, Inc.||Mask generation for multi-layer image decomposition|
|US6741655 *||May 5, 1998||May 25, 2004||The Trustees Of Columbia University In The City Of New York||Algorithms and system for object-oriented content-based video search|
|US7079675 *||Aug 8, 2003||Jul 18, 2006||Qinetiq Limited||Histological assessment|
|US7145676 *||Jan 31, 2001||Dec 5, 2006||Hewlett-Packard Development Company, L.P.||Compound document image compression using multi-region two layer format|
|US20020027561 *||Oct 5, 1998||Mar 7, 2002||Jing Wu||Colour adviser|
|US20020114512 *||Feb 20, 2001||Aug 22, 2002||Ravishankar Rao||Color clustering and segmentation using sigma filtering|
|US20030035580 *||May 7, 2002||Feb 20, 2003||Kongqiao Wang||Method and device for character location in images from digital camera|
|US20030110181 *||Oct 19, 1999||Jun 12, 2003||Hinrich Schuetze||System and method for clustering data objects in a collection|
|US20030198382 *||Jul 11, 2002||Oct 23, 2003||Jiann-Jone Chen||Apparatus and method for removing background on visual|
|EP0712094A2||Oct 30, 1995||May 15, 1996||Eastman Kodak Company||A multi-windowing technique for threshholding an image using local image properties|
|EP1006716A2||Nov 29, 1999||Jun 7, 2000||Xerox Corporation||Method and apparatus for segmenting data to create mixed raster content planes|
|WO2002056255A2||Oct 25, 2001||Jul 18, 2002||Hewlett Packard Co||Methods and apparatus for analyzing an image|
|1||De Queiroz et al., "Mixed Raster Content (MRC) Model for Compound Image Compression", 1998, pp. 1-12.|
|2||*||Queiroz et al, "Mixed raster content (MRC) model for compound image compression," Feb. 1999, SPIE, vol. 3653, pp. 1106-1117.|
|3||*||Queiroz, "Mixed Raster Content (MRC) Model for Compound Image Compression", (1998), Proc. SPIE 3653, 1106-1117.|
|4||R. De Queiroz, "Compression of Compound Documents," IEEE, Oct. 1999, pp. 209-213.|
|5||Said et al., "Simplified Segmentation for Compound Image Compression", 1999, pp. 229-233.|
|6||U.S. Appl. No. 10/187,499, filed Jul. 1, 2002, Curry et al.|
|7||U.S. Appl. No. 10/188,026, filed Jul. 1, 2002, Curry et al.|
|8||U.S. Appl. No. 10/188,157, filed Jul. 1, 2002, Curry et al.|
|9||U.S. Appl. No. 10/188,249, filed Jul. 1, 2002, Curry et al.|
|10||U.S. Appl. No. 10/188,277, filed Jul. 1, 2002, Curry et al.|
|11||U.S. Appl. No. 10/612,057, filed Jul. 1, 2003, Curry et al.|
|12||U.S. Appl. No. 10/612,062, filed Jul. 1, 2003, Curry et al.|
|13||U.S. Appl. No. 10/612,063, filed Jul. 1, 2003, Curry et al.|
|14||U.S. Appl. No. 10/612,064, filed Jul. 1, 2003, Curry et al.|
|15||U.S. Appl. No. 10/612,084, filed Jul. 1, 2003, Curry et al.|
|16||U.S. Appl. No. 10/612,234, filed Jul. 1, 2003, Curry et al.|
|17||U.S. Appl. No. 10/612,246, filed Jul. 1, 2003, Curry et al.|
|18||U.S. Appl. No. 10/612,248, filed Jul. 1, 2003, Curry et al.|
|19||U.S. Appl. No. 10/612,250, filed Jul. 1, 2003, Curry et al.|
|20||U.S. Appl. No. 10/612,261, filed Jul. 1, 2003, Curry et al.|
|21||U.S. Appl. No. 10/612,368, filed Jul. 1, 2003, Curry et al.|
|22||U.S. Appl. No. 10/612,461, filed Jul. 1, 2003, Curry et al.|
|23||U.S. Appl. No. 10/776,508, filed Feb. 12, 2004, Curry et al.|
|24||U.S. Appl. No. 10/776,509, filed Feb. 12, 2004, Claassen et al.|
|25||U.S. Appl. No. 10/776,514, filed Feb. 12, 2004, Curry et al.|
|26||U.S. Appl. No. 10/776,515, filed Feb. 12, 2004, Curry et al.|
|27||U.S. Appl. No. 10/776,516, filed Feb. 12, 2004, Curry et al.|
|28||U.S. Appl. No. 10/776,603, filed Feb. 12, 2004, Curry et al.|
|29||U.S. Appl. No. 10/776,608, filed Feb. 12, 2004, Curry et al.|
|30||U.S. Appl. No. 10/776,612, filed Feb. 12, 2004, Curry et al.|
|31||U.S. Appl. No. 10/776,620, filed Feb. 12, 2004, Curry et al.|
|32||Zugaj et al., "A New Approach of Color Images Segmentation Based on Fusing Region and Edge Segmentation Outputs", 1998, pp. 105-113.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8306345||Sep 22, 2009||Nov 6, 2012||Xerox Corporation||3+N layer mixed raster content (MRC) images and processing thereof|
|US8335379||Dec 28, 2009||Dec 18, 2012||Xerox Corporation||System and method for cleanup of MRC images for improved compression and image quality|
|US8339399||Oct 31, 2007||Dec 25, 2012||Microsoft Corporation||Declustering point-of-interest icons|
|US8345998||Aug 10, 2010||Jan 1, 2013||Xerox Corporation||Compression scheme selection based on image data type and user selections|
|US8351691||Dec 14, 2009||Jan 8, 2013||Canon Kabushiki Kaisha||Object extraction in colour compound documents|
|US8457426 *||May 18, 2011||Jun 4, 2013||Adobe Systems Incorporated||Method and apparatus for compressing a document using pixel variation information|
|US8488190 *||Feb 17, 2010||Jul 16, 2013||Canon Kabushiki Kaisha||Image processing apparatus, image processing apparatus control method, and storage medium storing program|
|US8503036||Jun 23, 2008||Aug 6, 2013||Xerox Corporation||System and method of improving image quality in digital image scanning and printing by reducing noise in output image data|
|US8565531||Oct 9, 2009||Oct 22, 2013||Xerox Corporation||Edge detection for mixed raster content (MRC) images for improved compression and image quality|
|US8577153||Oct 28, 2010||Nov 5, 2013||Alibaba Group Holding Limited||Method and system for sample image index creation and image filtering and search|
|US8582906 *||Mar 3, 2010||Nov 12, 2013||Aod Technology Marketing, Llc||Image data compression and decompression|
|US8650473||Jul 3, 2009||Feb 11, 2014||Canon Kabushiki Kaisha||Document managing apparatus, document managing method, and storage medium|
|US8659620 *||Apr 13, 2009||Feb 25, 2014||Accusoft Corporation||Methods and apparatus for rendering images|
|US8750612 *||Aug 19, 2013||Jun 10, 2014||Fuji Xerox Co., Ltd.||Image processing apparatus and computer readable medium|
|US8761465||Mar 18, 2010||Jun 24, 2014||Microsoft Corporation||Centroid processing|
|US8792719||Jul 30, 2012||Jul 29, 2014||Brother Kogyo Kabushiki Kaisha||Image processing device determining attributes of regions|
|US8811733 *||Jan 14, 2013||Aug 19, 2014||Stmicroelectronics S.R.L.||Method of chromatic classification of pixels and method of adaptive enhancement of a color image|
|US8830529||Jul 30, 2012||Sep 9, 2014||Brother Kogyo Kabushiki Kaisha||Image processing device for accurately identifying region in image without increase in memory requirement|
|US8837836||Jul 30, 2012||Sep 16, 2014||Brother Kogyo Kabushiki Kaisha||Image processing device identifying attribute of region included in image|
|US8929663 *||Jul 30, 2012||Jan 6, 2015||Brother Kogyo Kabushiki Kaisha||Image processing device identifying region in image as one of uniform region and nonuniform region|
|US9552129||Mar 23, 2012||Jan 24, 2017||Microsoft Technology Licensing, Llc||Interactive visual representation of points of interest data|
|US9697751||Mar 9, 2010||Jul 4, 2017||Microsoft Technology Licensing, Llc||Interactive representation of clusters of geographical entities|
|US20090316213 *||Jun 23, 2008||Dec 24, 2009||Xerox Corporation||System and method of improving image quality in digital image scanning and printing by reducing noise in output image data|
|US20100157340 *||Dec 14, 2009||Jun 24, 2010||Canon Kabushiki Kaisha||Object extraction in colour compound documents|
|US20100231995 *||Feb 17, 2010||Sep 16, 2010||Canon Kabushiki Kaisha||Image processing apparatus, image processing apparatus control method, and storage medium storing program|
|US20100239139 *||Mar 18, 2010||Sep 23, 2010||Microsoft Corporation||Centroid processing|
|US20100259557 *||Apr 13, 2009||Oct 14, 2010||Mcmullen Roderick A||Methods and apparatus for rendering images|
|US20110032552 *||Jul 23, 2010||Feb 10, 2011||Canon Kabushiki Kaisha||Data processing apparatus and data processing method|
|US20110069885 *||Sep 22, 2009||Mar 24, 2011||Xerox Corporation||3+n layer mixed rater content (mrc) images and processing thereof|
|US20110072019 *||Jul 3, 2009||Mar 24, 2011||Canon Kabushiki Kaisha||Document managing apparatus, document managing method, and storage medium|
|US20110085735 *||Oct 9, 2009||Apr 14, 2011||Xerox Corporation||Edge detection for mixed raster content (mrc) images for improved compression and image quality|
|US20110134245 *||Dec 1, 2010||Jun 9, 2011||Irvine Sensors Corporation||Compact intelligent surveillance system comprising intent recognition|
|US20110158529 *||Dec 28, 2009||Jun 30, 2011||Xerox Corporation||System and method for cleanup of mrc images for improved compression and image quality|
|US20110216969 *||Mar 3, 2010||Sep 8, 2011||Chu Wai C||Image Data Compression and Decompression|
|US20110225541 *||Mar 9, 2010||Sep 15, 2011||Microsoft Corporation||Interactive representation of clusters of geographical entities|
|US20110225546 *||Mar 9, 2010||Sep 15, 2011||Ramos Gonzalo A||Map spotlights|
|US20130028524 *||Jul 30, 2012||Jan 31, 2013||Brother Kogyo Kabushiki Kaisha||Image processing device identifying region in image as one of uniform region and nonuniform region|
|US20130136352 *||Jan 14, 2013||May 30, 2013||Stmicroelectronics S.R.L.||Method of chromatic classification of pixels and method of adaptive enhancement of a color image|
|U.S. Classification||382/243, 382/225, 382/162, 382/173, 382/302, 382/232|
|International Classification||G06T7/00, G06K9/54, G06T5/00, G06K9/00, G06T1/20, G06K9/46, G06T1/00, G06K9/34, G06K9/62, H04N1/56, G06K9/20|
|Cooperative Classification||G06T7/90, G06T7/11, H04N1/56, G06T2207/20021, G06T2207/20016, G06T2207/10008, G06T2207/10024, G06K9/00456, G06T2207/30176|
|European Classification||G06K9/00L2, H04N1/56, G06T7/00S1|
|Feb 12, 2004||AS||Assignment|
Owner name: XEROX CORPORATION, CONNECTICUT
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CURRY, DONALD J.;NAFARIEH, ASGHAR;KLETTER, DORON;REEL/FRAME:014987/0019
Effective date: 20040211
|Aug 31, 2004||AS||Assignment|
Owner name: JPMORGAN CHASE BANK, AS COLLATERAL AGENT, TEXAS
Free format text: SECURITY AGREEMENT;ASSIGNOR:XEROX CORPORATION;REEL/FRAME:015722/0119
Effective date: 20030625
Owner name: JPMORGAN CHASE BANK, AS COLLATERAL AGENT,TEXAS
Free format text: SECURITY AGREEMENT;ASSIGNOR:XEROX CORPORATION;REEL/FRAME:015722/0119
Effective date: 20030625
|Jul 20, 2011||FPAY||Fee payment|
Year of fee payment: 4
|May 26, 2015||AS||Assignment|
Owner name: XEROX CORPORATION, NEW YORK
Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:035771/0458
Effective date: 20061204
|Jul 28, 2015||AS||Assignment|
Owner name: III HOLDINGS 6, LLC, DELAWARE
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:XEROX CORPORATION;REEL/FRAME:036201/0584
Effective date: 20150529
|Jul 28, 2015||FPAY||Fee payment|
Year of fee payment: 8