Publication number | US20070076968 A1 |

Publication type | Application |

Application number | US 11/534,942 |

Publication date | Apr 5, 2007 |

Filing date | Sep 25, 2006 |

Priority date | Sep 23, 2005 |

Also published as | CA2620885A1, CA2620893A1, CA2620893C, CN101283377A, CN101283377B, CN101317197A, CN101317197B, CN102196266A, CN102196266B, EP1927083A1, EP1927083A4, EP1927084A2, EP1927084A4, EP2461293A1, EP2461294A1, US7903893, US8000547, US8260071, US8265408, US8503807, US8559740, US20070071339, US20110123110, US20120039530, US20120308129, US20120321183, WO2007033483A1, WO2007033487A2, WO2007033487A3, WO2007033487A8 |

Publication number | 11534942, 534942, US 2007/0076968 A1, US 2007/076968 A1, US 20070076968 A1, US 20070076968A1, US 2007076968 A1, US 2007076968A1, US-A1-20070076968, US-A1-2007076968, US2007/0076968A1, US2007/076968A1, US20070076968 A1, US20070076968A1, US2007076968 A1, US2007076968A1 |

Inventors | En-hui Yang, Longji Wang |

Original Assignee | Yang En-Hui, Longji Wang |

Export Citation | BiBTeX, EndNote, RefMan |

Patent Citations (4), Referenced by (6), Classifications (28), Legal Events (2) | |

External Links: USPTO, USPTO Assignment, Espacenet | |

US 20070076968 A1

Abstract

A method, system and computer program product are provided for progressively encoding a digitized color image with M distinct colors allocated to a set of pixels using a data processing system, the digitized color image being provided by assigning each of the M distinct colors to at least one pixel in the set of pixels. This involves: (a) initializing a tree structure by providing at least one starting leaf node comprising a subset of pixels from the set of pixels; (b) determining at least one representative color for each starting leaf node; and (c) growing the tree structure by (i) selecting a leaf node n to become a non-leaf node n linked to two new leaf nodes based on an associated achievable cost, wherein the associated achievable cost is determined by 1) determining an associated change in distortion resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; 2) determining an associated increase in entropy rate resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; and, 3) determining the associated achievable cost based on the associated change in distortion and the associated increase in entropy rate; (ii) creating the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes; (iii) determining a representative color for each of the two new leaf nodes; and (iv) encoding the index information associated with the leaf node n, the representative colors of the two new leaf nodes, and information regarding each pixel in n.

Claims(28)

(a) initializing a tree structure by providing at least one starting leaf node comprising a subset of pixels from the set of pixels;

(b) determining at least one representative color for each starting leaf node; and

(c) growing the tree structure by

(i) selecting a leaf node n to become a non-leaf node n linked to two new leaf nodes based on an associated achievable cost, wherein the associated achievable cost is determined by

determining an associated change in distortion resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes;

determining an associated increase in entropy rate resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; and,

determining the associated achievable cost based on the associated change in distortion and the associated increase in entropy rate;

(ii) creating the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes;

(iii) determining a representative color for each of the two new leaf nodes; and

(iv) encoding the index information associated with the leaf node n, the representative colors of the two new leaf nodes, and information regarding each pixel in n.

determining an associated change in distortion resulting from allocating the pixel to the new leaf node;

determining an associated increase in entropy rate resulting from allocating the pixel to the new leaf node; and,

calculating the associated biased distortion measure based on the associated change in distortion and the associated increase in entropy rate.

determining a set of neighboring pixels, wherein each pixel in the set of neighboring pixels borders the pixel, and

determining an entropy rate increase for the pixel based on the set of neighboring pixels and the new leaf node.

determining at least one of A) an associated increased entropy rate resulting from turning the leaf node n into the non-leaf node n linked to the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes and, B) a new total number of leaf nodes resulting from turning the leaf node n into the non-leaf node n linked to the two new leaf nodes;

if the associated increased entropy rate is determined, then repeating step c) if the associated increased entropy rate is less than a target entropy rate; and

if the new total number of leaf nodes is determined, then repeating step c) if the new total number of leaf nodes is less than a target number of leaf nodes.

determining all of the colors of all pixels in the leaf node;

for each determined color, determining a number of pixels in the leaf node having that color in the digitized color image; and,

determining a centroid of all of the determined colors associated with the leaf node based on the number of pixels in the leaf node having each of the determined colors associated with the leaf node.

initializing a lower resolution image based on the at least one representative color for each starting node

where step (c)(iv) further comprises

determining information bits for all pixels in n according to allocation of pixels in n to one of the two new leaf nodes in step (c)(ii); and

encoding the information bits based on the lower resolution image by using arithmetic coding

initializing an encoded image based on the at least one representative color for each starting node;

wherein step (c) further comprises progressively updating the encoded image by appending the encoded bits obtained in step (c)iv).

a recording medium; and,

means recorded on the recording medium for instructing the computer system to perform the steps of:

(a) initializing a tree structure by providing at least one starting leaf node comprising a subset of pixels from the set of pixels;

(b) determining at least one representative color for each starting leaf node; and

(c) growing the tree structure by

(i) selecting a leaf node n to become a non-leaf node n linked to two new leaf nodes based on an associated achievable cost, wherein the associated achievable cost is determined by

determining an associated change in distortion resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes;

determining an associated increase in entropy rate resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; and,

determining the associated achievable cost based on the associated change in distortion and the associated increase in entropy rate;

(ii) creating the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes;

(iii) determining a representative color for each of the two new leaf nodes; and

(iv) encoding the index information associated with the leaf node n, the representative colors of the two new leaf nodes, and information regarding each pixel in n.

determining an associated change in distortion resulting from allocating the pixel to the new leaf node;

determining an associated increase in entropy rate resulting from allocating the pixel to the new leaf node; and,

calculating the associated biased distortion measure based on the associated change in distortion and the associated increase in entropy rate.

determining a set of neighboring pixels, wherein each pixel in the set of neighboring pixels borders the pixel, and

determining an entropy rate increase for the pixel based on the set of neighboring pixels and the new leaf node.

determining at least one of A) an associated increased entropy rate resulting from turning the leaf node n into the non-leaf node n linked to the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes and, B) a new total number of leaf nodes resulting from turning the leaf node n into the non-leaf node n linked to the two new leaf nodes;

if the associated increased entropy rate is determined, then repeating step c) if the associated increased entropy rate is less than a target entropy rate; and

if the new total number of leaf nodes is determined, then repeating step c) if the new total number of leaf nodes is less than a target number of leaf nodes.

determining all of the colors of all pixels in the leaf node;

for each determined color, determining a number of pixels in the leaf node having that color in the digitized color image; and,

determining a centroid of all of the determined colors associated with the leaf node based on the number of pixels in the leaf node having each of the determined colors associated with the leaf node.

the instructions recorded on the recording medium are further operable to instruct the computer system to initialize a lower resolution image based on the at least one representative color for each starting node

step (c)(iv) further comprises

determining information bits for all pixels in n according to allocation of pixels in n to one of the two new leaf nodes in step (c)(ii); and

encoding the information bits based on the lower resolution image by using arithmetic coding.

the instructions recorded on the recording medium are further operable to instruct the computer system to initialize an encoded image based on the at least one representative color for each starting node;

wherein step (c) further comprises progressively updating the encoded image by appending the encoded bits obtained in step (c)iv).

a recording medium; and,

means including a processor for performing the steps of:

(a) initializing a tree structure by providing at least one starting leaf node comprising a subset of pixels from the set of pixels;

(b) determining at least one representative color for each starting leaf node;and

(c) growing the tree structure by

(i) selecting a leaf node n to become a non-leaf node n linked to two new leaf nodes based on an associated achievable cost, wherein the associated achievable cost is determined by

determining an associated change in distortion resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes;

determining an associated increase in entropy rate resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; and,

determining the associated achievable cost based on the associated change in distortion and the associated increase in entropy rate;

(ii) creating the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes;

(iii) determining a representative color for each of the two new leaf nodes; and

(iv) encoding the index information associated with the leaf node n, the representative colors of the two new leaf nodes, and information regarding each pixel in n.

determining an associated change in distortion resulting from allocating the pixel to the new leaf node;

determining an associated increase in entropy rate resulting from allocating the pixel to the new leaf node; and,

calculating the associated biased distortion measure based on the associated change in distortion and the associated increase in entropy rate.

determining a set of neighboring pixels, wherein each pixel in the set of neighboring pixels borders the pixel, and

determining an entropy rate increase for the pixel based on the set of neighboring pixels and the new leaf node.

determining at least one of A) an associated increased entropy rate resulting from turning the leaf node n into the non-leaf node n linked to the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes and, B) a new total number of leaf nodes resulting from turning the leaf node n into the non-leaf node n linked to the two new leaf nodes;

if the associated increased entropy rate is determined, then repeating step c) if the associated increased entropy rate is less than a target entropy rate; and

if the new total number of leaf nodes is determined, then repeating step c) if the new total number of leaf nodes is less than a target number of leaf nodes.

determining all of the colors of all pixels in the leaf node;

for each determined color, determining a number of pixels in the leaf node having that color in the digitized color image; and,

determining a centroid of all of the determined colors associated with the leaf node based on the number of pixels in the leaf node having each of the determined colors associated with the leaf node.

the steps performed further comprise initializing a lower resolution image based on the at least one representative color for each starting node

step (c)(iv) further comprises

determining information bits for a plurality of pixels of the digitized color image corresponding to the two representative colors of the two new leaf nodes according to allocation of colors in n to one of the two new leaf nodes in step (c)(ii); and

encoding the information bits based on the lower resolution image by using arithmetic coding.

the steps performed further comprise initializing an encoded image based on the at least one representative color for each starting node;

wherein step (c) further comprises progressively updating the encoded image by appending the encoded bits obtained in step (c)iv)

Description

- [0001]The present invention relates to progressive encoding of image data. This application claims the benefit of U.S. Provisional Application Ser. No. 60/719,585, filed Sep. 23, 2005, the entire contents of which is hereby incorporated by reference.
- [0002]Pictures with thousands or millions colors can be represented as true color images, storing the exact color for each pixel in the formats such as BMP (bitmap) and JPEG. In contrast, pictures containing only a few (typically up to 256) colors can be stored more efficiently as colormapped images or palettized images like the well known GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format. A significant portion of web images are in the format of GIF or PNG.
- [0003]When browsing palettized images over low-bandwidth communication links, such as wireless links or dial-up links, progressive transmission of these palette images can support fast browsing since it enables the users to view complete versions of these images (but at a low resolution) before all progressively coded image data has been received. However, even though both GIF and PNG support an interlaced format, which refines the resolution of an image progressively by encoding the image in multiple passes and skipping a number of pixels in each pass, interlaced GIF and PNG are not in a progressive format in the strict sense that the color information of each pixel is refined progressively up to lossless recovery.
- [0004]Rauschenbach investigated the progressive refinement of color information for palette images in his so-called MCQ (Mobile visualization Color Quantized format) method (U. Rauschenbach, “Compression of palettized images with progressive coding of the color information,” in Proc. SPIE Visual
- [0005]Communications and Image Processing (VCIP2000), Perth, Australia, June 2000, hereinafter referred to as [1]). Unlike the interlaced GIF and PNG formats, the refinement of color information in MCQ is done bit-plane by bit-plane, and allows the early recognition of shapes with fine details (e. g. text), normally after decoding a small portion of the compressed bit stream. Although MCQ supports progressive refinement of color information in contrast to resolution refinement, it only achieves a compression ratio comparable to GIF or PNG format. To improve the compression ratio while supporting the progressive transmission of palette images, Chen et al. (X. Chen, S. Kwong, and J. Feng, “A new compression scheme for color-quantized images,”
*IEEE Transactions on Circuits and System for Video Technology,*Vol. 12, No. 10, pp. 904˜908, October 2002 , hereinafter referred to as [2]) proposed a new color refinement scheme based on MCQ and the distortion-based hierarchical splitting color quantization algorithm pioneered by Orchard and Bouman (M. Orchard and C. Bouman, “Color quantization of images,”*IEEE Transactions on Signal Processing,*Vol. 39, No. 12, pp. 2677˜2690, December 1991, hereinafter referred to as [3]). Unlike MCQ where the color information is refined bit-plane by bit-plane with the number of bit-planes equal to the color depth of the palette, Chen's algorithm refines the color information color by color. For lossless compression of palette images, the color splitting algorithm proposed in [2] achieves around 30˜40% size reduction compared to GIF or PNG format in addition to its decent feature of supporting the progressive transmission. - [0006]In accordance with an aspect of an embodiment of the invention, there is provided a method for progressively encoding a digitized color image with M distinct colors allocated to a set of pixels using a data processing system, the digitized color image being provided by assigning each of the M distinct colors to at least one pixel in the set of pixels. The method comprises: (a) initializing a tree structure by providing at least one starting leaf node comprising a subset of pixels from the set of pixels; (b) determining at least one representative color for each starting leaf node; and (c) growing the tree structure by (i) selecting a leaf node n to become a non-leaf node n linked to two new leaf nodes based on an associated achievable cost, wherein the associated achievable cost is determined by 1) determining an associated change in distortion resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; 2) determining an associated increase in entropy rate resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; and, 3) determining the associated achievable cost based on the associated change in distortion and the associated increase in entropy rate; (ii) creating the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes; (iii) determining a representative color for each of the two new leaf nodes; and (iv) encoding the index information associated with the leaf node n, the representative colors of the two new leaf nodes, and information regarding each pixel in n.
- [0007]In accordance with an embodiment of the invention, there is provided a computer program product for use on a computer to progressively encode a digitized color image with M distinct colors allocated to a set of pixels, the digitized color image being provided by assigning each of the M distinct colors to an associated subset of pixels in the set of pixels. The computer program product comprises a recording medium; and, means recorded on the recording medium for instructing the computer system to perform the steps of: (a) initializing a tree structure by providing at least one starting leaf node comprising a subset of pixels from the set of pixels; (b) determining at least one representative color for each starting leaf node; and (c) growing the tree structure by (i) selecting a leaf node n to become a non-leaf node n linked to two new leaf nodes based on an associated achievable cost, wherein the associated achievable cost is determined by 1) determining an associated change in distortion resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; 2) determining an associated increase in entropy rate resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; and, 3) determining the associated achievable cost based on the associated change in distortion and the associated increase in entropy rate; (ii) creating the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes; (iii) determining a representative color for each of the two new leaf nodes; and (iv) encoding the index information associated with the leaf node n, the representative colors of the two new leaf nodes, and information regarding each pixel in n.
- [0008]In accordance with a further embodiment of the invention, there is provided a system for progressively encoding a digitized color image with M distinct colors allocated to a set of pixels, the digitized color image being provided by assigning each of the M distinct colors to an associated subset of pixels in the set of pixels. The system comprises a recording medium; and, means including a processor for performing the steps of: (a) initializing a tree structure by providing at least one starting leaf node comprising a subset of pixels from the set of pixels; (b) determining at least one representative color for each starting leaf node; and (c) growing the tree structure by (i) selecting a leaf node n to become a non-leaf node n linked to two new leaf nodes based on an associated achievable cost, wherein the associated achievable cost is determined by 1) determining an associated change in distortion resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; 2) determining an associated increase in entropy rate resulting from turning the leaf node into the non-leaf node linked to the two new leaf nodes; and, 3) determining the associated achievable cost based on the associated change in distortion and the associated increase in entropy rate; (ii) creating the two new leaf nodes by allocating each pixel in n to one of the two new leaf nodes; (iii) determining a representative color for each of the two new leaf nodes; and (iv) encoding the index information associated with the leaf node n, the representative colors of the two new leaf nodes, and information regarding each pixel in n.
- [0009]For a better understanding of the various embodiments described herein and to show more clearly how they may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings which show at least one exemplary embodiment and in which:
- [0010]
FIG. 1 in a flow chart, illustrates an overall algorithm for entropy constrained color splitting for palette images with color wise splitting between leaf nodes in accordance with an aspect of an embodiment of the invention; - [0011]
FIG. 2 in a flow chart, illustrates an overall algorithm for entropy constrained color splitting for palette images with pixel wise splitting between leaf nodes in accordance with a further aspect of a further embodiment of the invention; - [0012]
FIG. 3 in a flow chart, illustrates an initialization sub-process of the methods ofFIGS. 1 and 2 ; - [0013]
FIG. 4 in a flow chart, illustrates an example of a sub-process of the method ofFIG. 1 in which a leaf node is split; - [0014]
FIG. 5 , in a flow chart, illustrates an example of a sub-process of the method ofFIG. 2 in which a leaf node is tentatively split; - [0015]
FIG. 6 , in a flow chart, illustrates a sub-process of the method ofFIG. 2 in which a leaf node is permanently split; - [0016]
FIG. 7 , in a flow chart, illustrates a variant of a sub-process of the method ofFIG. 1 for splitting a leaf node; - [0017]
FIG. 8 , in a flow chart, illustrates a further variant of splitting a leaf node in the method ofFIG. 1 ; - [0018]
FIG. 9 , in a flow chart, illustrates a sub-process of the methods ofFIGS. 1 and 2 in which new representative colors are progressively encoded; - [0019]
FIG. 10 , in a block diagram, illustrates a computer system in accordance with an embodiment of the invention; - [0020]
FIG. 11 , in a block diagram, illustrates a processor of the computer system ofFIG. 10 in more detail; - [0021]
FIGS. 12 *a*-*d,*in graphs, illustrates the progressive splitting of leaf nodes to provide progressive encoding of colored data in accordance with an aspect of an embodiment of the invention; - [0022]
FIG. 13 is a block diagram of an exemplary embodiment of a mobile device; - [0023]
FIG. 14 is a block diagram of an exemplary embodiment of a communication subsystem component of the mobile device ofFIG. 13 ; - [0024]Pictures with thousands or millions colors can be represented as true color images, storing the exact color for each pixel in the formats such as BMP (bitmap) and JPEG. In contrast, pictures containing only a few (typically up to 256) colors can be stored more efficiently as colormapped images or palettized images like the well known GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format. A significant portion of web images are in the format of GIF or PNG.
- [0025]When browsing palettized images over low-bandwidth communication links, such as wireless links or dial-up links, progressive transmission of these palette images can support fast browsing since it enables the users to view complete versions of these images (but at a low resolution) before all progressively coded image data has been received. However, even though both GIF and PNG support an interlaced format, which refines the resolution of an image progressively by encoding the image in multiple passes and skipping a number of pixels in each pass, interlaced GIF and PNG are not in a progressive format in the strict sense that the color information of each pixel is refined progressively up to lossless recovery.
- [0026]Rauschenbach investigated the progressive refinement of color information for palette images in his so-called MCQ (Mobile visualization Color Quantized format) method (U. Rauschenbach, “Compression of palettized images with progressive coding of the color information,” in
*Proc. SPIE Visual Communications and Image Processing*(*VCIP*2000), Perth, Australia, June 2000, hereinafter referred to as [1]). Unlike the interlaced GIF and PNG formats, the refinement of color information in MCQ is done bit-plane by bit-plane, and allows the early recognition of shapes with fine details (e. g. text), normally after decoding a small portion of the compressed bit stream. Although MCQ supports progressive refinement of color information in contrast to resolution refinement, it only achieves a compression ratio comparable to GIF or PNG format. To improve the compression ratio while supporting the progressive transmission of palette images, Chen et al. (X. Chen, S. Kwong, and J. Feng, “A new compression scheme for color-quantized images,”*IEEE Transactions on Circuits and System for Video Technology,*Vol. 12, No. 10, pp. 904-908, October 2002 , hereinafter referred to as [2]) proposed a new color refinement scheme based on MCQ and the distortion-based hierarchical splitting color quantization algorithm pioneered by Orchard and Bouman (M. Orchard and C. Bouman, “Color quantization of images,”*IEEE Transactions on Signal Processing,*Vol. 39, No. 12, pp. 2677˜2690, December 1991, hereinafter referred to as [3]). Unlike MCQ where the color information is refined bit-plane by bit-plane with the number of bit-planes equal to the color depth of the palette, Chen's algorithm refines the color information color by color. For lossless compression of palette images, the color splitting algorithm proposed in [2] achieves around 30˜40% size reduction compared to GIF or PNG format in addition to its decent feature of supporting the progressive transmission. - [0027]The color splitting algorithm in [2], [3] splits a representative color based on a distortion criterion only and does not consider the rate constraint at all. On the other hand, each color in a palette image is a 3-D vector. Color quantization or color splitting is essentially a vector quantization (VQ) problem (Y, Linde, A, Buzo, and R. Gray, “An algorithm for vector quantizer design,”
*IEEE Trans. Communications,*Vol. COM-28, No. 1, pp. 84-95, January 1980, hereinafter referred to as [4]). Entropy constrained vector quantization (ECVQ) or conditional entropy constrained vector quantization (CECVQ) proposed by Chou et al. (P. Chou, T. Lookabaugh, and R. Gray, “Entropy-constrained vector quantization,”*IEEE Trans. Acous., Speech, and Signal Processing,*Vol. 37, No. 1, pp. 31-42, January 1989 , hereinafter referred to as [5], and P. Chou and T. Lookabaugh, “Conditional entropy constrained vector quantization,” Proceeding ICASSP 1990, pp. 197-200, April 1990, hereinafter referred to as [6]) can trade off the distortion with the entropy rate. To reduce the complexity of a normal vector quantization algorithm and make it feasible in practical applications, tree-structured vector quantization (TSVQ) algorithms (A. Buzo, A. Gray, Jr., R. Gray, and J. Markel, “Speech coding based upon vector quantization,”*IEEE Trans. Acous., Speech, and Signal Processing,*Vol. ASSP-28, No. 5, pp. 562-574, October 1980, hereinafter referred to as [7], E. Riskin, and R. Gray, “A greedy tree growing algorithm for the design of variable rate vector quantizers,”*IEEE Transactions on Signal Processing,*Vol. 39, No. 11, pp. 2500˜2507, November 1991, hereinafter referred to as [8], and M. Mahesh, W. Pearlman, and L. Lu, “Variable-rate tree-structured vector quantizers,”*IEEE Trans. Information Theory,*Vol. 41, No. 4, pp. 917-930, July 1995, hereinafter referred to as [9]) have been developed. In this approach, we apply the idea of ECVQ/CECVQ and TSVQ into a binary tree based color splitting scenario and develop four entropy constrained color splitting algorithms for palette images, which not only result in progressive transmission schemes but significantly outperform the scheme in [2] in R-D (rate-distortion) sense. - [0028]The rest of this document is organized as follows. The next section first describes the unconditional entropy constrained color splitting problem, and then provides two splitting algorithms using a fixed Lagrangian multiplier and a dynamically determined Lagrangian multiplier, respectively, in accordance with an embodiment. In Section III, in accordance with another embodiment, a pixel-wise conditional entropy constrained color splitting algorithm using a fixed Lagrangian multiplier is described. In Section IV, a color-wise conditional entropy constrained color splitting algorithm is presented according to another embodiment, which adds a restriction that all the pixels with the same color value in the original image can be updated by the same representative color values during the splitting process.
- [0000]II. Unconditional Entropy Constrained Color Splitting
- [0029]A binary tree structure, which can preserve more correlation between color indices than a linear structure, is used in [2] and [3] to represent the relationship of the colors of a palette image for the purpose of partitioning the color map, or equivalently, splitting a representative color progressively during the encoding process. Assume that the color map of a palette image is C={c
_{0},c_{1}, . . . ,c_{M-1}}, where M is the number of distinct colors in the original palette image. Each color vector (or called color entry) c_{i }corresponds to a color value(r_{i},g_{i},b_{i}). Let f(c_{i}) be defined as the occurrence number of c_{i }in the given image. Our goal is to design a binary tree-based color splitting algorithm that minimizes the distortion subject to an unconditional entropy constraint. Such a color splitting algorithm partitions the color map C or splits the representative color of C progressively until the original palette image is losslessly reconstructed or a user-specified target number of colors is reached. The partitioning process yields a dynamic binary tree with each leaf node n corresponding to a subset S_{n }of C such that C=∪_{n}S_{n }and S_{m}∩S_{n}=Ø, where m≠n,0≦m,n<N and N is the total number of leaf nodes of the dynamic binary tree. Denote q_{n }as the representative color of S_{n }that is defined as the centroid of S_{n}. Then the dynamic binary tree, together with q_{n}, 0≦n<N, gives rise to a tree-structured vector quantizer such that as the dynamic binary tree grows, the current vector quantizer is fully embedded into a subsequent high rate vector quantizer. A key question is how to grow the dynamic binary tree. - [0030]Starting from the root node, which represents the original color map C, we will grow the dynamic binary tree by splitting a desired leaf node each time. The process of selecting and splitting such a desired leaf node is conducted so as to minimize a Lagrangian cost

*J=D+λ·R*(1)

where D and R are, respectively, the distortion and entropy of the reconstructed palette image corresponding to the newly created dynamic binary tree after one leaf node is split into two new leaf nodes. The Lagrangian multiplier λ, which represents the tradeoff of rate for distortion, may be determined dynamically for each leaf node being split or fixed during the entire splitting process. We consider both cases in this section. - [0031]Let T
_{N }be a binary tree having N leaf nodes and T_{N+1 }a binary tree generated from T_{N }by splitting a leaf node n of T_{N }into two child nodes n′ and n″. Let d(c_{i},q_{n}) be the color quantization error when a color entry c_{i}=(r_{i},g_{i},b_{i})∈C is replaced by a representative color q_{n}=({circumflex over (r)}_{n},ĝ_{n},{circumflex over (b)}_{n}). Specifically,

*d*(*c*_{i}*,q*_{n})=∥*c*_{i}*−q*_{n}∥^{2}=(*r*_{i}*−{circumflex over (r)}*_{n})^{2}+(g_{i}*−ĝ*_{n})^{2}+(*b*_{i}*−{circumflex over (b)}*_{n})^{2 }(2) - [0032]Then the distortion reduction resulting from splitting the leaf node n into n′ and n″ is
$\begin{array}{cc}\Delta \text{\hspace{1em}}{D}_{N}={D}_{N}-{D}_{N+1}=\sum _{{c}_{i}\in {S}_{n}\text{\hspace{1em}}}^{\text{\hspace{1em}}}\text{\hspace{1em}}f\left({c}_{i}\right)d\left({c}_{i},{q}_{n}\right)-\sum _{{c}_{i}\in {S}_{{n}^{\prime}}\text{\hspace{1em}}}^{\text{\hspace{1em}}}\text{\hspace{1em}}f\left({c}_{i}\right)d\left({c}_{i},{q}_{{n}^{\prime}}\right)-\sum _{{c}_{i}\in {S}_{{n}^{\mathrm{\prime \prime}}}\text{\hspace{1em}}}^{\text{\hspace{1em}}}\text{\hspace{1em}}f\left({c}_{i}\right)d\left({c}_{i},{q}_{{n}^{\mathrm{\prime \prime}}}\right)& \left(3\right)\end{array}$

where D_{N }and D_{N+1 }are, respectively, the distortion corresponding to the trees T_{N }and T_{N+1}; q_{n′}and q_{n″}are, respectively, the representative colors of S_{n′}and S_{n″}. Note that S_{n}=S_{n′}∪S_{n″}and S_{n′}∩S_{n″}=Ø. Furthermore, let$f\left({S}_{{n}^{\prime}}\right)=\sum _{{c}_{i}\in {S}_{{n}^{\prime}}\text{\hspace{1em}}}^{\text{\hspace{1em}}}\text{\hspace{1em}}f\left({c}_{i}\right)\text{\hspace{1em}}\mathrm{and}\text{\hspace{1em}}f\left({S}_{{n}^{\mathrm{\prime \prime}}}\right)=\sum _{{c}_{i}\in {S}_{{n}^{\mathrm{\prime \prime}}}\text{\hspace{1em}}}^{\text{\hspace{1em}}}\text{\hspace{1em}}f\left({c}_{i}\right)\text{\hspace{1em}}$

be respectively, the numbers of pixels whose color indices fall into S_{n′}and S_{n″}when S_{n }is partitioned into S_{n′}and S_{n″}, and denote f(S_{n})=f(S_{n′})+f(S_{n″}). Then the increment of the unconditional entropy resulting from splitting the leaf node n into n′ and n″ can be expressed as$\begin{array}{cc}\Delta \text{\hspace{1em}}{R}_{N}={R}_{N+1}-{R}_{N}=-f\left({S}_{{n}^{\prime}}\right){\mathrm{log}}_{2}\left(\frac{f\left({S}_{{n}^{\prime}}\right)}{f\left({S}_{n}\right)}\right)-f\left({S}_{{n}^{\mathrm{\prime \prime}}}\right){\mathrm{log}}_{2}\left(\frac{f\left({S}_{{n}^{\mathrm{\prime \prime}}}\right)}{f\left({S}_{n}\right)}\right)& \left(4\right)\end{array}$

where R_{N }and R_{N+1 }are, respectively, the entropy corresponding to the trees T_{N }and T_{N+1}. Note that when splitting the leaf node n, neither ΔD_{N }nor ΔR_{N }depends on other leaf nodes in this binary tree-based color splitting. - [0033]An iterative process can be applied to split the leaf node n such that a minimum cost is achieved. The unconditional entropy constrained color splitting algorithm with a fixed λ is summarized as follows.
- [0000]Algorithm
**1**: Unconditional Entropy Constrained Color Splitting Algorithm with a Fixed λ. - [0034]Step 1—Initialization.

Given a palette image / and a target number of colors N_{target }or a target rate R_{target}, obtain a binary tree T_{1 }which only has a root node. The representative color corresponding the root node is the centroid of the original color map, i.e.$\begin{array}{cc}{q}_{0}=\sum _{i=0}^{M-1}\text{\hspace{1em}}\left(f\left({c}_{i}\right){c}_{i}\right)/f\left({S}_{0}\right)& \left(5\right)\end{array}$

where S_{0}=C and$f\left({S}_{0}\right)=\sum _{i=0}^{M-1}\text{\hspace{1em}}f\left({c}_{i}\right).$

Calculate the principal eigenvalue of the cluster covariance matrix associated to S_{0 }as discussed in [2] and [3]. Set N=1; R_{1}=0; and D_{1}=σ^{2}, where σ^{2 }is the variance of C. Select a convergence criterion ε and label the root node as a leaf node. - [0035]Step 2—Find the Minimum Achievable Cost for Splitting Each Leaf Node.
- [0036]For each leaf node n of T
_{N}, use the following steps to find the minimum achievable cost from tentatively splitting n. - [0000]Step 2.1

Initialize J_{old}^{n }as D_{N}+λ·R_{N}.

Step 2.2

For each color c_{i}∈S_{n}, assign c_{i }to leaf node n′ or n″ based on the biased distortion measure. That is, assign c_{i }to n′ if$\begin{array}{cc}d\left({c}_{i},{q}_{{n}^{\prime}}\right)-{\mathrm{\lambda log}}_{2}\left(\frac{f\left({S}_{{n}^{\prime}}\right)}{f\left({S}_{n}\right)}\right)\le d\left({c}_{i},{q}_{{n}^{\prime}}\right)-{\mathrm{\lambda log}}_{2}\left(\frac{f\left({S}_{{n}^{\mathrm{\prime \prime}}}\right)}{f\left({S}_{n}\right)}\right)& \left(6\right)\end{array}$

Otherwise, assign c_{i }to n″. Note that q_{n′}, q_{n″}, f(S_{n′}), and f(S_{n″}) are unknown for the first iteration. Instead, the first iteration splits node n using distortion-based criterion as described in [2].

Step 2.3

Calculate the new representative colors, q_{n′}and q_{n″}, of the leaf nodes n′ and n″ as the centriods of the new partition S_{n′}and S_{n′}obtained in Step 2.2, and update f(S_{n′}) and f(S_{n″}). Calculate ΔD_{N }and ΔR_{N }resulting from the current splitting using (3) and (4), which results in a new cost J_{new}^{n}=(D_{N}−ΔD_{N})+λ·(R_{N}+ΔR_{N}). If |J_{old}^{n}−J_{new}^{n}|/J_{old}^{n}≦ε or only one color entry is included in S_{n′}or S_{n″}, the minimum achievable cost from splitting n is taken as J_{new}^{n}. Otherwise, replace J_{old}^{n }with J_{new}^{n }and go to step 2.2 for further iteration. - [0037]Step 3—Grow T
_{N }to T_{N+1}. - [0038]Find leaf node n which achieves the minimum cost among all the leaf nodes, i.e.,
${J}_{\mathrm{new}}^{n}=\underset{k\text{\hspace{1em}}\mathrm{is}\text{\hspace{1em}}a\text{\hspace{1em}}\mathrm{leaf}\text{\hspace{1em}}\mathrm{node}}{\mathrm{argmin}}{J}_{\mathrm{new}}^{k}.$

Split n to obtain a new binary tree T_{N+1 }via the following iterative procedure.

Step 3.1

Initialize J_{old }as D_{N}+λ·R_{N}.

Step 3.2

For each color entry c_{i}∈S_{n}, assign c_{i }to leaf node n′ or n″ using the biased distortion measure (6). Note that q_{n′}, q_{n″}, f(S_{n′}), and f(S_{n″}) in (6) are unknown for the first iteration. Instead, the first iteration splits node n using distortion-based criterion as described in [2].

Step 3.3

Calculate the new representative colors, q_{n′}and q_{n″}, of the leaf nodes n′ and n″ as the centriods of the new partition S_{n′}and S_{n″}obtained in Step 3.2, and update f(S_{n′}) and f(S_{n″}). Calculate ΔD_{N }and ΔR_{N }resulting from the current splitting using (3) and (4), which results in a new cost J_{new}=(D_{N}−ΔD_{N})+λ·(R_{N}+ΔR_{N}). If |J_{old}−J_{new}|/J_{old}≦ε or only one color entry is included in S_{n′}or S_{n″}, stop the iteration of splitting the leaf node n, extend the node n into two new leaf nodes labeled as n′ and n″ respectively, and go to step 4. Otherwise, replace J_{old }with J_{new }and go to step 3.2 for further iterations. - [0039]Step 4—Progressively Transmitting (or Merely Encoding and Then, Say, Storing) the Color Updating Bits.
- [0040]Send out the node index being split (i.e., n), the new representative colors (i.e., q
_{n′}and q_{n″}) as well as the color updating bits for pixels whose color fall into S_{n}. Update the total bit rate as R_{N+1}=R_{N}+ΔR_{N}. If N+1<N_{target }or R_{N+1}<R_{target}, then update the new distortion as D_{N+1}=D_{N}−ΔD_{N}; increase N by 1; and then go to step 2 for further splitting. Otherwise, stop the splitting and transmitting process. - [0041]During the above splitting process, Algorithm
**1**employs a fixed λ, which has the physical meaning of the slope of the distortion-rate function. Another approach is to use a variable λ and perform the splitting so as to find a good trade-off between the overall distortion decrease and rate increase. In other words, when we grow T_{N }into T_{N+1}, we aim at maximizing λ, where$\begin{array}{cc}\lambda =\frac{{D}_{N}-{D}_{N+1}}{{R}_{N+1}-{R}_{N}},& \left(7\right)\end{array}$

over all possible iterations at each leaf node of T_{N }and across all leaf nodes of T_{N}. At each leaf node of T_{N}, we find its maximum achievable λ through an iterative process, which is similar to the one to be used when the node is to be split. Specifically, the splitting begins with the assumption that λ is zero. The two representative colors that minimize (1) for this value of λ are found. The decrease in distortion and increase in rate are calculated. The new value of λ is the ratio of the decrease in distortion to the increase in rate. The splitting is now redone with this new value of λ. This iterative process continues until λ does not increase. The converged value is then regarded as the maximum achievable λ at this leaf node. The actual leaf node to be split is the one giving rise to the maximum converged value across all leaf nodes of T_{N}. Algorithm**2**below gives the detailed description.

Algorithm**2**: Unconditional Entropy Constrained Color Splitting Algorithm with a Dynamically Determined λ. - [0042]Step 1—Initialization.
- [0043]Given a palette image / and a target number of colors N
_{target }or a target rate R_{target}, obtain a binary tree T_{1 }which only has the root node. The representative color corresponding to the root node is the centroid of the color map that can be calculated using (5). Set N=1; R_{1}=0; and D_{1}=σ^{2}. Select a convergence criterion ε and label the root node as a leaf node. - [0044]Step 2—Find the Maximum Achievable λ for Each New Leaf Node.
- [0045]For each leaf node n of T
_{N}, use the following steps to find the maximum attainable λ_{n,max }from splitting n as if the leaf node n is to be split. - [0000]Step 2.1

Set λ_{n}^{t}=0 where t in λ_{n}^{t }is an iterative index.

Step 2.2

For each color c_{i}∈S_{n}, assign c_{i }to an intended new leaf node n′ or n″, based on the biased distortion measure. That is, assign c_{i }to n′ if$\begin{array}{cc}d\left({c}_{i},{q}_{{n}^{\prime}}\right)-{\lambda}_{n}^{t}{\mathrm{log}}_{2}\left(\frac{f\left({S}_{{n}^{\prime}}\right)}{f\left({S}_{n}\right)}\right)\le d\left({c}_{i},{q}_{{n}^{\mathrm{\prime \prime}}}\right)-{\lambda}_{n}^{t}{\mathrm{log}}_{2}\left(\frac{f\left({S}_{{n}^{\mathrm{\prime \prime}}}\right)}{f\left({S}_{n}\right)}\right)& \left(8\right)\end{array}$

Otherwise, assign c_{i }to node n″. Note that q_{n′}and q_{n″}are unknown for the first iteration. Instead, the first iteration splits the leaf node n using the distortion-based spitting algorithm discussed in [2]. Alternatively, we may perturb q_{n }into q_{n′}, q_{n″}for the first iteration. Even though f(S_{n′}) and f(S_{n″}) are also unknown for the first iteration, the algorithm works fine since λ_{n}^{t }is assumed to be zero for the first iteration.

Step 2.3

Update the new representative colors, q_{n′}and q_{n″}, of the leaf nodes n′ and n″ to be the centriods of the new partition S_{n′}and S_{n″}obtained in Step 2.2, respectively, and update f(S_{n′}) and f(S_{n″}) into the numbers of pixels whose colors fall into S_{n′}and S_{n″}obtained in Step 2.2, respectively. Calculate ΔD_{N }and ΔR_{N }resulting from the current splitting using (3) and (4). If$\frac{\Delta \text{\hspace{1em}}{D}_{N}}{\Delta \text{\hspace{1em}}{R}_{N}}>{\lambda}_{n}^{t},$

set${\lambda}_{n}^{t}=\frac{\Delta \text{\hspace{1em}}{D}_{N}}{\Delta \text{\hspace{1em}}{R}_{N}}$

and then go to step 2.2 for further iterations. Otherwise, take the current λ_{n}^{t }as the maximum achievable λ_{n,max }at the leaf node n. - [0046]Step 3—Grow T
_{N }to T_{N+1}. - [0047]Find the leaf node n that has the maximum value λ
_{n,max }among all the leaf nodes of${T}_{N},i.e.,{\lambda}_{n,\mathrm{max}}=\underset{k\text{\hspace{1em}}\mathrm{is}\text{\hspace{1em}}a\text{\hspace{1em}}\mathrm{leaf}\text{\hspace{1em}}\mathrm{node}}{\mathrm{argmax}}{\lambda}_{k,\mathrm{max}}.$

Split the leaf node n to obtain a new binary tree T_{N+1 }via the following iteration process.

Step 3.1

Initialize J_{old }as D_{N}+λ_{n,max}·R_{N}.

Step 3.2

For each color entry c_{i}∈S_{n}, assign c_{i }to an intended new leaf node n′ or n″ based on the resulting Lagrangian cost. That is, assign c_{i }to node n′ if$\begin{array}{cc}d\left({c}_{i},{q}_{{n}^{\prime}}\right)-{\lambda}_{n,\mathrm{max}}{\mathrm{log}}_{2}\left(\frac{f\left({S}_{{n}^{\prime}}\right)}{f\left({S}_{n}\right)}\right)\le d\left({c}_{i},{q}_{{n}^{\mathrm{\prime \prime}}}\right)-{\lambda}_{n,\mathrm{max}}{\mathrm{log}}_{2}\left(\frac{f\left({S}_{{n}^{\mathrm{\prime \prime}}}\right)}{f\left({S}_{n}\right)}\right)& \left(9\right)\end{array}$

Otherwise, assign c_{t }to node n″. Note that q_{n′}, q_{n″}, f(S_{n′}), and f(S_{n″}) are unknown for the first iteration. Instead, the first iteration splits node n using the distortion-based criterion as described in [2].

Step 3.3

Update the new representative colors, q_{n′}and q_{n″}, of the leaf nodes n′ and n″ to be the centriods of the new partition S_{n′}and S_{n″}obtained in Step 3.2, respectively, and update f(S_{n′}) and f(S_{n″}) into the numbers of pixels whose colors fall into S_{n′}and S_{n″}obtained in Step 3.2, respectively. Calculate ΔD_{N }and ΔR_{N }resulting from current splitting using (3) and (4), which results in a new cost J_{new}=(D_{N}−ΔD_{N})+λ_{n,max}·(R_{N}+ΔR_{N}) If |J_{old}−J_{new}|/J_{old}≦ε, or only one color entry is included in S_{n′}or S_{n″}, stop the iteration of splitting the leaf node n, extend the node n into two new leaf nodes labeled as n′ and n″ respectively, and go to step 4. Otherwise, replace J_{old }with J_{new }and go to step 3.2 for further iterations. - [0048]Step 4—Progressively Transmitting (or Merely Encoding and Then, Say, Storing) the Color Updating Bits.
- [0049]Send out the node index being split (i.e., n), the new representative colors (i.e., q
_{n′}and q_{n″}) as well as the update bits for pixels whose colors fall into S_{n}. Update the total bit rate as R_{N+1}=R_{N}+ΔR_{N}. If N+1<N_{target }or R_{N+1}<R_{target}, then update the new distortion as D_{N+1}=D_{N}−ΔD_{N}; increase N by 1; and then go to step 2. Otherwise, stop the splitting and transmitting process. - [0000]III. Conditional Entropy Constrained Color Splitting: Pixel-Wise Splitting
- [0050]In practical applications, context-based arithmetic coding like the one used in [2] is usually employed to encode the update bits in the entropy coding stage in Algorithms
**1**and**2**so as to obtain a high compression gain. In Algorithms**1**and**2**, however, the rate constraint used is an unconditional entropy. If we could use a conditional entropy as the constraint in a color splitting algorithm, the entropy used in the splitting process would match the entropy of the encoding stage more accurately. Consequently, better results in the rate-distortion sense can be obtained. In this section, we will extend the unconditional entropy constrained color splitting algorithm with a fixed λ to the conditional entropy constrained scenario. - [0051]For conditional entropy constrained color splitting, we will consider two separate cases. We call them pixel-wise splitting and color-wise splitting, respectively, in what follows. Pixel-wise splitting means pixels with the same color in the original image may be mapped to different leaf nodes during the splitting process. In other words, pixels with the same color in the original image may be updated by different representative colors during the splitting process based on different context. For a palette image with M distinct colors, in general, the original image may not be perfectly reconstructed after M-1 steps of node splitting even though the original palette image can be losslessly reconstructed eventually. On the other hand, color-wise splitting means all the pixels with the same color in the original image must be mapped to the same node during the splitting process. In other words, all the pixels with the same color in the original image will be updated by the same representative colors during the splitting process. For a palette image with M distinct colors, the original image is guaranteed to be perfectly reconstructed after M-1 steps of node splitting. We will develop the pixel-wise conditional entropy constrained color splitting algorithm in this section and consider the color-wise conditional entropy constrained color splitting algorithm in Section IV. We only consider using a fixed λ in both cases.
- [0052]The main difference between the unconditional entropy constrained color splitting algorithm and conditional entropy constrained color splitting algorithm is the calculation of the increase of entropy rate ΔR
_{N }shown in (4) and the biased distortion measure as shown in (6). To calculate the conditional entropy rate during the splitting process, we may use an arbitrary number of neighboring pixels for the context modeling purpose. In what follows, we calculate the conditional entropy rate conditioned on the left and upper pixels only. The extension to more neighboring pixels is straightforward. - [0053]As indicated by the name, a pixel-wise conditional entropy constrained color splitting algorithm updates the color information in a pixel-wise manner. Let c
_{p }be the original color of pixel p in C and denote Ψ_{n }as the pixel set corresponding to the leaf node n during the splitting process. For a pixel p∈Ψ_{n}, whose left and upper neighboring pixels' representative color indices are u and v in the current reconstructed image, mapping p to the leaf node n′ results in an increase of entropy under context u and v as$\begin{array}{cc}{H}_{{n}^{\prime}}^{u,v}=-{\mathrm{log}}_{2}\left(\frac{{f}_{u,v}\left({\Psi}_{{n}^{\prime}}\right)}{{f}_{u,v}\left({\Psi}_{{n}^{\prime}}\right)+{f}_{u,v}\left({\Psi}_{{n}^{\mathrm{\prime \prime}}}\right)}\right)& \left(10\right)\end{array}$

where f_{u,v}(Ψ_{n′}) and f_{u,v}(Ψ_{n″}) are respectively the number of pixels mapped to the leaf node n′ or n″ under context u and v from the previous iteration. Similarly, mapping p to the leaf node n″ results in an increase of entropy under context u and v as$\begin{array}{cc}{H}_{{n}^{\mathrm{\prime \prime}}}^{u,v}=-{\mathrm{log}}_{2}\left(\frac{{f}_{u,v}\left({\Psi}_{{n}^{\mathrm{\prime \prime}}}\right)}{{f}_{u,v}\left({\Psi}_{{n}^{\prime}}\right)+{f}_{u,v}\left({\Psi}_{{n}^{\mathrm{\prime \prime}}}\right)}\right)& \left(11\right)\end{array}$ - [0054]Each pixel p∈Ψ
_{n }is mapped to the leaf node n′ or n″ separately based on the resulting incremental Lagrangian cost. Pixel-wise conditional entropy constrained color splitting algorithm is summarized as follows. - [0000]Algorithm
**3**: Pixel-Wise Conditional Entropy Constrained Color Splitting Algorithm with Fixed λ. - [0055]Step 1—Initialization.
- [0056]Given a palette image / with color map C and a target number of colors N
_{target }or a target rate R_{target}, obtain a binary tree T_{1 }which only has a root node. Initialize N as 1. Set R_{1}=0 and D_{1}=σ^{2}. Calculate the representative color q_{0 }and the principal eigenvalue of the cluster covariance matrix associated to the root node. Select a convergence criterion E and label the root node as a leaf node. - [0057]Step 2—Find the Minimum Achievable Cost for Splitting Each Leaf Node.
- [0058]For each leaf n of T
_{N}, use the following steps to find the minimum achievable cost from tentatively splitting n. - [0000]Step 2.1

Initialize J_{old}^{n }as D_{N}+λ·R_{N}.

Step 2.2

For each pixel p∈Ψ_{n}, whose left and upper neighboring pixels' representative colors are u and v in current reconstructed image, map p to leaf node n′ or n″ based on the biased distortion measure. That is, map p to node n′ if

*d*(*c*_{p}*,q*_{n′})+λ·*H*_{n′}^{u,v}*≦d*(*c*_{p}*,q*_{n′})+λ·H_{n″}^{u,v }(12)

Otherwise, map p to node n″. Note that q_{n′}, q_{n″}, H_{n′}^{u,v}, and H_{n″}^{u,v }are unknown for the first iteration. Instead, the distortion-based criterion is used for the first iteration.

Step 2.3

Update the new representative colors, q_{n′}and q_{n″}, of the leaf nodes n′ and n″ and the statistics f_{u,v}(Ψ_{n′}) and f_{u,v}(Ψ_{n″}). Calculate H_{n′}^{u,v }and H_{n″}^{u,v }using (10) and (11). Calculate ΔD_{N }and ΔR_{N }where$\begin{array}{cc}\Delta \text{\hspace{1em}}{D}_{N}=\sum _{p\in {\Psi}_{n}}^{\text{\hspace{1em}}}\text{\hspace{1em}}d\left({c}_{p},{q}_{n}\right)-\sum _{p\in {\Psi}_{{n}^{\prime}}}^{\text{\hspace{1em}}}\text{\hspace{1em}}d\left({c}_{p},{q}_{{n}^{\prime}}\right)-\sum _{p\in {\Psi}_{{n}^{\mathrm{\prime \prime}}}}^{\text{\hspace{1em}}}\text{\hspace{1em}}d\left({c}_{p},{q}_{{n}^{\mathrm{\prime \prime}}}\right)\text{}\mathrm{and}& \left(13\right)\\ \Delta \text{\hspace{1em}}{R}_{N}=\sum _{u,v}^{\text{\hspace{1em}}}\text{\hspace{1em}}\left[{f}_{u,v}\left({\Psi}_{{n}^{\prime}}\right)\xb7{H}_{{n}^{\prime}}^{u,v}+{f}_{u,v}\left({\Psi}_{{n}^{\mathrm{\prime \prime}}}\right)\xb7{H}_{{n}^{\mathrm{\prime \prime}}}^{u,v}\right],& \left(14\right)\end{array}$

which results in a new cost J_{new}^{n}=(D_{N}−ΔD_{N})+λ·(R_{N}+ΔR_{N}). If |J_{old}^{n}−J_{new}^{n}|/J_{old}^{n}≦ε or only one pixel is included in Ψ_{n′}or Ψ_{n″}, the minimum achievable cost from splitting n is taken as J_{new}^{n}. Otherwise, replace J_{old}^{n }with J_{new}^{n }and go to step 2.2 for further iteration. - [0059]Step 3—Grow T
_{N }to T_{N+1}. - [0060]Find leaf node n which achieves the minimum cost among all the leaf nodes, i.e.,
${J}_{\mathrm{new}}^{n}=\underset{k\text{\hspace{1em}}\mathrm{is}\text{\hspace{1em}}a\text{\hspace{1em}}\mathrm{leaf}\text{\hspace{1em}}\mathrm{node}}{\mathrm{argmin}}{J}_{\mathrm{new}}^{k}.$

Split n to obtain a new binary tree T_{N+1 }via the following iterative procedure.

Step 3.1

Initialize J_{old }as D_{N}+λ·R_{N}.

Step 3.2

For each pixel p∈Ψ_{n}, whose left and upper neighboring pixels' representative colors are u and v in current reconstructed image, map p to leaf node n′ or n″ using the biased distortion measure (12). Note that q_{n′}, q_{n″}, H_{n′}^{u,v}, and H_{n″}^{u,v }in (12) are unknown for the first iteration. Instead, the distortion-based criterion is used for the first iteration.

Step 3.3

Update the new representative colors, q_{n′}and q_{n″}, of the leaf nodes n′ and n″ and the statistics f_{u,v}(Ψ_{n′}) and f_{u,v}(Ψ_{n″}). Calculate H_{n′}^{u,v }and H_{n″}^{u,v }using (10) and (11). Calculate ΔD_{N }and ΔR_{N }resulting from the current splitting using (13) and (14), which results in a new cost J_{new}=(D_{N}−ΔD_{N})+λ·(R_{N}+ΔR_{N}). If |J_{old}−J_{new}|/J_{old}≦ε, or only one pixel is included in Ψ_{n′}or Ψ_{n″}, stop the iteration of splitting node n, label nodes n′ and n″ as new leaf node, and go to step 4. Otherwise, replace J_{old }with J_{new }and go to step 3.2 for further iterations. - [0061]Step 4—Progressively Transmitting (or Merely Encoding and Then, Say, Storing) the Color Updating Bits.
- [0062]Send out the node index being split (i.e., n), the new representative colors (i.e., q
_{n′}and q_{n″}) as well as the color updating bits for all pixels p∈Ψ_{n}. Update the total bit rate as R_{N+1}=R_{N}+ΔR_{N}. If N+1<N_{target }or R_{N+1}<R_{target}, then update the new distortion as D_{N+1}=D_{N}−ΔD_{N}; increase N to by 1; and then go to step 2 for further splitting. Otherwise, stop the splitting and transmitting process. - [0000]IV. Conditional Entropy Constrained Color Splitting: Color-Wise Splitting
- [0063]Pixel-wise splitting means the pixels with the same color value in the original image may be updated to their final color through different intermediate representative colors. In other words, the pixels with the same color in the original image may be mapped to different nodes during the splitting process. In order to ensure the same color in the original image will be updated by the same representative colors during the splitting process, we may add a restriction that all the pixels with the same color in the original image will be mapped to the same node during the splitting process. This leads to the color-wise conditional entropy constrained splitting algorithm that will be described in detail in this section.
- [0064]During splitting a leaf node n into n′ and n″, the pixels corresponding to a color entry c
_{i}∈S_{n }may have different contexts u and v in the current reconstructed image, where each pixel p (c_{p}=c_{i}) corresponds to an entropy H_{n′}^{u,v }or H_{n″}^{u,v }as defined in (10) and (11). Assign c_{i }to n′ results in an increase of entropy per pixel as$\begin{array}{cc}{H}_{{n}^{\prime}}^{{c}_{i}}=\frac{\sum _{{c}_{p}={c}_{i}}^{\text{\hspace{1em}}}\text{\hspace{1em}}{H}_{{n}^{\prime}}^{u,v}}{f\left({c}_{i}\right)}& \left(15\right)\end{array}$ - [0065]Similarly, assign c
_{i }to n″ results in an increase of entropy per pixel as$\begin{array}{cc}{H}_{{n}^{\mathrm{\prime \prime}}}^{{c}_{i}}=\frac{\sum _{{c}_{p}={c}_{i}}^{\text{\hspace{1em}}}\text{\hspace{1em}}{H}_{{n}^{\mathrm{\prime \prime}}}^{u,v}}{f\left({c}_{i}\right)}& \left(16\right)\end{array}$ - [0066]Each color entry c
_{i}∈S_{n }is still mapped to leaf node n′ or n″ to minimize the resulting Lagrangian cost through an iterative process. The color-wise conditional entropy constrained color splitting algorithm is summarized as follows. - [0000]Algorithm
**4**: Color-Wise Conditional Entropy Constrained Color Splitting Algorithm with a Fixed λ. - [0067]Step 1—Initialization.
- [0068]Given a palette image / with color map C and a target number of colors N
_{target }or a target rate R_{target}, obtain a binary tree T_{1 }which only has a root node. Set S_{0}=C and N=1. Set R_{1}=0 and D_{1}=σ^{2}. Calculate the representative color of S_{0 }using Equation (5) and the principal eigenvalue of the cluster covariance matrix associated to S_{0}. Select a convergence criterion ε and label the root node as a leaf node. - [0069]Step 2—Find the Minimum Achievable Cost for Splitting Each Leaf Node.
- [0070]For each leaf node n of T
_{N}, use the following steps to find the minimum achievable cost from tentatively splitting n. - [0000]Step 2.1

Initialize J_{old}^{n }as D_{N}+λ·R_{N}.

Step 2.2

For each color c_{i}∈S_{n}, assign c_{i }to leaf node n′ or n″ based on the biased distortion measure. That is, assign c_{i }to n′ if

*d*(*c*_{i}*,q*_{n′})+λ·*H*_{n′}^{c}^{ 1 }*≦d*(*c*_{i}*,q*_{n″})+λ·H_{n″}^{c}^{ i }(17)

Otherwise, assign c_{i }to n″. Note that q_{n′}, q_{n″}, H_{n′}^{c}^{ i }, and H_{n″}^{c}^{ i }are unknown for the first iteration. Instead, distortion-based criterion is used for the first iteration.

Step 2.3

Update the new representative colors, q_{n′}and q_{n″}, of the leaf nodes n′ and n″, and the statistics f_{u,v}(Ψ_{n′}) and f_{u,v}(Ψ_{n″}). Calculate H_{n′}^{c}^{ i }and H_{n″}^{c}^{ i }using (15) and (16), which in turn use (10) and (11). Calculate ΔD_{N }and ΔR_{N }resulting from current splitting using (3) and (14), which results in a new cost J_{new}^{n}=(D_{N}−ΔD_{N})+λ·(R_{N}+ΔR_{N}). If |J_{old}^{n}−J_{new}^{n}|/J_{old}^{n}≦ε or only one color entry is included in S_{n′}or S_{n″}, the minimum achievable cost from splitting n is taken as J_{new}^{n}. Otherwise, replace J_{old}^{n }with J_{new}^{n }and go to step 2.2 for further iteration. - [0071]Step 3—Grow T
_{N }to T_{N+1}. - [0072]Find leaf node n which achieves the minimum cost among all the leaf nodes, i.e.,
${J}_{\mathrm{new}}^{n}=\underset{k\text{\hspace{1em}}\mathrm{is}\text{\hspace{1em}}a\text{\hspace{1em}}\mathrm{leaf}\text{\hspace{1em}}\mathrm{node}}{\mathrm{argmin}}{J}_{\mathrm{new}}^{k}.$

Split n to obtain a new binary tree T_{N+1 }via the following iterative procedure.

Step 3.1

Initialize J_{old }as D_{N}+λ·R_{N}.

Step 3.2

For each color c_{i}∈S_{n}, assign c_{i }to leaf node n′ or n″ using the biased distortion measure (17). Note that q_{n′}, q_{n″}, H_{n}^{c}^{ i }, and H_{n″}^{c}^{ i }are unknown for the first iteration. Instead, distortion-based criterion is used for the first iteration.

Step 3.3

Update the new representative colors, q_{n′}and q_{n″}, of the leaf nodes n′ and n″, and the statistics f_{u,v}(Ψ_{n′}) and f_{u,v}(Ψ_{n″}). Calculate H_{n′}^{c}^{ i }and H_{n″hu c}^{ i }using (15) and (16). Calculate ΔD_{N }and ΔR_{N }resulting from current splitting using (3) and (14), which results in a new cost J_{new}=(D_{N}−ΔD_{N})+λ·(R_{N}+ΔR_{N}). If |J_{old}−J_{new}|/J_{old}≦ε, or only one color entry is included in S_{n′}or S_{n″}, stop the iteration of splitting node n, label nodes n′ and n″ as new leaf node, and go to step 4. Otherwise, replace J_{old }with J_{new }and go to step 3.2 for further iterations. - [0073]
- [0074]Send out the node index being split (i.e., n), the new representative colors (i.e., q
_{n′}and q_{n″}) as well as the color updating bits for pixels whose color fall into S_{n}. Update the total bit rate as R_{N+1}=R_{N}+ΔR_{N}. If N+1<N_{target }or R_{N+1}<R_{target}, then update the new distortion as D_{N+1}=D_{N}−ΔD_{N}; increase N to by 1; and then go to step 2 for further splitting. Otherwise, stop the splitting and transmitting process. - [0075]Referring now to
FIG. 1 , the basic operation of algorithms**1**,**3**, and**4**will be described. Algorithms**1**,**3**, and**4**all employ a fixed Lagrangian multiplier λ representing a trade-off between distortion and entropy rate. Process**110**shows the steps common to these three algorithms. - [0076]Process
**110**begins at step**111**and immediately progresses to step**112**where the splitting process is initialized. Step**112**will be described in greater detail below with reference toFIG. 3 . Once the splitting process is initialized, the Lagrangian multiplier λ is selected at step**113**. The first iteration of sub-process**127**then begins. - [0077]Each iteration of sub-process
**127**splits a leaf node n by creating two new leaf nodes n′ and n″ and progressively encodes the new representative colors for the new leaf nodes as well as information regarding which pixels in the image should be updated using these colors. Once n is split, it becomes a non-leaf node. - [0078]The first objective of sub-process
**127**is to determine which leaf node must be split. This is done in steps**114**-**118**. At step**114**, a leaf node index n is initialized to 0. Leaf node n (the leaf node with index n) is then split according to the splitting process in algorithm**1**,**3**, or**4**at step**115**. Each of these splitting processes are described in greater detail below with reference toFIG. 4, 7 and**8**respectively. Each of these processes includes the determination of a minimum cost for splitting node n into two new leaf nodes based on the associated change in distortion and the associated increase in the entropy rate. Once the splitting of leaf node n is complete, sub-process**127**determines whether or not all leaf nodes in the tree structure have been split at step**117**using the variable N, representing the number of leaf nodes in the tree structure, which is initialized at step**314**of process**310**described below with reference toFIG. 3 . If all N leaf nodes have not been split, leaf node index n is incremented by one at step**116**and process**110**returns to step**115**to continue the splitting process. Once all leaf nodes in the tree structure have been split, process**110**continues to step**118**. At step**118**, the leaf node n which achieves the lowest minimum cost among all the leaf nodes is selected. - [0079]Once the leaf node n with the lowest minimum cost has been determined, n is split according to the splitting process in algorithm
**1**,**3**, or**4**at step**119**. Each of these splitting algorithms are described in greater detail below with reference toFIG. 4, 7 and**8**respectively. During this process, leaf node n becomes a non-leaf node and two new leaf nodes, n′ and n″, are created. At step**120**, leaf nodes n′ and n″ are labeled as new leaf nodes. - [0080]At step
**121**, the new representative colors are progressively encoded along with the color updating bits. Step**121**will be described in greater detail below with reference toFIG. 9 . - [0081]At step
**122**, the entropy rate for the new tree structure which includes n′ and n″ is calculated by adding the entropy rate for the old tree structure before n was split and the increase in entropy caused by the splitting of n into n′ and n″. - [0082]At step
**123**, the entropy rate for the new tree structure, calculated in step**122**, may be compared to a target entropy rate set during the initialization process described below with reference toFIG. 3 . Alternatively, the number of representative colors, which corresponds to the number of leaf nodes in the new tree structure, is compared to a target number of colors set during the initialization process described below with reference toFIG. 3 . If the new entropy rate is greater than or equal to the target entropy rate or if the number of colors in the new tree structure is greater than or equal to the target number of colors then process**110**proceeds to step**126**and terminates. Otherwise, process**110**continues to step**124**. - [0083]At step
**124**, the distortion for the new tree structure is calculated as the sum of the distortion of the tree structure before node n was split and the decrease is distortion resulting from splitting node n into leaf nodes n′ and n″. The number of leaf nodes in the tree structure N is then incremented by one at step**125**, the process returns to step**114**and the splitting process continues. - [0084]Referring now to
FIG. 2 , the basic operation of algorithm**2**will be described. Process**210**functions similarly to process**110**except that process**210**employs a dynamically determined Lagrangian multiplier rather than a fixed Lagrangian multiplier as in process**110**. - [0085]Process
**210**begins at step**211**and immediately progresses to step**212**where the splitting process is initialized. Step**212**will be described in greater detail with reference toFIG. 3 . The first iteration of sub-process**227**then begins. - [0086]Each iteration of sub-process
**227**splits a leaf node n by creating two new leaf nodes n′ and n″ and progressively encodes the new representative colors for the new leaf nodes along with information regarding which pixels in the image should be updated using these colors. Once n is split, it becomes a non-leaf node. - [0087]The first objective of sub-process
**227**is to determine which leaf node must be split. This is done in steps**214**-**218**. At step**214**, a leaf node index n is initialized to 0. Leaf node n (the leaf node with index n) is then tentatively split according to the tentative splitting process for algorithm**2**at step**215**. This tentative splitting process is described in greater detail below with reference toFIG. 5 . The tentative splitting processes includes the determination of a maximum achievable Lagrangian multiplier λ_{n,max }for each leaf node n. Once the tentative splitting of leaf node n is complete, the process determines whether or not all leaf nodes in the tree structure have been tentatively split at step**217**using the variable N, representing the number of leaf nodes in the tree structure, which is initialized at step**314**of process**310**described below with reference toFIG. 3 . If all N leaf nodes have not been tentatively split, leaf node index n is incremented at step**216**, process**210**returns to step**215**and the tentative splitting process continues. Once all N leaf nodes in the tree structure have been tentatively split, process**210**proceeds to step**218**. At step**218**, the leaf node n with the highest maximum achievable Lagrangian multiplier λ_{n,max}, calculated in step**215**, is selected. - [0088]Once the leaf node n with the highest maximum achievable Lagrangian multiplier λ
_{n,max }has been determined, leaf node n is permanently split according to the permanent splitting process for algorithm**2**at step**219**. The permanent splitting process is described in greater detail below with reference toFIG. 6 . During the permanent splitting process, leaf node n becomes a non-leaf node and two new leaf nodes, n′ and n″, are created. At step**220**, leaf nodes n′ and n″ are labeled as new leaf nodes. - [0089]At step
**221**, the new representative colors are progressively encoded along with the color updating bits. Step**221**will be described in greater detail below with reference toFIG. 9 . - [0090]At step
**222**, the entropy rate for the new tree structure which includes n′ and n″ is calculated by adding the entropy rate for the old tree structure before n was split and the increase in entropy caused by the splitting of n into n′ and n″. The increase in entropy is calculated using equation (4). - [0091]At step
**223**, the entropy rate for the new tree structure, calculated in step**222**, may be compared to a target entropy rate set during the initialization process described below with reference toFIG. 3 . Alternatively, the number of representative colors, which corresponds to the number of leaf nodes in the new tree structure, is compared to a target number of colors set during the initialization process described below with reference toFIG. 3 . If the new entropy rate is greater than or equal to the target entropy rate or if the number of colors in the new tree structure is greater than or equal to the target number of colors then process**210**proceeds to step**226**and terminates. Otherwise, the distortion for the new tree structure is calculated at step**224**as the sum of the distortion of the tree structure before node n was split and the decrease is distortion resulting from splitting node n into leaf nodes n′ and n″. The number of leaf nodes in the tree structure N is then incremented at step**225**, process**210**returns to step**214**and the splitting process continues. - [0092]Referring now to
FIG. 3 , the initialization of the splitting process will be described. Process**310**begins at step**311**and immediately progresses to step**312**. - [0093]At step
**312**, a target number of colors N_{target }or a target entropy rate R_{target }are specified. These values will be used in steps**123**and**223**of processes**110**and**210**respectively to determine when the splitting process should be terminated. - [0094]A number of variables are then initialized. At step
**313**, the initial representative color for the starting leaf node is calculated, using equation 5, as the centroid of the original color map for the image. The number of leaf nodes in the tree structure, N, is initialized to 1 at step**314**and the entropy rate for the initial tree structure is set to 0 in step**315**. The distortion associated with the initial tree structure is set to be the variance of the original color map for the image at step**316**. At step**317**a convergence criterion ε is selected. The convergence criterion is used in steps**422**,**622**,**723**and**824**of processes**410**,**610**,**710**and**810**respectively to determine when the splitting process for a node should terminate. These processes will be described in detail below. Once these variables have been initialized, process**310**continues to step**318**and terminates. - [0095]Referring now to
FIG. 4 , the process of splitting a leaf node n according to algorithm**1**will be described. Specifically, process**410**transforms n into a non-leaf node, creates two new leaf nodes n′ and n″ and assigns each color in n to either n′ or n″. The set of colors assigned to a node n is denoted as S_{n}. - [0096]Process
**410**begins at step**411**and immediately progresses to step**412**. At step**412**, a cost measurement J^{n}_{old }is initialized to the value attained by adding the distortion of the current tree structure to a product obtained by multiplying the entropy rate of the current tree structure by the Lagrangian multiplier λ selected in step**113**of process**110**. - [0097]At step
**413**, an iterative index t, used to keep track of the number of iterations of sub-process**427**, is initialized to 0. The iterative index t is used at step**414**to determine whether this is the first iteration of sub-process**427**(i.e. t=0). If this is the first iteration of sub-process**427**, at step**415**, each color in n is assigned to n′ or to n″ based on a distortion based criterion as described in [2]. In subsequent iterations of sub-process**427**, where t≠0, each color in n is assigned to n′ or n″ based on the biased distortion measure of equation (6) at step**416**. - [0098]Next, a number of values are calculated based on n′ and n″. First the new representative colors for n′ and n″, q
_{n′}and q_{n″}are calculated at step**417**. At step**418**, the number of occurrences in the original image of each color in n′ and n″, f(S_{n′}) and f(S_{n″}) respectively, are updated. At step**419**, the reduction in distortion resulting from splitting node n into n′ and n″, ΔD_{N }is calculated using equation (3). At step**420**, the increase in entropy rate resulting from splitting node n into n′ and n″, ΔR_{N }is calculated using equation (4). Finally, at step**421**, the values of ΔD_{N }and ΔR_{N }calculated in steps**419**and**420**respectively are used to calculate a cost measure J^{n}_{new }representing the cost of the new tree structure. - [0099]At step
**422**, the convergence criterion ε, selected in step**317**of process**310**, is used to determine whether sub-process**427**should be terminated. If the change in cost resulting from the most recent split, calculated as the difference between J^{n}_{new }and J^{n}_{old }divided by J^{n}_{old}, is less than or equal to the convergence criterion E then another iteration of sub-process**427**is not required and the process**410**proceeds to step**426**and terminates. Otherwise, process**410**proceeds to step**423**. - [0100]At step
**423**, the sets of colors assigned to n′ and n″, S_{n′}and S_{n″}respectively, are examined to determine if either set includes only one color. If either set S_{n′}or S_{n″}does include only one color then process**410**proceeds to step**426**and terminates. If both S_{n′}and S_{n″}include more than one color, sub-process**427**will be repeated by incrementing iterative index t by one, at step**424**, setting J^{n}_{old }to the value of J^{n}_{new }at step**425**, and returning to step**414**. - [0101]Referring now to
FIG. 5 , the process of tentatively splitting a leaf node n according to algorithm**2**will be described. Specifically, process**510**determines the maximum value for a Lagrangian multiplier λ_{n,max }if n were split into two new leaf nodes n′ and n″ and each color in n were assigned to n′ or n″. The set of colors assigned to a node n is denoted as S_{n}. - [0102]Process
**510**begins at step**511**and immediately progresses to step**512**. At step**512**, an iterative index t, used to keep track of the number of iterations of sub-process**527**, is initialized to 0. The value of λ^{t}_{n}, representing the dynamically determined Lagrangian multiplier for node n and iteration t of sub-process**527**is initialized to 0 at step**513**. - [0103]The iterative index t is used at step
**514**to determine whether this is the first iteration of sub-process**527**(i.e. t=0). If this is the first iteration of sub-process**527**, at step**515**, each color in n is assigned to n′ or to n″ based on a distortion based criterion as described in [2]. In subsequent iterations of sub-process**527**, where t≠0, each color in n is assigned to n′ or n″ based on the biased distortion measure of equation (8) at step**516**. - [0104]Next, a number of values are calculated based on n′ and n″. First the new representative colors for n′ and n″, q
_{n′}and q_{n″}are calculated at step**517**. At step**518**, the number of occurrences in the original image of each color in n′ and n″, f(S_{n′}) and f(S_{n″}) respectively, are calculated. At step**519**, the reduction in distortion resulting from splitting node n into n′ and n″, ΔD_{N }is calculated using equation (3). At step**520**, the increase in entropy rate resulting from splitting node n into n′ and n″, ΔR_{N }is calculated using equation (4). - [0105]At step
**521**, the decrease in distortion, ΔD_{N }divided by the increase in entropy rate, ΔR_{N}, is compared to the Lagrangian multiplier λ^{t}_{n}. If ΔD_{N}/ΔR_{N }is less than or equal to λ^{t}_{n }then the maximum achievable Lagrangian multiplier for node n, λ_{n,max }is set to λ^{t}_{n }at step**522**and process**510**proceeds to step**526**and terminates. If ΔD_{N}/ΔR_{N }is greater than λ^{t}_{n }then process**510**proceeds to step**523**. - [0106]At step
**523**, the sets of colors assigned to n′ and n″, S_{n′}and S_{n″}respectively, are examined to determine if either set includes only one color. If either set S_{n′}or S_{n″}does include only one color then process**510**proceeds to step**522**where the maximum achievable Lagrangian multiplier for node n, λ_{n,max}, is set to λ^{t}_{n }at step**522**and process**510**proceeds to step**526**and terminates. If both S_{n′}and S_{n″}includes more one color, sub-process**527**will be repeated by incrementing iterative index t by one, at step**524**, setting λ^{t}_{n }to be ΔD_{N}/ΔR_{N }at step**525**, and returning to step**514**. - [0107]Referring now to
FIG. 6 , the process of permanently splitting a leaf node n according to algorithm**2**will be described. Specifically, process**610**transforms n into a non-leaf node, creates two new leaf nodes n′ and n″ and assigns each color in n to either n′ or n″. The set of colors assigned to a node n is denoted as S_{n}. - [0108]Process
**610**begins at step**611**and immediately progresses to step**612**. At step**612**, a cost measurement J_{old }is initialized to the value attained by adding the distortion of the current tree structure, D_{N}, to a product obtained by multiplying the entropy rate of the current tree structure by the Lagrangian multiplier λ_{n,max }determined in step**218**of process**210**. - [0109]At step
**613**, an iterative index t, used to keep track of the number of iterations of sub-process**627**, is initialized to 0. The iterative index t is used at step**614**to determine whether this is the first iteration of sub-process**627**(i.e. t=0). If this is the first iteration of sub-process**627**, at step**615**, each color in n is assigned to n′ or to n″ based on a distortion based criterion as described in [2]. In subsequent iterations of sub-process**627**, where t≠0, each color in n is assigned to n′ or n″ based on the biased distortion measure of equation (9) at step**616**. - [0110]Next, a number of values are calculated based on n′ and n″. First the new representative colors for n′ and n″, q
_{n′}and q_{n″}are calculated at step**617**. At step**618**, the number of occurrences in the original image of each color in n′ and n″, f(S_{n′}) and f(S_{n″}) respectively, are calculated. At step**619**, the reduction in distortion resulting from splitting node n into n′ and n″, ΔD_{N }is calculated using equation (3). At step**620**, the increase in entropy rate resulting from splitting node n into n′ and n″, ΔR_{N }is calculated using equation (4). Finally, at step**621**, the values of ΔD_{N }and ΔR_{N }calculated in steps**619**and**620**respectively are used to calculate a cost measure J_{new }representing the cost of the new tree structure. - [0111]At step
**622**, the convergence criterion ε, selected in step**317**of process**310**, is used to determine whether sub-process**627**should be terminated. If the change in cost resulting from the most recent split, calculated as the difference between J_{new }and J_{old }divided by J_{old}, is less than or equal to the convergence criterion ε then another iteration of sub-process**627**is not required and the process**610**proceeds to step**626**and terminates. Otherwise, process**610**proceeds to step**623**. - [0112]At step
**623**, the sets of colors assigned to n′ and n″, S_{n′}and S_{n″}respectively, are examined to determine if either set includes only one color. If either set S_{n′}or S_{n″}does include only one color then process**610**proceeds to step**626**and terminates. If both S_{n′}and S_{n″}include more than one color, sub-process**627**will be repeated by incrementing iterative index t by one at step**624**, setting J_{old }to the value of J_{new }at step**625**, and returning to step**614**. - [0113]Referring now to
FIG. 7 , the process of splitting a leaf node n according to algorithm**3**will be described. In this process, each leaf node n has a corresponding set of pixels, denoted Ψ_{n}. Process**710**transforms n into a non-leaf node, creates two new leaf nodes n′ and n″ and assigns each pixel in Ψ_{n }to either n′ or n″. - [0114]Process
**710**begins at step**711**and immediately progresses to step**712**. At step**712**, a cost measurement J^{n}_{old }is initialized to the value attained by adding the distortion of the current tree structure to a product obtained by multiplying the entropy rate of the current tree structure by the Lagrangian multiplier λ selected in step**113**of process**110**. - [0115]At step
**713**, an iterative index t, used to keep track of the number of iterations of sub-process**728**, is initialized to 0. The iterative index t is then used at step**714**to determine whether this is the first iteration of sub-process**728**(i.e. t=0). If this is the first iteration of sub-process**728**, at step**715**, each pixel in Ψ_{n }is assigned to n′ or to n″ based on a distortion based criterion as described in [2]. In subsequent iterations of sub-process**728**, where t≠0, each pixel in Ψ_{n }is assigned to n′ or n″ based on the biased distortion measure of equation (12) at step**716**. - [0116]Next, a number of values are calculated based on n′ and n″. First the new representative colors for n′ and n″, q
_{n′}and q_{n″}are calculated at step**717**. At step**718**, f_{u,v}(Ψ_{n′}) and f_{u,v}(Ψ_{n″}) are updated where u and v are the representative color indices of the left and upper neighboring pixels of a pixel p in the current reconstructed image and f_{u,v}(Ψ_{n}) is the number of pixels mapped to the leaf node n under context u and v. At step**719**, H_{n′}^{u,v }and H_{n″}^{u,v}, the entropy rates of mapping each p∈Ψ_{n }to n′ or n″ under context u and v are calculated using equations (10) and (11) respectively. At step**720**, the reduction in distortion resulting from splitting node n into n′ and n″, ΔD_{N}, is calculated using equation (13). At step**721**, the increase in entropy rate resulting from splitting node n into n′ and n″, denoted ΔR_{N}, is calculated using equation (14) which takes into account the entropy values calculated in step**719**. Finally, at step**722**, the values of ΔD_{N }and ΔR_{N }calculated in steps**720**and**721**respectively are used to calculate a cost measure J^{n}_{new }representing the cost of the new tree structure. - [0117]At step
**723**, the convergence criterion ε, selected in step**317**of process**310**, is used to determine whether sub-process**728**should be terminated. If the change in cost resulting from the most recent split, calculated as the difference between J^{n}_{new }and J^{n}_{old }divided by J^{n}_{old}, is less than or equal to the convergence criterion ε then another iteration of sub-process**728**is not required and the process**710**proceeds to step**727**and terminates. Otherwise, process**710**proceeds to step**724**. - [0118]At step
**724**, the sets of pixels assigned to n′ and n″, Ψ_{n′}and Ψ_{n″}respectively, are examined to determine if either set includes only one pixel. If either set Ψ_{n′}or Ψ_{n″}does include only one pixel then process**710**proceeds to step**727**and terminates. If both Ψ_{n′}and Ψ_{n″}include more than one pixel, sub-process**728**will be repeated by incrementing iterative index t by one, at step**725**, setting J^{n}_{old }to the value of J^{n}_{new }at step**726**, and returning to step**714**. - [0119]Referring now to
FIG. 8 , the process of splitting a leaf node n according to algorithm**4**will be described. In this process, each leaf node n has a corresponding set of pixels, denoted Ψ_{n }as well as a corresponding set of colors, S_{n}. Process**810**transforms n into a non-leaf node, creates two new leaf nodes n′ and n″ and assigns each color in n to either n′ or n″. - [0120]Process
**810**begins at step**811**and immediately progresses to step**812**. At step**812**, a cost measurement J^{n}_{old }is initialized to the value attained by adding the distortion of the current tree structure to a product obtained by multiplying the entropy rate of the current tree structure by the Lagrangian multiplier λ selected in step**113**of process**110**. - [0121]At step
**813**, an iterative index t, used to keep track of the number iterations of sub-process**829**, is initialized to 0. The iterative index t is then used at step**814**to determine whether this is the first iteration of sub-process**829**(i.e. t=0). If this is the first iteration of sub-process**829**, at step**815**, each color in n is assigned to n′ or to n″ based on a distortion based criterion as described in [2]. In subsequent iterations of sub-process**829**, where t≠0, each color in n is assigned to n′ or n″ based on the biased distortion measure shown in equation (17) at step**816**. - [0122]Next, a number of values are calculated based on n′ and n″. First the new representative colors for n′ and n″, q
_{n′}and q_{n″}are calculated at step**817**. At step**818**, f_{u,v}(Ψ_{n′}) and f_{u,v}(Ψ_{n″}) are updated where f_{u,v}(Ψ_{n}) is the number of pixels assigned to node n under context u and v. At step**819**, the entropy rates of mapping each p∈Ψ_{n }to n′ or n″ under context u and v are calculated using equations (10) and (11) respectively. At step**820**, the increase in entropy per pixel resulting from the assignment of c_{i }to n′ where c_{i }∈ S_{n }is calculated using equation (15) and the same is done for n″ using equation (16). At step**821**, the reduction in distortion resulting from splitting node n into n′ and n″, ΔD_{N}, is calculated using equation (3). At step**822**, the increase in entropy rate resulting from splitting node n into n′ and n″, ΔR_{N }is calculated using equation (14) which takes into account the entropy values calculated in step**819**. Finally, at step**823**, the values of ΔD_{N }and ΔR_{N}, calculated in steps**821**and**822**respectively, are used to calculate a cost measure J^{n}_{new }representing the cost of the new tree structure. - [0123]At step
**824**, the convergence criterion ε, selected in step**317**of process**310**, is used to determine whether sub-process**829**should be terminated. If the change in cost resulting from the most recent split, calculated as the difference between J^{n}_{new }and J^{n}_{old }divided by J^{n}_{old}, is less than or equal to the convergence criterion ε then another iteration of sub-process**829**is not required and the process**810**proceeds to step**828**and terminates. Otherwise, process**810**proceeds to step**825**. - [0124]At step
**825**, the sets of colors assigned to n′ and n″, S_{n′}and S_{n″}respectively, are examined to determine if either set includes only one color. If either set S_{n′}or S_{n″}does include only one color then process**810**proceeds to step**828**and terminates. If both S_{n′}and S_{n″}include more than one color, sub-process**829**will be repeated by incrementing iterative index t by one at step**826**, setting J^{n}_{old }to the value of J^{n}_{new }at step**827**, and returning to step**814**. - [0125]Referring now to
FIG. 9 , the process**910**of progressively encoding the new representative colors and color updating bits will be described. - [0126]Process
**910**begins at step**911**and immediately proceeds to step**912**where the node index of the node n, which was selected to be split at step**118**of process**110**or at step**218**of process**210**, is encoded. At step**913**, the representative colors of n′ and n″, q_{n′}and q_{n″}, are encoded. - [0127]At step
**914**, a pixel index i, used to keep track of which pixel is currently being considered by sup-process**922**, is initialized to 0. Process**910**then enters the first iteration of sub-process**922**. - [0128]If pixel i or the color from the original color palette corresponding to pixel i have not been assigned to node n then process
**910**proceeds to step**919**. If pixel i or the color from the original color palette corresponding to pixel i have been assigned to node n then process**910**proceeds to step**916**. At step**916**, process**910**determines whether pixel i or the color from the original color palette corresponding to pixel i was assigned to n′ or to n″ during step**119**of process**110**or step**219**of process**210**. If pixel i or the color from the original color palette corresponding to pixel i was assigned to n′ then the color updating bit corresponding to pixel i is encoded as 1. Otherwise, pixel i or the color from the original color palette corresponding to pixel i must have been assigned to n″ then the color updating bit corresponding to pixel i is encoded as 0. - [0129]Process
**910**then proceeds to step**919**where it is determined whether all the pixels in the original image have been considered by sub-process**922**. If not, the pixel index i is incremented by one in step**920**and process**910**returns to step**915**. If all the pixels from the original image have been considered by sub-process**922**then process**910**proceeds to step**921**and terminates. - [0130]Once process
**910**is completed, the encoded information can be used to update the color of all the pixels assigned to n (in the case of algorithm**3**) or all the pixels whose color is assigned to n (in the case of algorithms**1**,**2**and**4**). If the color updating bit associated with pixel p is 1 then p will be updated to the representative color q_{n′}in the current reconstructed image. If the color updating bit associated with pixel p is 0 then p will be updated to the representative color q_{n″}in the current reconstructed image. - [0131]Referring to
FIG. 10 , there is illustrated in a block diagram, a computer system**1010**in accordance with an aspect of an invention. The computer system**1010**comprises a memory**1012**for storing color image data, a monitor**1014**for displaying digital color images, a CPU**1016**for image processing and for providing progressive encoding of color data based on both distortion and entropy rate, and an encoder**1018**for encoding the color image data. After the color image data is encoded, the encoded color image data can be stored, or can be transmitted from the computer system**1010**via transmission conduit**1020**. - [0132]Referring to
FIG. 11 , the CPU**1016**ofFIG. 10 is illustrated in more detail. Specifically, as shown the CPU**1016**comprises a tree management module**1122**, a distortion quantizer**1124**and an entropy rate quantizer**1126**. As shown, the tree management module**1122**is linked to both the distortion quantizer**1124**and the entropy rate quantizer**1126**. - [0133]In operation, the tree management module
**1122**is operable to perform operations on a tree structure, such as, for example, both initializing the tree structure by dividing up the colors between the initial starting leaf nodes, which may be just the root node, or, alternatively a plurality of starting leaf and subsequently growing the tree structure. As described above, the process of growing the tree structure is based on both entropy rate and distortion; hence, the tree management module**1122**can be linked to both the entropy rate quantizer**1126**and the distortion quantizer**1124**. That is, when selecting a leaf node to be divided to become a non-leaf node linked to two new leaf nodes, the tree management module**1122**can experiment with a particular leaf node by allocating a color in that leaf node to one, or the other, of two new leaf nodes resulting from splitting that original leaf node. Based on both the entropy rate and distortion respectively determined by the entropy rate quantizer**1126**and the distortion quantizer**1124**, the tree management module**1122**can determine a particular color allocation between two new leaf nodes of all the colors in the prospective parent node for those two new leaf nodes. By performing this operation over many or all of the leaf nodes, the tree management module**1122**can determine both the particular leaf node to be divided, as well as the color allocation between the two new leaf nodes resulting from this division. The tree management module**1122**can also perform other functions, such as, for example, determining representative colors for new leaf nodes, and maintaining a running count of the total number of leaf nodes. - [0134]According to Algorithm
**3**described above, the same color in different pixel positions may be assigned to different leaf nodes as leaf node assignment is conducted on a pixel-by-pixel basis, determined both by the context of the particular pixel and color pairs. In implementing Algorithm**3**, the operation of entropy rate quantizer**1126**and distortion quantizer**1124**is analogous to that described above. Specifically, whether it is a color or a pixel being allocated to a particular new leaf node in a pair of new leaf nodes, the entropy rate quantizer will determine the marginal increase in entropy rate resulting from this assignment, while the distortion quantizer determines any change in distortion. Of course, as described above, how the change in entropy rate, for example, is determined will be different in Algorithm**3**as this will also depend on how neighboring pixels are assigned between the respective new leaf nodes. - [0135]Referring to
FIGS. 12 *a*-*d,*the process of progressively splitting leaf nodes to provide progressive encoding of color data is illustrated in a sequence of graphs. In the graph ofFIG. 12 *a,*only a root node**1200**is provided. All of the colors in the M distinct colors of the original digitized color image are included in this root node. In the first step of the iterative method described above, the root node**1200**is split into two new leaf nodes**1202**and**1204**. With this initial division of the root node**1200**, there was, of course, no choice about the particular leaf node to be divided, there being only one node. However, the allocation of all of the M distinct colors in the root node between the new leaf nodes**1202**and**1204**is jointly based on entropy rate and distortion criteria as described above. Specifically, the entropy rate quantizer**1126**can determine a marginal increase in entropy rate for allocating a particular color to either of the new leaf nodes**1202**and**1204**. Similarly, the distortion quantizer**1124**can determine a new distortion measure for allocating this color to each of the new leaf nodes**1202**and**1204**. An associated biased distortion measure based on both rate and distortion can then be determined for each of the leaf nodes**1202**and**1204**, and the color can then be allocated to which ever of leaf nodes**1202**and**1204**has a lower associated biased distortion measure. - [0136]The transition from the graph of
FIG. 12 *b*to the graph ofFIG. 12 *c*is more complicated as in this instance there are two leaf nodes, leaf node**1202**and leaf node**1204**, to choose between dividing. Again, the decision of which of leaf nodes**1202**or**1204**to divide is based on both distortion and entropy rate considerations. According to some embodiments (see Algorithms**1**,**3**and**4**described above) of the present invention where a fixed Lagrangian multiplier is employed, a cost for turning that leaf node into a non-leaf node linked to two new leaf nodes (dividing the leaf node) is determined. Then, whichever of the leaf nodes**1202**and**1204**has the lower cost, will be divided. According to another embodiment (see Algorithm**2**described above), the trade-off or Lagrangian multiplier will be determined for each possible leaf node. Then, whichever of the leaf nodes**1202**and**1204**achieves a larger Lagrangian multiplier (i.e., the best trade-off between distortion decrease and entropy rate increase), will be divided. - [0137]The cost for dividing each of the leaf nodes
**1202**and**1204**is generally determined by the entropy rate quantizer**1126**and the distortion quantizer**1124**. That is, the entropy rate quantizer**1126**can determine an associated increase in entropy rate resulting from turning a particular leaf node into a non-leaf node linked to two new leaf nodes, while the distortion quantizer**1124**can determine an associated change in distortion resulting from turning this leaf node into a non-leaf node. Of course, in the case of each leaf node, this will also be an iterative process as different color allocations between the new leaf nodes are tested. Once the associated changes in distortion and entropy rate have been determined for each of the leaf nodes**1202**and**1204**, then, in the case where the trade-off between the change in distortion and entropy rate is fixed, the better leaf node to divide can be determined. In the example shown inFIG. 12 *c,*leaf node**1202**is divided into two non-leaf nodes**1206**and**1208**. At the point in the process of node division represented byFIG. 12 *c,*three leaf nodes remain: leaf nodes**1204**,**1206**and**1208**. - [0138]As described above, of course, the Lagrangian multiplier may not be fixed. In that case, a Lagrangian multiplier can be determined for each of the leaf nodes
**1202**and**1204**ofFIG. 12 *b.*Then, the leaf node that has a larger achievable Lagrangian multiplier can be divided. - [0139]The leaf node division or tree growth in the transition between
FIG. 12 *c*andFIG. 12 *d*is quite similar to that betweenFIGS. 12 *b*and**12***c.*In this latter transition betweenFIGS. 12 *c*and**12***d,*the leaf nodes under consideration for division are leaf nodes**1204**,**1206**, and**1208**. InFIG. 12 *d,*leaf node**1206**has been divided to create two new leaf nodes**1210**and**1212**. As a result, four leaf nodes**1204**,**1208**,**1210**and**1212**now exist. This process of leaf node division can continue until some target point is reached. This target point may be based on the total entropy rate or, alternatively, may be based on the total number of leaf nodes resulting from the last division. For example, each time a leaf node is divided, there is an associated increased entropy rate. At some point, the total entropy rate associated with the binary tree may exceed a target entropy rate. Also, each time a leaf node is divided, the total number of leaf nodes is increased by one. At some point, the new total number of leaf nodes in the tree structure may exceed some target number of leaf nodes. This target number of leaf nodes may be M (the number of colors in the original digitized color image) such that the original digitized color image can be completely reproduced for color-wise splitting algorithm (i.e., Algorithms**1**,**2**and**4**). Alternatively, this target number of leaf nodes may be less than M. In some embodiments, the process of tree growth or leaf node division will cease when either the associated total entropy rate exceeds the target entropy rate or the total number of leaf nodes exceeds the target number of leaf nodes. - [0140]In accordance with an aspect of an embodiment of the invention, a mobile wireless communication device can be used to provide entropy-constrained color splitting as described above. The embodiments described below generally relate to such a mobile wireless communication device, hereafter referred to as a mobile device, which can be configured according to an IT policy. It should be noted that the term IT policy, in general, refers to a collection of IT policy rules, in which the IT policy rules can be defined as being either grouped or non-grouped and global or per-user. The terms grouped, non-grouped, global and per-user are defined further below. Examples of applicable communication devices include pagers, cellular phones, cellular smart-phones, wireless organizers, personal digital assistants, computers, laptops, handheld wireless communication devices, wirelessly enabled notebook computers and the like.
- [0141]The mobile device is a two-way communication device with advanced data communication capabilities including the capability to communicate with other mobile devices or computer systems through a network of transceiver stations. The mobile device may also have the capability to allow voice communication. Depending on the functionality provided by the mobile device, it may be referred to as a data messaging device, a two-way pager, a cellular telephone with data messaging capabilities, a wireless Internet appliance, or a data communication device (with or without telephony capabilities). To aid the reader in understanding the structure of the mobile device and how it communicates with other devices and host systems, reference will now be made to
FIGS. 13 and 14 . - [0142]Referring first to
FIG. 13 , shown therein is a block diagram of an exemplary embodiment of a mobile device**1300**. The mobile device**1300**includes a number of components such as a main processor**1302**that controls the overall operation of the mobile device**1300**. Communication functions, including data and voice communications, are performed through a communication subsystem**1304**. The communication subsystem**1304**receives messages from and sends messages to a wireless network**1400**. In this exemplary embodiment of the mobile device**1300**, the communication subsystem**1304**is configured in accordance with the Global System for Mobile Communication (GSM) and General Packet Radio Services (GPRS) standards. The GSM/GPRS wireless network is used worldwide and it is expected that these standards will be superseded eventually by Enhanced Data GSM Environment (EDGE) and Universal Mobile Telecommunications Service (UMTS). New standards are still being defined, but it is believed that they will have similarities to the network behavior described herein, and it will also be understood by persons skilled in the art that the embodiments described herein are intended to use any other suitable standards that are developed in the future. The wireless link connecting the communication subsystem**1304**with the wireless network**1400**represents one or more different Radio Frequency (RF) channels, operating according to defined protocols specified for GSM/GPRS communications. With newer network protocols, these channels are capable of supporting both circuit switched voice communications and packet switched data communications. - [0143]Although the wireless network
**1400**associated with mobile device**1300**is a GSM/GPRS wireless network in one exemplary implementation, other wireless networks may also be associated with the mobile device**1300**in variant implementations. The different types of wireless networks that may be employed include, for example, data-centric wireless networks, voice-centric wireless networks, and dual-mode networks that can support both voice and data communications over the same physical base stations. Combined dual-mode networks include, but are not limited to, Code Division Multiple Access (CDMA) or CDMA2000 networks, GSM/GPRS networks (as mentioned above), and future third-generation (3G) networks like EDGE and UMTS. Some other examples of data-centric networks include WiFi 802.11, Mobitex™ and DataTAC™ network communication systems. Examples of other voice-centric data networks include Personal Communication Systems (PCS) networks like GSM and Time Division Multiple Access (TDMA) systems. - [0144]The main processor
**1302**also interacts with additional subsystems such as a Random Access Memory (RAM)**1306**, a flash memory**1308**, a display**1310**, an auxiliary input/output (I/O) subsystem**1312**, a data port**1314**, a keyboard**1316**, a speaker**1318**, a microphone**1320**, short-range communications**1322**and other device subsystems**1324**. - [0145]Some of the subsystems of the mobile device
**1300**perform communication-related functions, whereas other subsystems may provide “resident” or on-device functions. By way of example, the display**1310**and the keyboard**1316**may be used for both communication-related functions, such as entering a text message for transmission over the network**1400**, and device-resident functions such as a calculator or task list. - [0146]The mobile device
**1300**can send and receive communication signals over the wireless network**1400**after required network registration or activation procedures have been completed. Network access is associated with a subscriber or user of the mobile device**1300**. To identify a subscriber, the mobile device**1300**requires a SIM/RUIM card**1326**(i.e. Subscriber Identity Module or a Removable User Identity Module) to be inserted into a SIM/RUIM interface**1328**in order to communicate with a network. The SIM card or RUIM**1326**is one type of a conventional “smart card” that can be used to identify a subscriber of the mobile device**1300**and to personalize the mobile device**1300**, among other things. Without the SIM card**1326**, the mobile device**1300**is not fully operational for communication with the wireless network**1400**. By inserting the SIM card/RUIM**1326**into the SIM/RUIM interface**1328**, a subscriber can access all subscribed services. Services may include: web browsing and messaging such as e-mail, voice mail, Short Message Service (SMS), and Multimedia Messaging Services (MMS). More advanced services may include: point of sale, field service and sales force automation. The SIM card/RUIM**1326**includes a processor and memory for storing information. Once the SIM card/RUIM**1326**is inserted into the SIM/RUIM interface**1328**, it is coupled to the main processor**1302**. In order to identify the subscriber, the SIM card/RUIM**1326**can include some user parameters such as an International Mobile Subscriber Identity (IMSI). An advantage of using the SIM card/RUIM**1326**is that a subscriber is not necessarily bound by any single physical mobile device. The SIM card/RUIM**1326**may store additional subscriber information for a mobile device as well, including datebook (or calendar) information and recent call information. Alternatively, user identification information can also be programmed into the flash memory**1308**. - [0147]The mobile device
**1300**is a battery-powered device and includes a battery interface**1332**for receiving one or more rechargeable batteries**1330**. In at least some embodiments, the battery**1330**can be a smart battery with an embedded microprocessor. The battery interface**1332**is coupled to a regulator (not shown), which assists the battery**1330**in providing power V+ to the mobile device**1300**. Although current technology makes use of a battery, future technologies such as micro fuel cells may provide the power to the mobile device**1300**. - [0148]The mobile device
**1300**also includes an operating system**1334**and software components**1336**to**1346**which are described in more detail below. The operating system**1334**and the software components**1336**to**1346**that are executed by the main processor**1302**are typically stored in a persistent store such as the flash memory**1308**, which may alternatively be a read-only memory (ROM) or similar storage element (not shown). Those skilled in the art will appreciate that portions of the operating system**1334**and the software components**1336**to**1346**, such as specific device applications, or parts thereof, may be temporarily loaded into a volatile store such as the RAM**1306**. Other software components can also be included, as is well known to those skilled in the art. - [0149]The subset of software applications
**1336**that control basic device operations, including data and voice communication applications, will normally be installed on the mobile device**1300**during its manufacture. Other software applications include a message application**1338**that can be any suitable software program that allows a user of the mobile device**1300**to send and receive electronic messages. Various alternatives exist for the message application**1338**as is well known to those skilled in the art. Messages that have been sent or received by the user are typically stored in the flash memory**1308**of the mobile device**1300**or some other suitable storage element in the mobile device**1300**. In at least some embodiments, some of the sent and received messages may be stored remotely from the device**1300**such as in a data store of an associated host system that the mobile device**1300**communicates with. - [0150]The software applications can further include a device state module
**1340**, a Personal Information Manager (PIM)**1342**, and other suitable modules (not shown). The device state module**1340**provides persistence, i.e. the device state module**1340**ensures that important device data is stored in persistent memory, such as the flash memory**1308**, so that the data is not lost when the mobile device**1300**is turned off or loses power. - [0151]The PIM
**1342**includes functionality for organizing and managing data items of interest to the user, such as, but not limited to, e-mail, contacts, calendar events, voice mails, appointments, and task items. A PIM application has the ability to send and receive data items via the wireless network**1400**. PIM data items may be seamlessly integrated, synchronized, and updated via the wireless network**1400**with the mobile device subscriber's corresponding data items stored and/or associated with a host computer system. This functionality creates a mirrored host computer on the mobile device**1300**with respect to such items. This can be particularly advantageous when the host computer system is the mobile device subscriber's office computer system. - [0152]The mobile device
**1300**also includes a connect module**1344**, and an IT policy module**1346**. The connect module**1344**implements the communication protocols that are required for the mobile device**1300**to communicate with the wireless infrastructure and any host system, such as an enterprise system, that the mobile device**1300**is authorized to interface with. - [0153]The connect module
**1344**includes a set of APIs that can be integrated with the mobile device**1300**to allow the mobile device**1300**to use any number of services associated with the enterprise system. The connect module**1344**allows the mobile device**1300**to establish an end-to-end secure, authenticated communication pipe with the host system. A subset of applications for which access is provided by the connect module**1344**can be used to pass IT policy commands from the host system to the mobile device**1300**. This can be done in a wireless or wired manner. These instructions can then be passed to the IT policy module**1346**to modify the configuration of the device**1300**. Alternatively, in some cases, the IT policy update can also be done over a wired connection. - [0154]The IT policy module
**1346**receives IT policy data that encodes the IT policy. The IT policy module**1346**then ensures that the IT policy data is authenticated by the mobile device**1300**. The IT policy data can then be stored in the flash memory**1306**in its native form. After the IT policy data is stored, a global notification can be sent by the IT policy module**1346**to all of the applications residing on the mobile device**1300**. Applications for which the IT policy may be applicable then respond by reading the IT policy data to look for IT policy rules that are applicable. - [0155]The IT policy module
**1346**can include a parser (not shown), which can be used by the applications to read the IT policy rules. In some cases, another module or application can provide the parser. Grouped IT policy rules, described in more detail below, are retrieved as byte streams, which are then sent (recursively, in a sense) into the parser to determine the values of each IT policy rule defined within the grouped IT policy rule. In at least some embodiments, the IT policy module**1346**can determine which applications are affected by the IT policy data and send a notification to only those applications. In either of these cases, for applications that aren't running at the time of the notification, the applications can call the parser or the IT policy module**1346**when they are executed to determine if there are any relevant IT policy rules in the newly received IT policy data. - [0156]All applications that support rules in the IT Policy are coded to know the type of data to expect. For example, the value that is set for the “WEP User Name” IT policy rule is known to be a string; therefore the value in the IT policy data that corresponds to this rule is interpreted as a string. As another example, the setting for the “Set Maximum Password Attempts” IT policy rule is known to be an integer, and therefore the value in the IT policy data that corresponds to this rule is interpreted as such.
- [0157]After the IT policy rules have been applied to the applicable applications or configuration files, the IT policy module
**1346**sends an acknowledgement back to the host system to indicate that the IT policy data was received and successfully applied. - [0158]Other types of software applications can also be installed on the mobile device
**1300**. These software applications can be third party applications, which are added after the manufacture of the mobile device**1300**. Examples of third party applications include games, calculators, utilities, etc. - [0159]The additional applications can be loaded onto the mobile device
**1300**through at least one of the wireless network**1400**, the auxiliary I/O subsystem**1312**, the data port**1314**, the short-range communications subsystem**1322**, or any other suitable device subsystem**1324**. This flexibility in application installation increases the functionality of the mobile device**1300**and may provide enhanced on-device functions, communication-related functions, or both. For example, secure communication applications may enable electronic commerce functions and other such financial transactions to be performed using the mobile device**1300**. - [0160]The data port
**1314**enables a subscriber to set preferences through an external device or software application and extends the capabilities of the mobile device**1300**by providing for information or software downloads to the mobile device**1300**other than through a wireless communication network. The alternate download path may, for example, be used to load an encryption key onto the mobile device**1300**through a direct and thus reliable and trusted connection to provide secure device communication. - [0161]The data port
**1314**can be any suitable port that enables data communication between the mobile device**1300**and another computing device. The data port**1314**can be a serial or a parallel port. In some instances, the data port**1314**can be a USB port that includes data lines for data transfer and a supply line that can provide a charging current to charge the battery**1330**of the mobile device**1300**. - [0162]The short-range communications subsystem
**1322**provides for communication between the mobile device**1300**and different systems or devices, without the use of the wireless network**1400**. For example, the subsystem**1322**may include an infrared device and associated circuits and components for short-range communication. Examples of short-range communication standards include standards developed by the Infrared Data Association (IrDA), Bluetooth, and the 802.11 family of standards developed by IEEE. - [0163]In use, a received signal such as a text message, an e-mail message, or web page download will be processed by the communication subsystem
**1304**and input to the main processor**1302**. The main processor**1302**will then process the received signal for output to the display**1310**or alternatively to the auxiliary I/O subsystem**1312**. A subscriber may also compose data items, such as e-mail messages, for example, using the keyboard**1316**in conjunction with the display**1310**and possibly the auxiliary I/O subsystem**1312**. The auxiliary subsystem**1312**may include devices such as: a touch screen, mouse, track ball, infrared fingerprint detector, or a roller wheel with dynamic button pressing capability. The keyboard**1316**is preferably an alphanumeric keyboard and/or telephone-type keypad. However, other types of keyboards may also be used. A composed item may be transmitted over the wireless network**1400**through the communication subsystem**1304**. - [0164]For voice communications, the overall operation of the mobile device
**1300**is substantially similar, except that the received signals are output to the speaker**1318**, and signals for transmission are generated by the microphone**1320**. Alternative voice or audio I/O subsystems, such as a voice message recording subsystem, can also be implemented on the mobile device**1300**. Although voice or audio signal output is accomplished primarily through the speaker**1318**, the display**1310**can also be used to provide additional information such as the identity of a calling party, duration of a voice call, or other voice call related information. - [0165]Referring now to
FIG. 14 , an exemplary block diagram of the communication subsystem component**1304**is shown. The communication subsystem**1304**includes a receiver**1350**, a transmitter**1352**, as well as associated components such as one or more embedded or internal antenna elements**1354**and**1356**, Local Oscillators (LOs)**1358**, and a processing module such as a Digital Signal Processor (DSP)**1360**. The particular design of the communication subsystem**1304**is dependent upon the communication network**1400**with which the mobile device**1300**is intended to operate. Thus, it should be understood that the design illustrated inFIG. 14 serves only as one example. - [0166]Signals received by the antenna
**1354**through the wireless network**1400**are input to the receiver**1350**, which may perform such common receiver functions as signal amplification, frequency down conversion, filtering, channel selection, and analog-to-digital (A/D) conversion. A/D conversion of a received signal allows more complex communication functions such as demodulation and decoding to be performed in the DSP**1360**. In a similar manner, signals to be transmitted are processed, including modulation and encoding, by the DSP**1360**. These DSP-processed signals are input to the transmitter**1352**for digital-to-analog (D/A) conversion, frequency up conversion, filtering, amplification and transmission over the wireless network**1400**via the antenna**1356**. The DSP**1360**not only processes communication signals, but also provides for receiver and transmitter control. For example, the gains applied to communication signals in the receiver**1350**and the transmitter**1352**may be adaptively controlled through automatic gain control algorithms implemented in the DSP**1360**. - [0167]The wireless link between the mobile device
**1300**and the wireless network**1400**can contain one or more different channels, typically different RF channels, and associated protocols used between the mobile device**1300**and the wireless network**1400**. An RF channel is a limited resource that must be conserved, typically due to limits in overall bandwidth and limited battery power of the mobile device**1300**. - [0168]When the mobile device
**1300**is fully operational, the transmitter**1352**is typically keyed or turned on only when it is transmitting to the wireless network**1400**and is otherwise turned off to conserve resources. Similarly, the receiver**1350**is periodically turned off to conserve power until it is needed to receive signals or information (if at all) during designated time periods. - [0169]A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by any one of the patent document or patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
- [0170]Other variations and modifications of the invention are possible. For example, it will be appreciated by those of skill in the art that aspects of embodiments of the present invention can also be embodied in instructions stored on a suitable recording medium, which instructions could subsequently be used to configure a computer system to implement an embodiment of the present invention. All such modifications or variations are believed to be within the sphere and scope of the invention as defined by the claims appended hereto.

Patent Citations

Cited Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|

US5047842 * | Nov 3, 1989 | Sep 10, 1991 | The Trustees Of Princeton University | Color image display with a limited palette size |

US6496198 * | May 4, 1999 | Dec 17, 2002 | Canon Kabushiki Kaisha | Color editing system |

US20040170332 * | Jan 26, 2004 | Sep 2, 2004 | Canon Kabushiki Kaisha | Image coding method and apparatus for localised decoding at multiple resolutions |

US20060188024 * | Jan 6, 2006 | Aug 24, 2006 | Yoshinori Suzuki | Motion image encoding apparatus, motion image decoding apparatus, motion image encoding method, motion image decoding method, motion image encoding program, and motion image decoding program |

Referenced by

Citing Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|

US8775662 | Sep 27, 2006 | Jul 8, 2014 | Blackberry Limited | System and method for progressive delivery of multimedia objects |

US8830258 * | Mar 7, 2011 | Sep 9, 2014 | Ricoh Co., Ltd | Generating strokes in real-time on an electronic paper display |

US20070294333 * | Sep 27, 2006 | Dec 20, 2007 | Slipstream Data Inc, | System and method for progressive delivery of multimedia objects |

US20090010533 * | Oct 12, 2007 | Jan 8, 2009 | Mediatek Inc. | Method and apparatus for displaying an encoded image |

US20100121589 * | Sep 30, 2009 | May 13, 2010 | Samsung Electronics Co. Ltd. | Apparatus and method for implementing application using solar battery charging of portable terminal |

US20120229485 * | Sep 13, 2012 | Ricoh Co., Ltd. | Generating strokes in real-time on an electronic paper display |

Classifications

U.S. Classification | 382/240, 382/166, 375/E07.185, 375/E07.201, 375/E07.128, 375/E07.153, 375/E07.146 |

International Classification | G06K9/46, G06K9/00 |

Cooperative Classification | H04N19/182, H04N1/64, H04N19/96, H04N19/91, H04N19/103, H04N19/147, H04N19/94, H04N19/186, H04N19/19 |

European Classification | H04N7/26A8E, H04N7/28, H04N7/26A4V, H04N1/64, H04N7/26A6C8, H04N7/26A10L, H04N7/26Z10, H04N7/26A4C, H04N7/26A6D, H04N7/26A8U |

Legal Events

Date | Code | Event | Description |
---|---|---|---|

Dec 18, 2006 | AS | Assignment | Owner name: SLIPSTREAM DATA INC., CANADA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YANG, EN-HUI;WANG, LONGJI;REEL/FRAME:018647/0273 Effective date: 20061204 |

Sep 8, 2014 | FPAY | Fee payment | Year of fee payment: 4 |

Rotate