Publication number | USRE40081 E1 |
Publication type | Grant |
Application number | US 10/629,303 |
Publication date | Feb 19, 2008 |
Filing date | Jul 29, 2003 |
Priority date | Dec 16, 1998 |
Fee status | Lapsed |
Also published as | US6421464 |
Publication number | 10629303, 629303, US RE40081 E1, US RE40081E1, US-E1-RE40081, USRE40081 E1, USRE40081E1 |
Inventors | Trac D. Tran, Pankaj Topiwala |
Original Assignee | Fast Vdo Llc |
Export Citation | BiBTeX, EndNote, RefMan |
Patent Citations (24), Non-Patent Citations (4), Referenced by (9), Classifications (15), Legal Events (4) | |
External Links: USPTO, USPTO Assignment, Espacenet | |
More than one reissue application has been filed for the reissue of U.S. Pat. No. 6,421,464. The Instant Reissue application Ser. No. 10/629,303 filed Jul. 29, 2003 and U.S. application Ser. No. 11/896,522, filed Sep. 4, 2007 which is a Continuation of U.S. Reissue application Ser. No. 10/629,303.
The current invention relates to the processing of images such as photographs, drawings, and other two dimensional displays. It further relates to the processing of such images which are captured in digital format or after they have been converted to or expressed in digital format. This invention further relates to use of novel coding methods to increase the speed and compression ratio for digital image storage and transmission while avoiding introduction of undesirable artifacts into the reconstructed images.
In general, image processing is the analysis and manipulation of two-dimensional representations, which can comprise photographs, drawings, paintings, blueprints, x-rays of medical patients, or indeed abstract art or artistic patterns. These images are all two-dimensional arrays of information. Until fairly recently, images have comprised almost exclusively analog displays of analog information, for example, conventional photographs and motion pictures. Even the signals encoding television pictures, notwithstanding that the vertical scan comprises a finite number of lines, are fundamentally analog in nature.
Beginning in the early 1960's, images began to be captured or converted and stored as two-dimensional digital data, and digital image processing followed. At first, images were recorded or transmitted in analog form and then converted to digital representation for manipulation on a computer. Currently digital capture and transmission are on their way to dominance, in part because of the advent of charge coupled device (CCD) image recording arrays and in part because of the availability of inexpensive high speed computers to store and manipulate images.
An important task of image processing is the correction or enhancement of a particular image. For example, digital enhancement of images of celestial objects taken by space probes has provided substantial scientific information. However, the current invention relates primarily to compression for transmission or storage of digital images and not to enhancement.
One of the problems with digital images is that a complete single image frame can require up to several megabytes of storage space or transmission bandwidth. That is, one of today's 3½ inch floppy discs can hold at best a little more than one gray-scale frame and sometimes substantially less than one whole frame. A full-page color picture, for example, uncompressed, can occupy 30 megabytes of storage space. Storing or transmitting the vast amounts of data which would be required for real-time uncompressed high resolution digital video is technologically daunting and virtually impossible for many important communication channels, such as the telephone line. The transmission of digital images from space probes can take many hours or even days if insufficiently compressed images are involved. Accordingly, there has been a decades long effort to develop methods of extracting from images the information essential to an aesthetically pleasing or scientifically useful picture without degrading the image quality too much and especially without introducing unsightly or confusing artifacts into the image.
The basic approach has usually involved some form of coding of picture intensities coupled with quantization. One approach is block coding; another approach, mathematically equivalent with proper phasing, is multiphase filter banks. Frequency based multi-band transforms have long found application in image coding. For instance, the JPEG image compression standard, W. B. Pennebaker and J. L. Mitchell, “JPEG: Still Image Compression Standard,” Van Nostrand Reinhold, 1993, employs the 8×8 discrete cosine transform (DCT) at its transformation stage. At high bit rates, JPEG offers almost lossless reconstructed image quality. However, when more compression is needed, annoying blocking artifacts appear since the DCT bases are short and do not overlap, creating discontinuities at block boundaries.
The wavelet transform, on the other hand, with long, varying-length, and overlapping bases, has elegantly solved the blocking problem. However, the transform's computational complexity can be significantly higher than that of the DCT. This complexity gap is partly in terms of the number of arithmetical operations involved, but more importantly, in terms of the memory buffer space required. In particular, some implementations of the wavelet transform require many more operations per output coefficient as well as a large buffer.
An interesting alternative to wavelets is the lapped transform, e.g., H. S. Malvar, Signal Processing with Lapped Transforms, Artech House, 1992, where pixels from adjacent blocks are utilized in the calculation of transform coefficients for the working block. The lapped transforms outperform the DCT on two counts: (i) from the analysis viewpoint, they take into account inter-block correlation and hence provide better energy compaction; (ii) from the synthesis viewpoint, their overlapping basis functions decay asymptotically to zero at the ends, reducing blocking discontinuities dramatically.
Nevertheless, lapped transforms have not yet been able to supplant the unadorned DCT in international standard coding routines. The principal reason is that the modest improvement in coding performance available up to now has not been sufficient to justify the significant increase in computational complexity. In the prior art, therefore, lapped transforms remained too computationally complex for the benefits they provided. In particular, the previous lapped transformed somewhat reduced but did not eliminate the annoying blocking artifacts.
It is therefore an object of the current invention to provide a new transform which is simple and fast enough to replace the bare DCT in international standards, in particular in JPEG and MPEG-like coding standards. It is another object of this invention to provide an image transform which has overlapping basis functions so as to avoid blocking artifacts. It is a further object of this invention to provide a lapped transform which is approximately as fast as, but more efficient for compression than, the bare DCT. It is yet another object of this invention to provide dramatically improved speed and efficiency using a lapped transform with lifting steps in a butterfly structure with dyadic-rational coefficients. It is yet a further object of this invention to provide a transform structure such that for a negligible complexity surplus over the bare DCT a dramatic coding performance gain can be obtained both from a subjective and objective point of view while blocking artifacts are completely eliminated.
In the current invention, we use a family of lapped biorthogonal transforms implementing a small number of dyadic-rational lifting steps. The resulting transform, called the LiftLT, not only has high computation speed but is well-suited to implementation via VLSI.
Moreover, it also consistently outperforms state-of-the-art wavelet based coding systems in coding performance when the same quantizer and entropy coder are used. The LiftLT is a lapped biorthogonal transform using lifting steps in a modular lattice structure, the result of which is a fast, efficient, and robust encoding system. With only 1 more multiplication (which can also be implemented with shift-and-add operations), 22 more additions, and 4 more delay elements compared to the bare DCT, the LiftLT offers a fast, low-cost approach capable of straightforward VLSI implementation while providing reconstructed images which are high in quality, both objectively and subjectively. Despite its simplicity, the LiftLT provides a significant improvement in reconstructed image quality over the traditional DCT in that blocking is completely eliminated while at medium and high compression ratios ringing artifacts are reasonably contained. The performance of the LiftLT surpasses even that of the well-known 9/7-tap biorthogonal wavelet transform with irrational coefficients. The LiftLT's block-based structure also provides several other advantages: supporting parallel processing mode, facilitating region-of-interest coding and decoding, and processing large images under severe memory constraints.
Most generally, the current invention is an apparatus for block coding of windows of digitally represented images comprising a chain of lattices of lapped transforms with dyadic rational lifting steps. More particularly, this invention is a system of electronic devices which codes, stores or transmits, and decodes M×M sized blocks of digitally represented images, where M is an even number. The main block transform structure comprises a transform having M channels numbered 0 through M−1, half of said channel numbers being odd and half being even; a normalizer with a dyadic rational normalization factor in each of said M channels; two lifting steps with a first set of identical dyadic rational coefficients connecting each pair of adjacent numbered channels in a butterfly configuration, M/2 delay lines in the odd numbered channels; two inverse lifting steps with the first set of dyadic rational coefficients connecting each pair of adjacent numbered channels in a butterfly configuration; and two lifting steps with a second set of identical dyadic rational coefficients connecting each pair of adjacent odd numbered channels; means for transmission or storage of the transform output coefficients; and an inverse transform comprising M channels numbered 0 through M−1, half of said channel numbers being odd and half being even; two inverse lifting steps with dyadic rational coefficients connecting each pair of adjacent odd numbered channels; two lifting steps with dyadic rational coefficients connecting each pair of adjacent numbered channels in a butterfly configuration; M/2 delay lines in the even numbered channels; two inverse lifting steps with dyadic rational coefficients connecting each pair of adjacent numbered channels in a butterfly configuration; a denormalizer with a dyadic rational inverse normalization factor in each of said M channels; and a base inverse transform having M channels numbered 0 through M−1.
Typically, a block transform for image processing is applied to a block (or window) of, for example, 8×8 group of pixels and the process is iterated over the entire image. A biorthogonal transform in a block coder uses as a decomposition basis a complete set of basis vectors, similar to an orthogonal basis. However, the basis vectors are more general in that they may not be orthogonal to all other basis vectors. The restriction is that there is a “dual” basis to the original biorthogonal basis such that every vector in the original basis has a “dual” vector in the dual basis to which it is orthogonal. The basic idea of combining the concepts of biorthogonality and lapped transforms has already appeared in the prior art. The most general lattice for M-channel linear phase lapped biorthogonal transforms is presented in T. D. Tran, R. de Queiroz, and T. Q. Nguyen, “The generalized lapped biorthogonal transform,” ICASSP, pp. 1441-1444, Seattle, May 1998, and in T. D. Tran, R. L. de Queiroz, and T. Q. Nguyen, “Linear phase perfect reconstruction filter bank: lattice structure, design, and application in image coding” (submitted to EEE Trans. on Signal Processing, April 1998). A signal processing flow diagram of this well-known generalized filter bank is shown in FIG. 2.
In the current invention, which we call the Fast LiftLT, we apply lapped transforms based on using fast lifting steps in an M-channel uniform linear-phase perfect reconstruction filter bank, according to the generic polyphase representation of FIG. 1. In the lapped biorthogonal approach, the polyphase matrix E(z) can be factorized as
In these equations, I is the identity matrix, and J is the matrix with 1's on the anti-diagonal.
The transform decomposition expressed by equations (1) through (3) is readily represented, as shown in
U_{i}=U_{i0}Γ_{i}U_{i1}, V_{i}=V_{i0}Δ_{i}V_{i1}
where U_{i0}, U_{i1}, V_{i0}, V_{i1 }are diagonalizing orthogonal matrices and Γ_{i}, Δ_{i }are diagonal matrices with positive elements.
It is well known that any M/2×M/2 orthogonal matrix can be factorized into M(M−2)/8 plane rotations θ_{i }and that the diagonal matrices represent simply scaling factors α_{i}. Accordingly, the most general LT lattice consists of KM(M−2)/2 two dimensional rotations and 2M diagonal scaling factors α_{i}. Any invertible matrix can be expressed as a sequence of pairwise plane rotations θ and scaling factors α_{i }as shown in FIG. 3.
It is also well known that a plane rotation can be performed by 3 “shears”:
This can be easily verified by computation
Each of the factors above is capable of a “lifting” step in signal processing terminology. The product of two which effects a linear transform of pairs of coefficients:
The signal processing flow diagram of this operation is shown in FIG. 4. The crossing arrangement of these flow paths is also referred to as a butterfly configuration. Each of the above “shears” can be written as a lifting step.
Combining the foregoing, the shears referred to can be expressed as computationally equivalent “lifting steps” in signal processing. In other words, we can replace each “rotation” by 3 closely-related lifting steps with butterfly structure. It is possible therefore to implement the complete LT lattice shown in
In the simplest but currently preferred embodiment, to minimize the complexity of the transform we choose a small overlapping factor K=2 and set the initial stage E_{0 }to be the DCT itself Many other coding transforms can serve for the base stage instead of the DCT, and it should be recognized that many other embodiments are possible and can be implemented by one skilled in the art of signal processing.
Following the observation in H. S. Malvar, “Lapped biorthogonal transforms for transform coding with reduced blocking and ringing artifacts,” ICASSP97, Munich, April 1997, we apply a scaling factor to the first DCT's antisymmetric basis to generate synthesis LT basis functions whose end values decay smoothly to exact zero—a crucial advantage in blocking artifacts elimination. However, instead of scaling the analysis by √{square root over (2)} and the synthesis by 1/√{square root over (2)}, we opt for 25/16 and its inverse 16/25 since they allow the implementation of both analysis and synthesis banks in integer arithmetic. Another value that works almost as well as 25/16 is 5/4. To summarize, the following choices are made in the first stage: the combination of U_{00 }and V_{00 }with the previous butterfly form the DCT;
and Γ_{o}=U_{00}=V_{00}=I_{M/2}. See FIG. 2.
After 2 series of ±1 butterflies W and the delay chain Λ(z), the LT symmetric basis functions already have good attenuation, especially at DC (ω=0). Hence, we can comfortably set U_{1}=I_{M/2}.
As noted, V_{1 }is factorizable into a series of lifting steps and diagonal scalings. However, there are several problems: (i) the large number of lifting steps is costly in both speed and physical real-estate in VLSI implementation; (ii) the lifting steps are related; (iii) and it is not immediately obvious what choices of rotation angles will result in dyadic rational lifting multipliers. In the current invention, we approximate V_{1 }by (M/2)−1 combinations of block-diagonal predict-and-update lifting steps, i.e.,
Here, the free parameters u_{i }and p_{i }can be chosen arbitrarily and independently without affecting perfect reconstruction. The inverses are trivially obtained by switching the order and the sign of the lifting steps. Unlike popular lifting implementations of various wavelets, all of our lifting steps are of zero-order, namely operating in the same time epoch. In other words, we simply use a series of 2×2 upper or lower diagonal matrices to parameterize the invertible matrix V_{1}.
Most importantly, fast-computable VLSI-friendly transforms are readily available when u_{i }and p_{i }are restricted to dyadic rational values, that is, rational fractions having (preferably small) powers of 2 denominators. With such coefficients, transform operations can for the most part be reduced to a small number of shifts and adds. In particular, setting all of the approximating lifting step coefficients to −1/2 yields a very fast and elegant lapped transform. With this choice, each lifting step can be implemented using only one simple bit shift and one addition.
The resulting LiftLT lattice structures are presented in
The inverse or synthesis lattice is shown in FIG. 6. This system comprises a set of four fast dyadic lifting steps 11, a butterfly-structured set of lifting steps 12, a delay line 13 on four of the eight channels, 16/25 inverse normalization 14, and an inverse DCT block 15.
The LiftLT is sufficiently fast for many applications, especially in hardware, since most of the incrementally added computation comes from the 2 butterflies and the 6 shift-and-add lifting steps. It is faster than the type-I fast LOT described in H. S. Malvar, Signal Processing with Lapped Transforms, Artech House, 1992. Besides its low complexity, the LiftLT possesses many characteristics of a high-performance transform in image compression: (i) it has high energy compaction due to a high coding gain and a low attenuation near DC where most of the image energy is concentrated; (ii) its synthesis basis functions also decay smoothly to zero, resulting in blocking-free reconstructed images.
Comparisons of complexity and performance between the LiftLT and other popular transforms are tabulated in Table 1 and Table 2. The LiftLT's performance is already very close to that of the optimal generalized lapped biorthogonal transform, while its complexity is the lowest amongst the transforms except for the DCT.
To assess the new method in image coding, we compared images coded and decoded with four different transforms:
Table 1 contains a comparison of the complexity of these four coding systems, comparing numbers of operations needed per 8 transform coefficients:
No. | |||
Transform | Multiplications | No. Additions | No. Shifts |
8 × 8 DCT | 13 | 29 | 0 |
8 × 16 Type-I Fast LOT | 22 | 54 | 0 |
917 Wavelet, 1-level | 36 | 56 | 0 |
8 × 6 Fast LiftLT | 14 | 51 | 6 |
Table 2 sets forth a number of different performance measures for each of the four methods:
DC | Stopband | |||
Coding | Atten. | Atten. | Mir. Freq. | |
Transform | Gain (dB) | (−dB) | (−dB) | Atten. (−dB) |
8 × 8 DCT | 8.83 | 310.62 | 9.96 | 322.1 |
8 × 16 Type-I Fast LOT | 9.2 | 309.04 | 17.32 | 314.7 |
8 × 16 Optional LT | 9.62 | 327.4 | 13.5 | 55.54 |
8 × 16 Fast LiftLT | 9.54 | 312.56 | 13.21 | 304.85 |
Reconstructed images for a standard 512×512 “Barbara” test image at 1:32 compression ratio are shown in
Lena | Goldhill | Barbara | ||||||||||
Comp. | 9/7 WL | 8 × 8 | 8 × 16 | 8 × 16 | 9/7 WL | 8 × 8 | 8 × 16 | 8 × 16 | 9/7 WL | 8 × 8 | 8 × 16 | 8 × 16 |
Ratio | SPIHT | DCT | LOT | LiftLT | SPIHT | DCT | LOT | LiftLT | SPIHT | DCT | LOT | LiftLT |
8 | 40.41 | 39.91 | 40.02 | 40.21 | 36.55 | 36.25 | 36.56 | 36.56 | 36.41 | 36.31 | 37.22 | 37.57 |
16 | 37.21 | 36.38 | 36.69 | 37.11 | 33.13 | 32.76 | 33.12 | 33.22 | 31.4 | 31.11 | 32.52 | 32.82 |
32 | 34.11 | 32.9 | 33.49 | 34 | 30.56 | 30.07 | 30.52 | 30.63 | 27.58 | 27.28 | 28.71 | 28.93 |
64 | 31.1 | 29.67 | 30.43 | 30.9 | 28.48 | 27.93 | 28.34 | 28.54 | 24.86 | 24.58 | 25.66 | 25.93 |
100 | 29.35 | 27.8 | 28.59 | 29.03 | 27.38 | 26.65 | 27.08 | 27.28 | 23.76 | 23.42 | 24.32 | 24.5 |
128 | 28.38 | 26.91 | 27.6 | 28.12 | 26.73 | 26.01 | 26.46 | 26.7 | 23.35 | 22.68 | 23.36 | 23.47 |
The LiftLT outperforms its block transform relatives for all test images at all bit rates. Comparing to the wavelet transform, the LiftLT is quite competitive on smooth images—about 0.2 dB below on Lena. However, for more complex images such as Goldhill or Barbara, the LiftLT consistently surpasses the 9/7-tap wavelet. The PSNR improvement can reach as high as 1.5 dB.
Cited Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|
US5081645 * | Aug 6, 1990 | Jan 14, 1992 | Aware, Inc. | Novel spread spectrum codec apparatus and method |
US5339265 * | Aug 31, 1992 | Aug 16, 1994 | University Of Maryland At College Park | Optimal unified architectures for the real-time computation of time-recursive discrete sinusoidal transforms |
US5592569 * | May 11, 1995 | Jan 7, 1997 | Competitive Technologies, Inc. | Method for encoding and decoding images |
US5604824 * | Sep 22, 1994 | Feb 18, 1997 | Houston Advanced Research Center | Method and apparatus for compression and decompression of documents and the like using splines and spline-wavelets |
US5764698 * | Dec 30, 1993 | Jun 9, 1998 | International Business Machines Corporation | Method and apparatus for efficient compression of high quality digital audio |
US5805739 * | Apr 2, 1996 | Sep 8, 1998 | Picturetel Corporation | Lapped orthogonal vector quantization |
US5812219 * | Nov 1, 1995 | Sep 22, 1998 | U.S. Philips Corporation | System and method for transmitting picture or sound signals |
US5857036 * | Mar 4, 1996 | Jan 5, 1999 | Iterated Systems, Inc. | System and method for the fractal encoding of datastreams |
US5859788 * | Aug 15, 1997 | Jan 12, 1999 | The Aerospace Corporation | Modulated lapped transform method |
US5883981 * | Oct 18, 1996 | Mar 16, 1999 | Competitive Technologies Of Pa, Inc. | Lattice vector transform coding method for image and video compression |
US5898798 * | Oct 18, 1996 | Apr 27, 1999 | U.S. Philips Corporation | Region-based texture coding and decoding method and corresponding systems |
US5901251 * | Mar 18, 1997 | May 4, 1999 | Hewlett-Packard Company | Arithmetic coding compressor using a context model that is adaptive to variable length patterns in bi-level image data |
US5903669 * | Jan 25, 1996 | May 11, 1999 | Canon Kabushiki Kaisha | Image processing apparatus and method |
US5946038 * | Feb 24, 1997 | Aug 31, 1999 | U.S. Philips Corporation | Method and arrangement for coding and decoding signals |
US5960123 * | Jul 24, 1996 | Sep 28, 1999 | Fuji Photo Film Co., Ltd. | Method and apparatus for enhancing contrast in images by emphasis processing of a multiresolution frequency band |
US5973755 * | Apr 4, 1997 | Oct 26, 1999 | Microsoft Corporation | Video encoder and decoder using bilinear motion compensation and lapped orthogonal transforms |
US5995668 * | Oct 25, 1996 | Nov 30, 1999 | U.S. Philips Corporation | Segmented picture coding method and system, and corresponding decoding method and system |
US5999656 * | Jan 17, 1997 | Dec 7, 1999 | Ricoh Co., Ltd. | Overlapped reversible transforms for unified lossless/lossy compression |
US6018753 * | Jul 29, 1997 | Jan 25, 2000 | Lucent Technologies Inc. | Interpolating filter banks in arbitrary dimensions |
US6094631 * | Jul 9, 1998 | Jul 25, 2000 | Winbond Electronics Corp. | Method of signal compression |
US6104982 * | Nov 22, 1996 | Aug 15, 2000 | Schlumberger Technology Corporation | Compression method and apparatus for seismic data |
US6144771 * | Jun 27, 1997 | Nov 7, 2000 | Competitive Technologies Of Pa, Inc. | Method and apparatus for encoding and decoding images |
US6144773 * | Feb 27, 1996 | Nov 7, 2000 | Interval Research Corporation | Wavelet-based data compression |
US6198412 * | Jan 20, 1999 | Mar 6, 2001 | Lucent Technologies Inc. | Method and apparatus for reduced complexity entropy coding |
Reference | ||
---|---|---|
1 | * | Liang et al., "ITU-Telecommunications Standardization Sector", A 16-bit architecture for H.26L treating DCT Transforms and quantization, pp. 1-12, May 29, 2001. |
2 | * | Liang, et al., "ITU-Telecommunications Standardization Sector", A 16-bit architecture for H.26L treating DCT Transforms anf quantization, pp. 1-2, May 29, 2001. |
3 | * | Nayebi et al., "A time domain view of filter banks and wavelets", Signals, Systems and Computers, 1991. 1991 Conference Record of the Twenty-Fifth Asimolar Conference on, 1991, pp. 736-740 vol. 2. |
4 | * | Sweldens, Wim, "The Lifting Scheme: A custom design construction of biorthogonal wavelets", pp. 1-29, Nov. 1994. |
Citing Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|
US7689052 | Jun 9, 2006 | Mar 30, 2010 | Microsoft Corporation | Multimedia signal processing using fixed-point approximations of linear transforms |
US7773671 | Apr 12, 2005 | Aug 10, 2010 | Microsoft Corporation | Block transform and quantization for image and video coding |
US7839928 | Apr 12, 2005 | Nov 23, 2010 | Microsoft Corporation | Block transform and quantization for image and video coding |
US7881371 | Feb 25, 2005 | Feb 1, 2011 | Microsoft Corporation | Block transform and quantization for image and video coding |
US8942289 | Jun 29, 2007 | Jan 27, 2015 | Microsoft Corporation | Computational complexity and precision control in transform-based digital media codec |
US8971405 | Jan 19, 2011 | Mar 3, 2015 | Microsoft Technology Licensing, Llc | Block transform and quantization for image and video coding |
US20050141609 * | Feb 25, 2005 | Jun 30, 2005 | Microsoft Corporation | Block transform and quantization for image and video coding |
US20050180503 * | Apr 12, 2005 | Aug 18, 2005 | Microsoft Corporation | Block transform and quantization for image and video coding |
US20050213659 * | Apr 12, 2005 | Sep 29, 2005 | Microsoft Corporation | Block transform and quantization for image and video coding |
U.S. Classification | 382/232 |
International Classification | G06T9/00, G06K9/36, G06F17/14, H04N7/26 |
Cooperative Classification | H04N19/635, H04N19/42, H04N19/114, G06F17/147, H04N19/63 |
European Classification | H04N7/26L, H04N7/26H30D1, H04N7/26A4C6, G06F17/14M, H04N7/26H30D2 |
Date | Code | Event | Description |
---|---|---|---|
Aug 20, 2007 | AS | Assignment | Owner name: FASTVDO, LLC, MARYLAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PT-IP, LLC;REEL/FRAME:019747/0027 Effective date: 20070816 |
Nov 17, 2009 | FPAY | Fee payment | Year of fee payment: 8 |
Feb 21, 2014 | REMI | Maintenance fee reminder mailed | |
Jul 16, 2014 | LAPS | Lapse for failure to pay maintenance fees |