Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS7519635 B1
Publication typeGrant
Application numberUS 12/059,874
Publication dateApr 14, 2009
Filing dateMar 31, 2008
Priority dateMar 31, 2008
Fee statusLapsed
Publication number059874, 12059874, US 7519635 B1, US 7519635B1, US-B1-7519635, US7519635 B1, US7519635B1
InventorsNils Haustein, Craig A. Klein, Ulf Troppens, Daniel J. Winarski
Original AssigneeInternational Business Machines Corporation
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Method of and system for adaptive selection of a deduplication chunking technique
US 7519635 B1
Abstract
A method of adaptively selecting an optimum data deduplication chunking method receives a request to deduplicate a file, wherein the file has a file type. The method searches a table of file types, wherein the table includes, for each file type, a chunking method, a deduplication ratio, and a depulication ratio threshold. The method selects a chunking method for the file according to the table. The method chunks the file using the selected chunking method. The method deduplicates the chunked file according to prior art deduplication methods. The method calculates a deduplication ratio for the file type and updates the table with the calculated deduplication ratio for the file type. If the calculated deduplication ratio for the file type is less than the deduplication ratio threshold for the file type, the method selects a new chunking method for the file type and updates the table of file types with the new chunking method for the file type.
Images(4)
Previous page
Next page
Claims(1)
1. A method of selecting an optimum deduplication chunking method, which comprises:
receiving a request to deduplicate a file, said file having a file type;
searching a table of file types, said table including for each file type, a chunking method, a deduplication ratio, and a user configurable deduplication ratio threshold;
selecting a chunking method for said file according to said table;
chunking said file using said selected chunking method;
deduplicating said chunked file;
calculating a deduplication ratio for said file type by dividing a sum of individual deduplication ratios achieved for files of said file type by a number of files of said file type deduplicated;
updating said table with said calculated deduplication ratio for said file type;
if said calculated deduplication ratio for said file type is equal to or greater than said deduplication ratio for said file type, maintaining said chunking method for said file type in the said table;
if said calculated deduplication ratio for said file type is less than said deduplication ratio threshold for said file type, selecting a new chunking method for said file type; and,
updating said table with said new chunking method for said file type.
Description
BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates in general to the field of data deduplication, and more particularly to a method of and system for adaptively selecting an optimum deduplication chunking method for files of a particular type.

2. Description of the Related Art

A goal in data storage is to reduce the amount of space required to store the data. One method of reducing required storage space is data deduplication. In data deduplication, a data object, which may be a file, a data stream, or some other form of data, is broken down into one or more chunks using a chunking method. A hash is calculated for each chunk using any of several known hashing techniques. The hashes of all chunks are compared for duplicates. Duplicate hashes mean either the data chunks are identical or there has been a hash collision. A hash collision occurs when different chunks produce the same hash. To prevent hash collisions, other techniques such as bit-by-bit comparison may be performed. After the comparison of hashes and proof of their uniqueness, unique chunks are stored. Chunks that are duplicates of already stored chunks are not stored; rather, such chunks are referenced by pointers to the already stored chunks.

Data deduplication can yield storage space reductions of 20:1 or more. However, the deduplication ratio is highly dependent upon the method used to chunks the data. Several chunking techniques have been developed. Each chunking method is thought to be optimum for a set of file types. However, a particular chunking method may not in fact be optimum for a particular file type.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide methods of and systems adaptively selecting an optimum data deduplication chunking method. The method receives a request to deduplicate a file, wherein the file has a file type. The method searches a table of file types, wherein the table includes, for each file type, a chunking method, a deduplication ratio, and a depulication ratio threshold. The method selects a chunking method for the file according to the table. The method chunks the file using the selected chunking method. The method deduplicates the chunked file. The method calculates a deduplication ratio for the file type and updates the table with the calculated deduplication ratio for the file type. If the calculated deduplication ratio for the file type is less than the deduplication ratio threshold for the file type, the method selects a new chunking method for the file type and updates the table of file types with the new chunking method for the file type.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further purposes and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, where:

FIG. 1 is a block diagram of network file server based embodiment according to the present invention;

FIG. 2 is a block diagram of standalone client based embodiment according to the present invention.

FIG. 3 illustrates an embodiment of a table of file types according to the present invention.

FIG. 4 is a flow chart of an embodiment of a process for selecting an optimum chunking method according to the present invention; and,

FIG. 5 is a flow chart of an embodiment of a process for updating a table of file types according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now to drawings, and first to FIG. 1, an embodiment of a system according to the present invention is designated generally by the numeral 100. System 100 includes a plurality of client systems 101. Each client system 101 is coupled to a network 107. Network 107 may be a storage area network, or other suitable network. A file server 109 is coupled to network 107.

Each client system 101 includes at least one application 103, which may be a text processor, a database, or other repository managing images, documents, video streams, or any other kind of data. Client systems 101 also each include a file system client 105. The configuration of file system client 105 depends upon the configuration of the file system. For example, if the file system is a network file system then file system client 105 incorporates the appropriate interface of the network file system.

File server 109 includes a file system 111. File system 111 includes a deduplication component 113 according to the present invention. The operation of deduplication component 113 will be described in detail hereinafter. Data stored in file system 113 is placed on a storage system 115, which may be a disk storage system or the like.

Applications 103 create and manage data files that are stored in network attached file server 109 through respective file system clients 105. When an application 103 stores a file, the application passes the file to its respective file system client 105, which in turn transfers the file through network 107 to file server 109. Deduplication component 113 performs deduplication according to the present invention. The deduplicated file is stored in storage system 115.

Referring to FIG. 2, embodiments of the present invention may be included in a single server system 201. Server 201 includes an application 203, which may be any type of application described with reference to FIG. 1. Server 201 includes a file system 205 which includes a deduplication component 207 according to the present invention. Server 201 is coupled to a storage system 209, which may be any suitable storage system, such as a disk storage system. When application 203 stores the file, the application passes the file directly to file system 205. Deduplication component 207 deduplicates the file according to the present invention. Filesystem 205 stores the deduplicated file in storage system 209.

Embodiments of deduplication components according to the present invention include a table that maps file types to the most appropriate chunking methods. Referring to FIG. 3, there is illustrated an example of a table 301 of file types. Table 301 includes a file type column 303 that lists file types that may be deduplicated according to the present invention. The file types listed in column 303 are merely examples and do not comprise an exhaustive listing. An extension column 305 lists file name extensions associated with each file type. In embodiments of the present invention, a file's type may be determined by referring to its file name extension.

Table 301 includes a chunking method column 307 that lists the preferred chunking method associated with each file type in column 303. The chunking methods in column 307 include format aware or format based, fingerprinting, file days, and block-based. Format based chunking methods understand the data formats of the data file and breakup the data based upon its format. For example, a first PowerPoint™ file may comprise 30 pages and a second PowerPoint™ file may comprise the same 30 pages plus one additional page. Format based chunking breaks down the two PowerPoint files into one page chunks. Deduplication references the 30 pages of the second file to the 30 pages of the first file and stores the additional page separately.

Fingerprinting chunking methods look for logical breaks in the data and breaks up the data into chunks based upon logical breaks. The format of the data is not relevant. In file based chunking, the entire data file is kept as an entity and is not broken down. File based chunking is generally applied to files with fixed content such as PDF files. In block based chunking, the data file is broken down into fixed blocks.

Table 301 includes a deduplication ratio column 309. Deduplication ratio is a measure of the efficiency of a deduplication method. A deduplication ratio for a file is calculated by dividing the space that would required to store the original file by the space required to store the deduplicated file. As will be explained in detail hereinafter, as files are deduplicated according to an embodiment of the present invention, a deduplication ratio is calculated for each file. Then, an average deduplication ratio is calculated for the file type by dividing the sum of the individual deduplication ratios achieved for files of that type by the number of files of that type deduplicated. Then, the deduplication ratio for that file type is updated in column 309 with the calculated average deduplication ratio.

Table 301 includes a deduplication threshold column 311. A deduplication threshold is selected for each file type. As will be explained in detail hereinafter, if the average deduplication ratio in column 309 for a file type is less than the threshold list in column 311, a new chunking method is selected for the file type and column 307 is updated with the new chunking method. The deduplication threshold in column 311 is configurable by the user of system 100 or 200.

FIG. 4 is a flow chart of an embodiment for the adaptive selection of a deduplication chunking method implemented in deduplication component 113 of FIG.1 or 207 of FIG. 2 according to the present invention. The deduplication component receives a file, at step 401. The deduplication component determines the file type 305 of the received file by referring to its file name extension 305, at step 403, and using table 301. Then, the deduplication component selects from table 301 a chunking method 307 based on the file type 303, as indicated at block 405. The deduplication component breaks down or chunks the data, at block 407, using the selected chunking method. Then, the deduplication component deduplicates the broken down a data, at block 409, according prior art deduplication methods. The deduplication component calculates the deduplication ratio for the file and for the file type in the manner described above, at block 411. Then, the deduplication component updates the deduplication ratio for the file type in column 309 of table 301, at block 413.

FIG. 5 is a flow chart of an embodiment of a process for updating the chunking method according to the present invention which is implemented in the inventive deduplication component 113 of FIG.1 or 207 of FIG. 2. The deduplication component selects a first or next file type from table 301, at block 501. The deduplication component tests, at decision block 503 if the deduplication ratio in column 309 is greater than the threshold in column 311 of the file type. If not, the deduplication component tests, at decision block 505, if there are more file types. If not, processing ends. If, as determined at decision block 503, the deduplication ratio is greater than the threshold, a new chunking method is selected for the file type, at block 507, and column 307 of table 301 is updated with the new chunking method, as indicated at block 509. Processing according to FIG. 5 continues until all file types have been analyzed. FIG. 5 illustrates a periodic batch processing of table 301. Those skilled in the art will recognize that the chunking method may be analyzed and updated necessary after each deduplication of a file of a particular type.

From the foregoing, it will be apparent to those skilled in the art that systems and methods according to the present invention are well adapted to overcome the shortcomings of the prior art. While the present invention has been described with reference to presently preferred embodiments, those skilled in the art, given the benefit of the foregoing description, will recognize alternative embodiments. Accordingly, the foregoing description is intended for purposes of illustration and not of limitation.

Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US7092956 *Nov 2, 2001Aug 15, 2006General Electric Capital CorporationDeduplication system
US7103602 *Aug 31, 2001Sep 5, 2006Kroll Ontrack Inc.System and method for data management
US20080098083 *Oct 19, 2006Apr 24, 2008Oracle International CorporationSystem and method for data de-duplication
US20080098236 *Oct 19, 2006Apr 24, 2008Oracle International CorporationSystem and method for data encryption
US20080133561 *Dec 1, 2006Jun 5, 2008Nec Laboratories America, Inc.Methods and systems for quick and efficient data management and/or processing
US20080184001 *Jan 30, 2007Jul 31, 2008Network Appliance, Inc.Method and an apparatus to store data patterns
Non-Patent Citations
Reference
1 *Balachandran et al. (Sequence of Hashes Compression in Data De-duplication), Data Compression Conference, 2008, DCC, 2008 Mar. 25-27, 2008, pp. 505-505.
2 *Bilenko et al. (Adaptive duplicate detection using learnable string similarity measures, Conference on knowledge discovery in data, proceeding of the ninth ACM SIGKDD international conference on knowledge discover and data mining, 2003, 39-48.
3Bouffard, et al., "Sepaton S2100: Fast, Scalable, Space-efficient Disk-based Backup", ESG Lab Validation Report, Sep. 2007, pp. 1-17, Enterprise Strategy Group, Inc.
4 *Carvalho et al. (Learning to deduplicate), international conference on digital libraries, proceeding of the 6th ACM/IEEE-CS joint conference on digital libraries, 2006, pp. 41-50.
5 *Conrad et al. (Information retrieval 2: Essential deduplication functions for transactional databases in law, 2007, pp. 261-270.
6 *Conrad et al. (Online duplicate document detection: signature reliability in a dynamic retrieval environment), Conference on Information and knowledge management, proceedings of the twelfth international conference on information and knowledge management, 2003, 443-452.
7 *Culotta et al. (Joint deduplication of multiple record types in relational data), Conference on Information and knowledge management, proceedings of the 14th ACM international conference on information and knowledge management, 2005, pp. 257-258.
8 *Peter Christen (Febrl-: an open source data cleaning, deduplication and record linkage system with a Graphical User Interface, Conference on Knowledge Discovery in Data, Proceeding of the 14th ACM SIGKDD international conference on knowledge discovery and data mining, 2008, pp. 1065-1068.
9 *Sarawagi et al. (ALIAS: an active learning led interactive deduplication system), Very Large Data Bases, Proceeding of the 28th internaitonal conference on Very Large Data Bases. 2002, pp. 1103-1106.
10 *Sarawagi et al. (Interactive deduplication using active learning), Conference on knowledge discovery in data, proceeding of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 2002, 269-278.
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US7814074 *Oct 12, 2010International Business Machines CorporationMethod and system for assuring integrity of deduplicated data
US7814149 *Oct 12, 2010Symantec Operating CorporationClient side data deduplication
US8112496Feb 7, 2012Microsoft CorporationEfficient algorithm for finding candidate objects for remote differential compression
US8117173 *Apr 28, 2009Feb 14, 2012Microsoft CorporationEfficient chunking algorithm
US8140491 *Mar 26, 2009Mar 20, 2012International Business Machines CorporationStorage management through adaptive deduplication
US8140821Dec 18, 2009Mar 20, 2012Emc CorporationEfficient read/write algorithms and associated mapping for block-level data reduction processes
US8156306Dec 18, 2009Apr 10, 2012Emc CorporationSystems and methods for using thin provisioning to reclaim space identified by data reduction processes
US8166314Apr 24, 2012Emc CorporationSelective I/O to logical unit when encrypted, but key is not available or when encryption status is unknown
US8195629 *Dec 16, 2010Jun 5, 2012Tofano Jeffrey VincentEstablishing parse scope
US8224792 *Jul 17, 2012International Business Machines CorporationGeneration of realistic file content changes for deduplication testing
US8238549 *Aug 7, 2012Smith Micro Software, Inc.Efficient full or partial duplicate fork detection and archiving
US8261068Sep 4, 2012Emc CorporationSystems and methods for selective encryption of operating system metadata for host-based encryption of data at rest on a logical unit
US8280859Oct 2, 2012International Business Machines CorporationMethod and system for assuring integrity of deduplicated data
US8285690 *Sep 18, 2009Oct 9, 2012Hitachi, Ltd.Storage system for eliminating duplicated data
US8332612Mar 5, 2012Dec 11, 2012Emc CorporationSystems and methods for using thin provisioning to reclaim space identified by data reduction processes
US8380681Dec 16, 2010Feb 19, 2013Microsoft CorporationExtensible pipeline for data deduplication
US8401185 *Mar 19, 2013Symantec CorporationSystems and methods for securely deduplicating data owned by multiple entities
US8407193Mar 26, 2013International Business Machines CorporationData deduplication for streaming sequential data storage applications
US8416954Apr 9, 2013Emc CorporationSystems and methods for accessing storage or network based replicas of encrypted volumes with no additional key management
US8423556 *Apr 16, 2013Fujitsu LimitedArchive device
US8423590Apr 16, 2013International Business Machines CorporationFile generation for testing single-instance storage algorithm
US8458145Jun 4, 2013Infinidat Ltd.System and method of storage optimization
US8468320Jun 30, 2008Jun 18, 2013Symantec Operating CorporationScalability of data deduplication through the use of a locality table
US8515909Apr 29, 2008Aug 20, 2013International Business Machines CorporationEnhanced method and system for assuring integrity of deduplicated data
US8533407Dec 1, 2010Sep 10, 2013International Business Machines CorporationCapacity management in deduplication storage systems
US8560507May 7, 2012Oct 15, 2013International Business Machines CorporationGeneration of realistic file content changes for deduplication testing
US8589640Oct 14, 2011Nov 19, 2013Pure Storage, Inc.Method for maintaining multiple fingerprint tables in a deduplicating storage system
US8645333 *May 29, 2008Feb 4, 2014International Business Machines CorporationMethod and apparatus to minimize metadata in de-duplication
US8645335Dec 16, 2010Feb 4, 2014Microsoft CorporationPartial recall of deduplicated files
US8781800Nov 1, 2011Jul 15, 2014International Business Machines CorporationData de-duplication in computer storage systems
US8793227Sep 7, 2012Jul 29, 2014Hitachi, Ltd.Storage system for eliminating duplicated data
US8812461Sep 19, 2012Aug 19, 2014International Business Machines CorporationMethod and system for data deduplication
US8825617 *Mar 14, 2008Sep 2, 2014International Business Machines CorporationLimiting deduplication based on predetermined criteria
US8825720 *Apr 12, 2011Sep 2, 2014Emc CorporationScaling asynchronous reclamation of free space in de-duplicated multi-controller storage systems
US8848918Jun 4, 2012Sep 30, 2014International Business Machines CorporationMethod and system for deleting data
US8862876Nov 9, 2010Oct 14, 2014International Business Machines CorporationMethod and system for deleting data
US8909657Sep 30, 2011Dec 9, 2014Apple Inc.Content based file chunking
US8918605Jun 4, 2012Dec 23, 2014International Business Machines CorporationCapacity management in deduplication storage systems
US8930307Sep 30, 2011Jan 6, 2015Pure Storage, Inc.Method for removing duplicate data from a storage array
US8990171Sep 1, 2011Mar 24, 2015Microsoft CorporationOptimization of a partially deduplicated file
US9069786Nov 18, 2013Jun 30, 2015Pure Storage, Inc.Method for maintaining multiple fingerprint tables in a deduplicating storage system
US9147374May 21, 2013Sep 29, 2015International Business Machines CorporationControlling real-time compression detection
US9164688Jul 3, 2012Oct 20, 2015International Business Machines CorporationSub-block partitioning for hash-based deduplication
US9176978 *Feb 5, 2009Nov 3, 2015Roderick B. WidemanClassifying data for deduplication and storage
US9183218 *Jun 29, 2012Nov 10, 2015Emc CorporationMethod and system to improve deduplication of structured datasets using hybrid chunking and block header removal
US9305008Dec 8, 2014Apr 5, 2016Apple Inc.Content based file chunking
US9306912Sep 3, 2013Apr 5, 2016Lenovo Enterprise Solutions (Singapore) Pte. Ltd.Bookmarking support of tunneled endpoints
US9317519Jun 20, 2014Apr 19, 2016Hitachi, Ltd.Storage system for eliminating duplicated data
US20050187944 *Feb 10, 2004Aug 25, 2005Microsoft CorporationSystems and methods for a database engine in-process data provider
US20060047855 *May 13, 2004Mar 2, 2006Microsoft CorporationEfficient chunking algorithm
US20090089326 *Sep 28, 2007Apr 2, 2009Yahoo!, Inc.Method and apparatus for providing multimedia content optimization
US20090234795 *Mar 14, 2008Sep 17, 2009International Business Machines CorporationLimiting deduplcation based on predetermined criteria
US20090234892 *Mar 14, 2008Sep 17, 2009International Business Machines CorporationMethod and system for assuring integrity of deduplicated data
US20090271454 *Oct 29, 2009International Business Machines CorporationEnhanced method and system for assuring integrity of deduplicated data
US20090271528 *Apr 28, 2009Oct 29, 2009Microsoft CorporationEfficient chunking algorithm
US20090300321 *May 29, 2008Dec 3, 2009International Business Machines CorporationMethod and apparatus to minimize metadata in de-duplication
US20100064141 *Jul 31, 2009Mar 11, 2010Microsoft CorporationEfficient algorithm for finding candidate objects for remote differential compression
US20100142701 *Dec 5, 2008Jun 10, 2010Smith Micro Software, Inc.Efficient full or partial duplicate fork detection and archiving
US20100198797 *Feb 5, 2009Aug 5, 2010Wideman Roderick BClassifying data for deduplication and storage
US20100250501 *Sep 30, 2010International Business Machines CorporationStorage management through adaptive deduplication
US20100299311 *Aug 2, 2010Nov 25, 2010International Business Machines CorporationMethod and system for assuring integrity of deduplicated data
US20110016132 *Jan 20, 2011Fujitsu LimitedArchive device
US20110022718 *Jul 22, 2010Jan 27, 2011Evans Nigel RonaldData Deduplication Apparatus and Method for Storing Data Received in a Data Stream From a Data Store
US20110055171 *Aug 28, 2009Mar 3, 2011International Business Machines CorporationGeneration of realistic file content changes for deduplication testing
US20110185149 *Jul 28, 2011International Business Machines CorporationData deduplication for streaming sequential data storage applications
US20110191305 *Sep 18, 2009Aug 4, 2011Hitachi, Ltd.Storage system for eliminating duplicated data
US20120016848 *Jan 19, 2012Quantum CorporationEstablishing parse scope
US20150019500 *Jul 15, 2013Jan 15, 2015International Business Machines CorporationReducing activation of similarity search in a data deduplication system
US20150052103 *Aug 22, 2014Feb 19, 2015Emc CorporationData replication with delta compression
WO2012097217A1 *Jan 13, 2012Jul 19, 2012Apple Inc.Content based file chunking
Classifications
U.S. Classification1/1, 707/999.204, 707/999.202, 707/999.2
International ClassificationG06F17/30, G06F12/00
Cooperative ClassificationY10S707/99955, G06F17/30159, Y10S707/99953
European ClassificationG06F17/30F7R3C
Legal Events
DateCodeEventDescription
Mar 31, 2008ASAssignment
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HAUSTEIN, NILS;KLEIN, CRAIG A.;TROPPENS, ULF;AND OTHERS;REEL/FRAME:020730/0049;SIGNING DATES FROM 20080320 TO 20080325
Nov 26, 2012REMIMaintenance fee reminder mailed
Apr 14, 2013LAPSLapse for failure to pay maintenance fees
Jun 4, 2013FPExpired due to failure to pay maintenance fee
Effective date: 20130414