WO2008083914B1 - Method, apparatus, and computer program product for data upload in a computing system - Google Patents

Method, apparatus, and computer program product for data upload in a computing system

Info

Publication number
WO2008083914B1
WO2008083914B1 PCT/EP2007/064431 EP2007064431W WO2008083914B1 WO 2008083914 B1 WO2008083914 B1 WO 2008083914B1 EP 2007064431 W EP2007064431 W EP 2007064431W WO 2008083914 B1 WO2008083914 B1 WO 2008083914B1
Authority
WO
WIPO (PCT)
Prior art keywords
file
slot
destination
computer
client
Prior art date
Application number
PCT/EP2007/064431
Other languages
French (fr)
Other versions
WO2008083914A1 (en
Inventor
Joseph Crichton
Michael Zarnick
Original Assignee
Ibm
Ibm Uk
Joseph Crichton
Michael Zarnick
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ibm, Ibm Uk, Joseph Crichton, Michael Zarnick filed Critical Ibm
Priority to DE602007006058T priority Critical patent/DE602007006058D1/en
Priority to EP07866303A priority patent/EP2108228B1/en
Priority to AT07866303T priority patent/ATE465589T1/en
Priority to KR1020097012918A priority patent/KR101063556B1/en
Publication of WO2008083914A1 publication Critical patent/WO2008083914A1/en
Publication of WO2008083914B1 publication Critical patent/WO2008083914B1/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40Support for services or applications
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061Partitioning or combining of resources
    • G06F9/5072Grid computing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/14Multichannel or multilink protocols
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04WWIRELESS COMMUNICATION NETWORKS
    • H04W4/00Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/06Selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS]; Services to user groups; One-way selective calling services

Abstract

A method, system, and computer program product for simultaneous multi-channel upload of a file to one or more servers while ensuring data integrity. A validation scheme employs hashes to allow segments of the data file to be separately validated. Thus, if the upload process is interrupted or otherwise corrupted, segments of previously transferred data which have been transferred correctly may be validated, eliminating the need for re-transmission of that correctly transferred data. Preferably, a grid broker may be incorporated, allowing simultaneous multi-channel upload of data in a grid computing environment.

Claims

24AMENDED CLAIMS received by the International Bureau on 15 July 2008 (15.07.2008)
1. A computer-implemented method for uploading a data file over multiple channels from a client computer to at least one remote destination computer in a computing system, the method comprising the steps of: dividing the data file into a plurality of file slots and a plurality of file components, wherein a file component comprises one or more slots; transferring multiple file slots, each file slot operable to be transferred over one of the multiple channels, to upload the data file from client computer to the at least ooe destination computer; calculating, based upon the data file transferred to the destination computer, a destination-side file slot hash based upon the data in each file slot, a destination-side file component hash based upon the data in each file component, and a destination-side cumulative hash based upon the transferred data file; calculating a client-side cumulative hash based upon the data file resident on the client computer; comparing the destination-side and client-side cumulative hashes to verify integrity of the transferred file; and if the client-side and destination-side cumulative hashes fail to match, determining in which file slot and in which file component an error has occurred, such that uploading of the data file is operable to be re-started beginning with the file slot and file component containing the error.
2. The computer-implemented method of claim 1 , wherein the step of determining in which file slot and in which file component the error has occurred further comprises: comparing the destination-side file component hash and the destination-side file slot hash with the client-side file component hash and client-side file slot hash, respectively.
3. The computer-implemented method of claim 1 , further comprising the step of: generating a culled slot; merging, with the culled slot, transferred data adjacent to the culled slot; wherein calculation of the destination-side cumulative hash is performed using the data corresponding to the resulting culled slot
4. The computer-implemented method of claim 3, further comprising the step of: saving the destination-side cumulative hash state as metadata.
5. The computer-implemented method of claim 1 , further comprising the step of: associating an expirable lease with a slot
6. The computer-implemented method of claim 5, further comprising the steps of: determining whether the lease has expired; and in response to determining that the lease has expired, reclaiming the slot,
7. The computer-implemented method of claim 1 , further comprising the step of: generating a new slot by selecting a smallest file offset that is not represented by the culled slot or any existing slots.
8. The computer-implemented method of claim 1 , wherein the steps are operable to be implemented using multiple client-side intelligent agents equal in number to the multiple channels and at least one destination-side intelligent agent
9. An apparatus for uploading a data file over multiple channels from a client computer to at least one remote destination computer in a computing system, the apparatus comprising: means for dividing the data file into a plurality of file slots and a plurality of file components, wherein a file component comprises one or more slots; means for controlling transfer of multiple file slots, each file slot operable to be transferred over one of the multiple channels, to upload the data file from client computer to the at least one remote destination computer; means for calculating, based upon the data file transferred to the destination computer, a destination-side file slot hash based upon the data in each file slot, a destination-side file component hash based upon the data in each file component, and a destination-side cumulative hash based upon the transferred data file; 26
means for calculating a client-side cumulative hash based upon the data file resident on the client computer; means for comparing the destination-side and client-side cumulative hashes to verify integrity of the transferred file; and means for determining, if the client-side and destination-side cumulative hashes fail to match, in which file slot and in which file component the error has occurred, such that uploading of the data file is operable to be re-started beginning with the file slot and file component containing the error.
10. The apparatus of claim 9, wherein the means for determining in which file slot and in which file component the error has occurred further comprises: means for comparing the destination-side file component hash and the destination-side file slot hash with the client-side file component hash and client-side file slot hash, respectively.
11. The apparatus of claim 9, further comprising: means for generating a culled slot; means for merging, with the culled slot, transferred data adjacent to the culled slot; wherein calculation of the destination-side cumulative hash is performed using the data corresponding to the resulting culled slot.
12. The apparatus of claim 11 , further comprising: means for saving the destination-side cumulative hash state as metadata.
13. The apparatus of claim 9, further comprising: means for associating an expirable lease with a slot.
14. The apparatus of claim 13, further comprising means for determining whether the lease has expired; and means, responsive to a determination that the lease has expired, for reclaiming the slot. 27
15. The apparatus of claim 9, further comprising: means for generating a new slot by selecting a smallest file offset that is not represented by the culled slot or any existing slots.
16. The apparatus of claim 9, wherein the means for controlling transfer of multiple file slots comprises: multiple client-side intelligent agents equal in number to the multiple channels and at least one destination-side intelligent agent.
17. A computer program product comprising: a computer usable medium having computer readable program code for uploading a data file over multiple channels from a client computer to at least one remote destination computer, the computer program product comprising: first computer program code means for dividing the data file into a plurality of file slots and a plurality of file components, wherein a file component comprises one or more slots; second computer program code means for controlling transfer of multiple file slots, each file slot being transferred over one of the multiple channels, to upload the data file from client computer to the at least one destination computer; third computer program code means for calculating, based upon the data file transferred to the destination computer, a destination-side file slot hash based upon the data in each file slot, a destination-side file component hash based upon the data in each file component, and a destination-side cumulative hash; fourth computer program code means for calculating a client-side cumulative hash based upon the data file resident on the client computer, fifth computer program code means for comparing the destination-side and client-side cumulative hashes to verify integrity of the transferred file; and sixth computer program code means for determining, if the client-side and destination-side cumulative hashes fail to match, in which file slot and in which file component the error has occurred, such that uploading of the data file is operable to be re-started beginning with the file slot and file component containing the error. 28
18. The computer program, product of claim 17S further comprising: program code means for comparing the destination-side file component hash and the destination-side file slot hash with the client-side file component hash and client-side file slot hash, respectively.
19. The computer program product of claim 17 further comprising: program code means for generating a culled slot; program code means for merging, with the culled slot, transferred data adjacent to the culled slot; wherein calculation of the destination-side cumulative hash is operable to be performed using the data corresponding to the resulting culled slot.
20. The computer program product of claim 19, further comprising: program code means for saving the destination-side cumulative hash state as metadata.
21. The computer program product of claim 17, further comprising program code means for associating an expirable lease with a slot.
22. The computer program product of claim 21 , further comprising program code means for determinmg whether the lease has expired; and responsive to a determination that the lease has expired, reclaiming the slot
23. The computer program product of claim 17, further comprising program code means for generating a new slot by selecting a smallest file offset that is not represented by the culled slot or any existing slots.
24. The computer program product of claim 17, further comprising multiple client-side intelligent agents equal in number to the multiple channels and at least one destination-side intelligent agent.
PCT/EP2007/064431 2007-01-12 2007-12-21 Method, apparatus, and computer program product for data upload in a computing system WO2008083914A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
DE602007006058T DE602007006058D1 (en) 2007-01-12 2007-12-21 Method, apparatus and computer program for uploading data in a data processing system
EP07866303A EP2108228B1 (en) 2007-01-12 2007-12-21 Method, apparatus, and computer program for data upload in a computing system
AT07866303T ATE465589T1 (en) 2007-01-12 2007-12-21 METHOD, APPARATUS AND COMPUTER PROGRAM FOR UPLOADING DATA INTO A DATA PROCESSING SYSTEM
KR1020097012918A KR101063556B1 (en) 2007-01-12 2007-12-21 Methods, devices, and computer program products for uploading data in computing systems

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US11/622,711 2007-01-12
US11/622,711 US8055902B2 (en) 2007-01-12 2007-01-12 Method, system, and computer program product for data upload in a computing system

Publications (2)

Publication Number Publication Date
WO2008083914A1 WO2008083914A1 (en) 2008-07-17
WO2008083914B1 true WO2008083914B1 (en) 2008-08-28

Family

ID=39332072

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2007/064431 WO2008083914A1 (en) 2007-01-12 2007-12-21 Method, apparatus, and computer program product for data upload in a computing system

Country Status (6)

Country Link
US (1) US8055902B2 (en)
EP (1) EP2108228B1 (en)
KR (1) KR101063556B1 (en)
AT (1) ATE465589T1 (en)
DE (1) DE602007006058D1 (en)
WO (1) WO2008083914A1 (en)

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100121828A1 (en) * 2008-11-11 2010-05-13 You Wang Resource constraint aware network file system
JP5534031B2 (en) * 2010-12-15 2014-06-25 富士通株式会社 Data migration program, computer and data migration method
US20120246205A1 (en) * 2011-03-23 2012-09-27 Hitachi, Ltd. Efficient data storage method for multiple file contents
CN102833294B (en) 2011-06-17 2015-05-20 阿里巴巴集团控股有限公司 File processing method and system based on cloud storage, and server cluster system
US8635270B2 (en) * 2011-09-19 2014-01-21 Verizon Patent And Licensing Inc. Thread mechanism for media and metadata upload
JP5857684B2 (en) * 2011-11-30 2016-02-10 ブラザー工業株式会社 Communication device
US9160697B2 (en) * 2012-01-01 2015-10-13 Qualcomm Incorporated Data delivery optimization
CN103309768B (en) * 2012-03-16 2015-03-11 腾讯科技(深圳)有限公司 Method and device for repairing system files
JP5987445B2 (en) * 2012-04-20 2016-09-07 富士通株式会社 COMMUNICATION CONTROL DEVICE, METHOD, PROGRAM, AND SYSTEM
CA2881567C (en) * 2012-08-08 2023-10-17 Amazon Technologies, Inc. Archival data storage system and method
US20140143201A1 (en) * 2012-11-20 2014-05-22 International Business Machines Corporation Dynamic content file synchronization
CN103248711B (en) * 2013-05-23 2016-08-24 华为技术有限公司 A kind of method of files passe and server
CN107403016A (en) * 2017-08-08 2017-11-28 郑州云海信息技术有限公司 A kind of file uploading method and device
US11768954B2 (en) 2020-06-16 2023-09-26 Capital One Services, Llc System, method and computer-accessible medium for capturing data changes
KR102246581B1 (en) * 2021-01-04 2021-04-29 이선웅 Method for uploading file via cloud computing environment, and proxy server for executing the same
FI129641B (en) * 2021-01-27 2022-06-15 Nokia Technologies Oy Receiving segments of system information block message

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1431864B2 (en) * 1995-02-13 2012-08-22 Intertrust Technologies Corporation Systems and methods for secure transaction management and electronic rights protection
US6049892C1 (en) 1997-02-24 2002-06-04 Ethos Software Corp Process and apparatus for downloading data from a server computer to a client computer
US6477522B1 (en) 1999-06-10 2002-11-05 Gateway, Inc. Dynamic performance based server selection
US6339785B1 (en) 1999-11-24 2002-01-15 Idan Feigenbaum Multi-server file download
US20030074465A1 (en) 2001-10-15 2003-04-17 Zhicheng Tang ADSL downloading with priority transmit queue
US7716659B2 (en) 2001-10-23 2010-05-11 Pitney Bowes Inc. Remote monitoring and software distribution system for servicing inserter systems
US8261059B2 (en) 2001-10-25 2012-09-04 Verizon Business Global Llc Secure file transfer and secure file transfer protocol
US7555559B2 (en) 2003-02-28 2009-06-30 Onion Networks, KK Parallel data transfer over multiple channels with data order prioritization
US7627617B2 (en) * 2004-02-11 2009-12-01 Storage Technology Corporation Clustered hierarchical file services
US7631098B2 (en) 2004-06-08 2009-12-08 International Business Machines Corporation Method, system and program product for optimized concurrent data download within a grid computing environment
JP4337675B2 (en) * 2004-07-23 2009-09-30 ソニー株式会社 Cryptographic processing apparatus and cryptographic processing method
US8103870B2 (en) * 2006-09-12 2012-01-24 Foleeo, Inc. Hive-based peer-to-peer network

Also Published As

Publication number Publication date
WO2008083914A1 (en) 2008-07-17
US20080172392A1 (en) 2008-07-17
DE602007006058D1 (en) 2010-06-02
KR20090097165A (en) 2009-09-15
US8055902B2 (en) 2011-11-08
ATE465589T1 (en) 2010-05-15
KR101063556B1 (en) 2011-09-07
EP2108228B1 (en) 2010-04-21
EP2108228A1 (en) 2009-10-14

Similar Documents

Publication Publication Date Title
WO2008083914B1 (en) Method, apparatus, and computer program product for data upload in a computing system
US7660956B1 (en) Save set bundling for staging
CN106570086B (en) Data migration system and data migration method
US7895501B2 (en) Method for auditing data integrity in a high availability database
US9378155B2 (en) Method for processing and verifying remote dynamic data, system using the same, and computer-readable medium
US9235593B2 (en) Transmitting filesystem changes over a network
CN109799948B (en) Data storage method and device
US20090083732A1 (en) Creation and deployment of distributed, extensible applications
WO2007106569A3 (en) Systems and methods for controlling error reporting and resolution
WO2017039803A1 (en) Systems and methods for verification of code resiliency for data storage
US10146633B2 (en) Data recovery from multiple data backup technologies
WO2020035092A3 (en) Blockchain data storage based on error correction code for permissioned blockchain network
US20130166862A1 (en) Efficient backup replication
US9983948B2 (en) Caching of backup chunks
US9489254B1 (en) Verification of erasure encoded fragments
US7849355B2 (en) Distributed object sharing system and method thereof
US7770065B2 (en) File data restoring system and method of computer operating system and software thereof
US7765371B2 (en) Method and apparatus for full backups in advance
CN107316176A (en) The control method and device of declaration form processing
JP7139347B2 (en) Method for Partial Update of Data Content in Scattered Storage Networks
US9552254B1 (en) Verification of erasure encoded fragments
US20120017075A1 (en) Registry key federation systems and methods
CN105243077A (en) Network based unlimited-length file transmission method and system
US20130297283A1 (en) Information processing device, information processing method, and program
US10489247B2 (en) Generating time-ordered globally unique revision numbers

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 07866303

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 1020097012918

Country of ref document: KR

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2007866303

Country of ref document: EP