WO2002046930A3 - Data storage system and method employing a write-ahead hash log - Google Patents

Data storage system and method employing a write-ahead hash log Download PDF

Info

Publication number
WO2002046930A3
WO2002046930A3 PCT/US2001/044832 US0144832W WO0246930A3 WO 2002046930 A3 WO2002046930 A3 WO 2002046930A3 US 0144832 W US0144832 W US 0144832W WO 0246930 A3 WO0246930 A3 WO 0246930A3
Authority
WO
WIPO (PCT)
Prior art keywords
log
write
hash
ahead
volatile storage
Prior art date
Application number
PCT/US2001/044832
Other languages
French (fr)
Other versions
WO2002046930A2 (en
Inventor
John H Howard
Christopher A Stein
Original Assignee
Sun Microsystems Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sun Microsystems Inc filed Critical Sun Microsystems Inc
Priority to AU2002228677A priority Critical patent/AU2002228677A1/en
Publication of WO2002046930A2 publication Critical patent/WO2002046930A2/en
Publication of WO2002046930A3 publication Critical patent/WO2002046930A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery

Abstract

A data storage system includes a computer coupled to a non-volatile storage, such as a disk drive, through an interconnect. The computer may include a block cache for storing cached copies of data blocks, and a hash table that stores hash values corresponding to the data blocks. Prior to writing back a modified cache block to the non-volatile storage, a log recorder of the computer stores an updated hash value corresponding to the modified cache block within a write-ahead hash log, which may also be also contained in non-volatile storage. In one particular implementation, the log recorder may create a log record including an updated hash value and an address corresponding to a modified cache block. The log recorder may additionally maintain a first pointer value indicative of log records that have been stored to the write-ahead hash log, and a second pointer value indicative of the most recent log record stored in the write-ahead hash log for which a corresponding modified cache block has been stored to the non-volatile storage. These pointer values may be stored in the write ahead hash log with the log records. Log records may be grouped into log blocks which are eventually written to the non-volatile storage as a group. After the log record containing the updated hash value has been successfully written to write-ahead hash log, a cache manager may initiate the write-back of the dirty data block to the non-volatile storage. Until a verification has been made to ensure the corresponding dirty data block has been successfully written back to non-voliatile storage, the old hash value for the data block may also be retained in the write-ahead hash log. If a system, network or disk failure occurs between the writing of the log record containing the updated hash value to the write-ahead hash log and the writing of the corresponding dirty data block to the non-volatile storage, the hash table may be rebuilt according to the records in the write-ahead hash log.
PCT/US2001/044832 2000-12-08 2001-11-30 Data storage system and method employing a write-ahead hash log WO2002046930A2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU2002228677A AU2002228677A1 (en) 2000-12-08 2001-11-30 Data storage system and method employing a write-ahead hash log

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09/733,423 2000-12-08
US09/733,423 US6629198B2 (en) 2000-12-08 2000-12-08 Data storage system and method employing a write-ahead hash log

Publications (2)

Publication Number Publication Date
WO2002046930A2 WO2002046930A2 (en) 2002-06-13
WO2002046930A3 true WO2002046930A3 (en) 2002-08-29

Family

ID=24947523

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2001/044832 WO2002046930A2 (en) 2000-12-08 2001-11-30 Data storage system and method employing a write-ahead hash log

Country Status (3)

Country Link
US (1) US6629198B2 (en)
AU (1) AU2002228677A1 (en)
WO (1) WO2002046930A2 (en)

Families Citing this family (89)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB0030226D0 (en) * 2000-12-12 2001-01-24 Ibm A data storage system and a method of storing data including a multi-level cache
US7315884B2 (en) * 2001-04-03 2008-01-01 Hewlett-Packard Development Company, L.P. Reduction of network retrieval latency using cache and digest
US7428636B1 (en) * 2001-04-26 2008-09-23 Vmware, Inc. Selective encryption system and method for I/O operations
US7260820B1 (en) 2001-04-26 2007-08-21 Vm Ware, Inc. Undefeatable transformation for virtual machine I/O operations
US6751750B1 (en) * 2001-06-01 2004-06-15 Lsi Logic Corporation Method to protect and recover a write ahead log from interruptions
US7356641B2 (en) 2001-08-28 2008-04-08 International Business Machines Corporation Data management in flash memory
US20030046482A1 (en) * 2001-08-28 2003-03-06 International Business Machines Corporation Data management in flash memory
US7117323B1 (en) * 2001-10-23 2006-10-03 Lsi Logic Corporation Cyclic redundancy checking for managing the coherency of mirrored storage volumes
JP3980327B2 (en) * 2001-11-01 2007-09-26 富士通株式会社 Tamper detection system, tamper detection method, and program
US6981104B2 (en) * 2002-07-12 2005-12-27 Hewlett-Packard Development Company, L.P. Method for conducting checkpointing within a writeback cache
JP3944449B2 (en) * 2002-12-19 2007-07-11 株式会社日立製作所 Computer system, magnetic disk device, and disk cache control method
US7124243B2 (en) * 2003-05-16 2006-10-17 Pillar Data Systems, Inc. Methods and systems of cache memory management and snapshot operations
US7380059B2 (en) * 2003-05-16 2008-05-27 Pillar Data Systems, Inc. Methods and systems of cache memory management and snapshot operations
US7299379B2 (en) * 2003-06-27 2007-11-20 Intel Corporation Maintaining cache integrity by recording write addresses in a log
EP1669877B1 (en) * 2003-09-26 2017-11-15 Nippon Telegraph And Telephone Corporation Tag privacy protecting method, tag device, backend device, updating device, update requesting device, programs for these devices, and recording medium storing these programs
US8214328B2 (en) * 2003-11-25 2012-07-03 International Business Machines Corporation High-performance asynchronous peer-to-peer remote copy for databases
TWI289304B (en) * 2004-02-17 2007-11-01 Hon Hai Prec Ind Co Ltd Data structure design system and method for prolonging useful life of FRAM
US7266727B2 (en) * 2004-03-18 2007-09-04 International Business Machines Corporation Computer boot operation utilizing targeted boot diagnostics
ES2363517T3 (en) * 2004-04-02 2011-08-08 Panasonic Corporation SYSTEM OF DETECTION OF CONTENT NOT AUTHORIZED.
US7644239B2 (en) 2004-05-03 2010-01-05 Microsoft Corporation Non-volatile memory cache performance improvement
US7114658B2 (en) * 2004-10-08 2006-10-03 Lsi Logic Corporation Intelligent transportable memory apparatus and method for charging a battery for a memory in an intelligent transportable memory apparatus
US7490197B2 (en) 2004-10-21 2009-02-10 Microsoft Corporation Using external memory devices to improve system performance
US7380157B2 (en) * 2004-10-27 2008-05-27 Pillar Data Systems, Inc. Staggered writing for data storage systems
US7634657B1 (en) * 2004-12-23 2009-12-15 Symantec Corporation Reducing the probability of undetected collisions in hash-based data block processing
US7451166B2 (en) * 2005-01-13 2008-11-11 International Business Machines Corporation System and method for maintaining checkpoints of a keyed data structure using a sequential log
US8762635B1 (en) 2005-03-31 2014-06-24 Google Inc. Method and apparatus for selecting and storing data on a hard disk drive
EP1710764A1 (en) * 2005-04-07 2006-10-11 Sap Ag Authentication of products using identification tags
US20070038891A1 (en) * 2005-08-12 2007-02-15 Stratus Technologies Bermuda Ltd. Hardware checkpointing system
JP4783112B2 (en) * 2005-10-11 2011-09-28 株式会社日立製作所 Signature history storage device
US7721143B2 (en) * 2005-12-06 2010-05-18 Lsi Corporation Method for reducing rebuild time on a RAID device
US8914557B2 (en) 2005-12-16 2014-12-16 Microsoft Corporation Optimizing write and wear performance for a memory
US7702870B2 (en) * 2006-01-19 2010-04-20 Network Appliance Inc. Method and apparatus for defragmentation and for detection of relocated blocks
US7395377B2 (en) * 2006-04-20 2008-07-01 International Business Machines Corporation Method and system for adaptive back-off and advance for non-volatile storage (NVS) occupancy level management
US7603587B2 (en) * 2006-05-26 2009-10-13 Intel Corporation Updating a memory block without status logging
US7895211B2 (en) * 2006-11-03 2011-02-22 International Business Machines Corporation Method and system for reinserting a chain in a hash table
US8316258B2 (en) * 2007-05-03 2012-11-20 Oracle America, Inc. System and method for error detection in a data storage system
US9424266B2 (en) * 2007-10-01 2016-08-23 Microsoft Technology Licensing, Llc Efficient file hash identifier computation
US8631203B2 (en) 2007-12-10 2014-01-14 Microsoft Corporation Management of external memory functioning as virtual cache
US8204868B1 (en) * 2008-06-30 2012-06-19 Symantec Operating Corporation Method and system for improving performance with single-instance-storage volumes by leveraging data locality
US8032707B2 (en) 2008-09-15 2011-10-04 Microsoft Corporation Managing cache data and metadata
US9032151B2 (en) * 2008-09-15 2015-05-12 Microsoft Technology Licensing, Llc Method and system for ensuring reliability of cache data and metadata subsequent to a reboot
US7953774B2 (en) 2008-09-19 2011-05-31 Microsoft Corporation Aggregation of write traffic to a data store
US8037033B2 (en) * 2008-09-22 2011-10-11 Microsoft Corporation Log manager for aggregating data
US8090683B2 (en) * 2009-02-23 2012-01-03 Iron Mountain Incorporated Managing workflow communication in a distributed storage system
US8397051B2 (en) 2009-02-23 2013-03-12 Autonomy, Inc. Hybrid hash tables
US8145598B2 (en) * 2009-02-23 2012-03-27 Iron Mountain Incorporated Methods and systems for single instance storage of asset parts
US20100215175A1 (en) * 2009-02-23 2010-08-26 Iron Mountain Incorporated Methods and systems for stripe blind encryption
US8156290B1 (en) 2009-04-23 2012-04-10 Network Appliance, Inc. Just-in-time continuous segment cleaning
US8224780B2 (en) * 2010-06-15 2012-07-17 Microsoft Corporation Checkpoints for a file system
US10157051B2 (en) * 2012-03-02 2018-12-18 International Business Machines Corporation Upgrading devices in a dispersed storage network
WO2014087458A1 (en) * 2012-12-06 2014-06-12 Hitachi, Ltd. Storage apparatus and data management method
US9251002B2 (en) 2013-01-15 2016-02-02 Stratus Technologies Bermuda Ltd. System and method for writing checkpointing data
US9141554B1 (en) 2013-01-18 2015-09-22 Cisco Technology, Inc. Methods and apparatus for data processing using data compression, linked lists and de-duplication techniques
US8738577B1 (en) * 2013-03-01 2014-05-27 Storagecraft Technology Corporation Change tracking for multiphase deduplication
US8732135B1 (en) 2013-03-01 2014-05-20 Storagecraft Technology Corporation Restoring a backup from a deduplication vault storage
US8874527B2 (en) 2013-03-01 2014-10-28 Storagecraft Technology Corporation Local seeding of a restore storage for restoring a backup from a remote deduplication vault storage
US8682870B1 (en) 2013-03-01 2014-03-25 Storagecraft Technology Corporation Defragmentation during multiphase deduplication
US9448877B2 (en) * 2013-03-15 2016-09-20 Cisco Technology, Inc. Methods and apparatus for error detection and correction in data storage systems using hash value comparisons
JP6098301B2 (en) * 2013-03-29 2017-03-22 富士通株式会社 Storage control device, storage control method, and storage control program
GB2514563A (en) * 2013-05-28 2014-12-03 Ibm Selective purging of a log structure
WO2015102875A1 (en) 2013-12-30 2015-07-09 Stratus Technologies Bermuda Ltd. Checkpointing systems and methods of using data forwarding
EP3090345B1 (en) 2013-12-30 2017-11-08 Stratus Technologies Bermuda Ltd. Method of delaying checkpoints by inspecting network packets
JP6518672B2 (en) 2013-12-30 2019-05-22 ストラタス・テクノロジーズ・バミューダ・リミテッド Dynamic check pointing system and method
US8751454B1 (en) 2014-01-28 2014-06-10 Storagecraft Technology Corporation Virtual defragmentation in a deduplication vault
JP6335527B2 (en) * 2014-01-28 2018-05-30 キヤノン株式会社 System, system control method, and computer program
US9350732B1 (en) * 2014-11-07 2016-05-24 Elliptic Technologies Inc. Integrity protection for data storage
US9853873B2 (en) 2015-01-10 2017-12-26 Cisco Technology, Inc. Diagnosis and throughput measurement of fibre channel ports in a storage area network environment
US9900250B2 (en) 2015-03-26 2018-02-20 Cisco Technology, Inc. Scalable handling of BGP route information in VXLAN with EVPN control plane
US10222986B2 (en) 2015-05-15 2019-03-05 Cisco Technology, Inc. Tenant-level sharding of disks with tenant-specific storage modules to enable policies per tenant in a distributed storage system
US11588783B2 (en) 2015-06-10 2023-02-21 Cisco Technology, Inc. Techniques for implementing IPV6-based distributed storage space
US10778765B2 (en) 2015-07-15 2020-09-15 Cisco Technology, Inc. Bid/ask protocol in scale-out NVMe storage
US10209891B2 (en) 2015-08-24 2019-02-19 Western Digital Technologies, Inc. Methods and systems for improving flash memory flushing
US9892075B2 (en) 2015-12-10 2018-02-13 Cisco Technology, Inc. Policy driven storage in a microserver computing environment
US10067708B2 (en) 2015-12-22 2018-09-04 Arm Limited Memory synchronization filter
US10140172B2 (en) 2016-05-18 2018-11-27 Cisco Technology, Inc. Network-aware storage repairs
US20170351639A1 (en) 2016-06-06 2017-12-07 Cisco Technology, Inc. Remote memory access using memory mapped addressing among multiple compute nodes
US10664169B2 (en) 2016-06-24 2020-05-26 Cisco Technology, Inc. Performance of object storage system by reconfiguring storage devices based on latency that includes identifying a number of fragments that has a particular storage device as its primary storage device and another number of fragments that has said particular storage device as its replica storage device
US11563695B2 (en) 2016-08-29 2023-01-24 Cisco Technology, Inc. Queue protection using a shared global memory reserve
US10353601B2 (en) 2016-11-28 2019-07-16 Arm Limited Data movement engine
US10545914B2 (en) 2017-01-17 2020-01-28 Cisco Technology, Inc. Distributed object storage
US10243823B1 (en) 2017-02-24 2019-03-26 Cisco Technology, Inc. Techniques for using frame deep loopback capabilities for extended link diagnostics in fibre channel storage area networks
US10713203B2 (en) 2017-02-28 2020-07-14 Cisco Technology, Inc. Dynamic partition of PCIe disk arrays based on software configuration / policy distribution
US10254991B2 (en) 2017-03-06 2019-04-09 Cisco Technology, Inc. Storage area network based extended I/O metrics computation for deep insight into application performance
US10303534B2 (en) 2017-07-20 2019-05-28 Cisco Technology, Inc. System and method for self-healing of application centric infrastructure fabric memory
US10404596B2 (en) 2017-10-03 2019-09-03 Cisco Technology, Inc. Dynamic route profile storage in a hardware trie routing table
US10942666B2 (en) 2017-10-13 2021-03-09 Cisco Technology, Inc. Using network device replication in distributed storage clusters
US11132145B2 (en) * 2018-03-14 2021-09-28 Apple Inc. Techniques for reducing write amplification on solid state storage devices (SSDs)
CN112989404A (en) * 2019-12-18 2021-06-18 中移雄安信息通信科技有限公司 Log management method based on block chain and related equipment
US20220014379A1 (en) * 2020-07-10 2022-01-13 Arm Limited Memory protection using cached partial hash values

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4507751A (en) * 1982-06-21 1985-03-26 International Business Machines Corporation Method and apparatus for logging journal data using a log write ahead data set
US5561795A (en) * 1994-05-13 1996-10-01 Unisys Corporation Method and apparatus for audit trail logging and data base recovery

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5440727A (en) 1991-12-18 1995-08-08 International Business Machines Corporation Asynchronous replica management in shared nothing architectures
US5353410A (en) 1992-03-18 1994-10-04 International Business Machines Corporation Method and system for deferred read in lazy-write disk cache systems
EP1003103B1 (en) * 1993-06-03 2008-10-01 Network Appliance, Inc. Write anywhere file-system layout method and apparatus
US6243795B1 (en) * 1998-08-04 2001-06-05 The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations Redundant, asymmetrically parallel disk cache for a data storage system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4507751A (en) * 1982-06-21 1985-03-26 International Business Machines Corporation Method and apparatus for logging journal data using a log write ahead data set
US5561795A (en) * 1994-05-13 1996-10-01 Unisys Corporation Method and apparatus for audit trail logging and data base recovery

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
MOURAD A N ET AL: "RECOVERY ISSUES IN DATABASES USING REDUNDANT DISK ARRAYS", JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING, ACADEMIC PRESS, DULUTH, MN, US, vol. 17, no. 1 / 2, 1993, pages 75 - 89, XP000343144, ISSN: 0743-7315 *

Also Published As

Publication number Publication date
US6629198B2 (en) 2003-09-30
US20020073276A1 (en) 2002-06-13
WO2002046930A2 (en) 2002-06-13
AU2002228677A1 (en) 2002-06-18

Similar Documents

Publication Publication Date Title
WO2002046930A3 (en) Data storage system and method employing a write-ahead hash log
US11093149B2 (en) Method to efficiently store object data of an object storage service on a magnetic disk drive and magnetic SMR disk drive
US6732293B1 (en) Method, software and apparatus for recovering and recycling data in conjunction with an operating system
JP3751018B2 (en) LightAnywhere file system layout
Quinlan A cached worm file system
US7587564B2 (en) System, method and computer program product for managing data versions
CN103080910B (en) Storage system
JP5047308B2 (en) Nonvolatile disk cache for data security
EP1522927A3 (en) File storage apparatus capable of restoring file management information
US20060174067A1 (en) Method of caching data
US20070118695A1 (en) Decoupling storage controller cache read replacement from write retirement
CN101369218B (en) Method for deleting data volume in storage system and data storage controller
JP2010514008A5 (en)
WO2004059651A3 (en) Nonvolatile memory unit with specific cache
US20150248348A1 (en) Physically-addressable solid state disk (ssd) and a method of addressing the same
GB2278487B (en) Improved recording apparatus and method for an arrayed recording apparatus
JP2004213647A (en) Writing cache of log structure for data storage device and system
KR20090111806A (en) Improved sequential media reclamation and replication
CN101008919A (en) Audio/video data access method and device based on raw device
JP2010218529A (en) Method for storing data in flash memory by isolation-method at dbms, using page-differencial
CN105955664B (en) A kind of reading/writing method of watt record conversion layer based on segment structure
JP2002149499A5 (en)
US8019953B2 (en) Method for providing atomicity for host write input/outputs (I/Os) in a continuous data protection (CDP)-enabled volume using intent log
CN102314503A (en) Indexing method
CN107273046A (en) A kind of data processing method and system based on Solid-state disc array

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PH PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP