DE112004002315B4 - Virtual ordered write entries for multiple storage devices - Google Patents

Abstract

A method of ordering data write entries, comprising the steps of: at least one of a group of primary storage devices receiving a first plurality of data write entries made a first time; a transmit cycle switch is caused to a new send cycle for the at least one of the groups of primary storage devices, the first plurality of data write entries being associated with a particular send cycle for the at least one of the primary storage devices in the group; the at least one of the group of primary storage devices receives a second plurality of write entries after initialization of the transmit cycle switch, wherein all write entries of the second plurality of write entries are associated with a transmit cycle from the particular transmit cycle for the at least one of the primary storage devices is different and causes the second plurality of write entries after said first time; wherein the write entries to the group which started after the transmission cycle switching has been initialized are not completed until after the completion of the transmission cycle switching operation; and after completion of the transmit cycle switching operation, the at least one of the primary storage devices of the group initiates transfer of the first plurality of write entries to a corresponding secondary storage device, and wherein causing the transmit cycle switch comprises: waiting a predetermined amount of time; Determining if all the primary storage devices of the group of storage devices are ready for the switching operation; and sending a first command to each of the primary storage devices of the group to effect a send cycle switch.

Description

Background of the invention

1. Technical area

The present application relates to computer storage devices, and more particularly to the field of transferring data between storage devices.

2. Description of the Related Art

Host processor systems can store and retrieve data using a storage device containing a variety of host interface units (host adapters), disk drives, and disk interface units (disk adapters). Such storage devices are, for example, supplied by EMC Corporation of Hopkinton, Massachusetts, and are incorporated herein by reference U.S. Patent No. 5,206,939 by Yanai et al., the U.S. Patent No. 5,778,394 by Galtzur et al., the U.S. Patent No. 5,845,147 by Vishlitzky et al. and the U.S. Patent No. 5,857,208 revealed by Ofek. The host systems access the storage device via a plurality of channels provided therewith. The host systems provide data and access control information to the storage device through the channels, and the storage device provides data to the host systems through the channels. The host systems do not directly address the disk drives of the storage device, but implement an access that appears in the host systems in the form of a plurality of logical disk units. The logical disk units may or may not match the actual disk drives. The multiple host systems are given the ability to access the single storage device unit, and the host systems can share the data stored therein.

In some cases, it may be desirable to copy data from one storage device to another. When a host writes data to a first storage device, it may be desirable for the data to be copied to a second storage device provided at a different location such that when disaster occurs, the first storage device becomes inoperable will allow the host (or other host) to resume operation using the data stored in the second storage device. Such a capability is realized, for example, by the Remote Data Facility Product (RDF), which is manufactured by the EMC Corporation of Hopkinton, Massachusetts. Using the RDF, a first storage device, referred to as the "primary storage device" (or "R1"), is coupled to the host. One or more other storage devices, referred to as "secondary storage devices" (or "R2"), receive copies of the data written to the primary storage device by the host. The host interacts directly with the primary storage device, however, any data changes made to the primary storage device can be automatically realized on one or more of the secondary storage devices using RDF. The primary and secondary storage devices may be connected by a data link such as an ESCON link, a Fiber Channel link, and / or a Gigabit Ethernet link. The RDF functionality can be simplified using an RDF adapter (RA) provided with each of the storage devices.

RDF enables synchronous data transfer where, after the data written by a host to the primary storage device is transferred from the primary storage device to a secondary storage device using RDF, receipt by the secondary storage device is acknowledged to the primary storage device, which then returns a write acknowledgment message to the host. Thus, in the synchronous mode, the host does not receive a write acknowledgment from the primary storage device until the RDF transfer to the secondary storage device has been completed and acknowledged by the secondary storage device.

A disadvantage with the synchronous RDF system is that the latency of each of the write operations is increased due to a wait for acknowledgment of the RDF transfer. This problem is compounded when there is a long distance between the primary storage device and the secondary storage device; since transmission delays occur, there is also a time delay to perform the RDF transfer, and then to wait for an acknowledgment that will come back after the transfer has been completed, which is unacceptable.

It is also possible to use RDF in a semi-synchronous mode, in which case the data is written by the host to the primary storage device confirming the write immediately and then at the same time the process of transferring the data to the secondary storage device starts. Thus, for a single data transfer, this scheme overcomes some of the disadvantages of using the RDF in synchronous mode.

However, for the purpose of data integrity, the half-synchronous transfer mode does not allow the primary storage device to transfer data to the secondary storage device until earlier transfer by the secondary storage device has been confirmed. Thus, the bottle necks associated with RDF in the synchronous mode are simply delayed by an iteration, since the transfer of a second amount of data can not take place until the transfer of previous data by the secondary storage device has been acknowledged or acknowledged.

Another possibility is to have the host write data to the primary storage device in an asynchronous mode and cause the primary storage device to copy data to the secondary storage device in the background. Background copying involves traversing each of the tracks of the primary storage device sequentially and, if it is determined that a particular block has been modified, since the last time the block was copied, the block becomes the one from the primary storage device transmitted secondary storage device. Although this mechanism can alleviate the latency problem associated with the synchronous and half-synchronous data transfer modes, there is a difficulty in providing no guarantee of data consistency between the primary and secondary storage devices. When problems occur, such as a failure of the primary system, the secondary system may end up with out-of-order changes which then renders the data unusable.

One proposed solution to this problem is the Symmetrix Automated Replication (SAR) process described in pending US patent applications 10 / 224,918 and 10 / 225,021, both of which were filed on August 21, 2002. The SAR process uses devices (BCVs) that can mirror standard logical devices. A BCV device may also be split from its standard logical device after mirroring has occurred and may be re-integrated (that is, restored as a mirror) to the standard logical devices after splitting. Additionally, BCV may be remotely mirrored using RDF, in which case the BCV may propagate data changes made thereto (using BCV acting as a mirror) to the remote BCV mirror when BCV has been split off the corresponding standard logical device.

However, the use of the SAR process requires a significant overhang of a continuous split operation and reintegration operation of the BCVs. The SAR process also uses host control and management based on the control of the host that is operating. In addition, the cycle time for a practical implementation of a SAR process may be on the order of twenty to thirty minutes, and thus an amount of data may be lost if an RDF link and / or a primary device fails, which will then take twenty to thirty minutes of data can cost.

It would therefore be desirable to have an RDF system which has some of the advantageous qualities of each of the different techniques as discussed above but whose disadvantages are reduced. Such a system should have a low latency for each host write regardless of the distance between the primary device and the secondary device and would provide consistency (recoverability) to the second device in the event of failure.

Further relevant prior art is in US 6 301 643 B1 . US Pat. No. 6,415,292 B1 . US 6 157 991 A . US Pat. No. 6,209,002 B1 . US 5 901 327 A and US 5,845,147 A disclosed.

In particular disclosed US Pat. No. 6,415,292 B1 a shuttle-based mechanism for numbering simultaneous chains of independent data transfers. Data is transferred from a source to a destination, which involves receiving data from the source as well as providing the data to the destination using a network. The data is confirmed by the source as successfully received at the destination before all the data is made available to the network. The source may be a primary storage device and the destination may be a secondary storage device. The data may be provided from the first source in a first format and may be provided to the network in a second format that differs from the first format and where the first format may be RDF and the second format TCP / IP or UDP. The destination may receive the data in a first format that differs from the second format used to transmit the data over the network. The first format can be RDF and the second format can be TCP / IP or UDP.

Summary of the invention

The object of the present invention is therefore a method for sorting To create data write entries for a computer storage system, such as an RDF system, that has a low latency f for each write regardless of the distance between the primary device and the secondary device.

This object is achieved by the combination of features of claim 1. Further advantageous embodiments and further developments are the subject of the subsequent claims.

In accordance with the present invention, ordered data write entries include at least some of a group of the primary storage devices receiving a first plurality of data write entries causing a cycle switch operation for the group of primary storage devices where the first plurality of data write entries are one particular cycle at each one primary memory device is allocated in each group, and wherein at least some of the group of primary memory devices receives a second plurality of write entries after initializing the cycle switching operation in which all of the second plurality of write entries associated with a cycle which of the one certain cycle at each primary storage device is different. The write entries to the group which started after the cycle cycle initialization are not completed until after the cycle switch operation has been completed. The ordering of the data write entries, even after the completion of the cycle switching operation, may include that each of the primary storage devices of the group being assigned a transfer of the first plurality of the write entries to a corresponding secondary storage device. The ordering of the data write entries or writes may also include subsequent to completing the transfer of the first plurality of write entries to a corresponding secondary storage device through each of the primary storage devices of the group, each of the primary storage devices sending a message to the corresponding secondary storage device. The ordering of the data write entries may also include providing the first plurality of data write entries to cache slots of the group of the primary storage device. Receiving a first plurality of data write entries may also include receiving a plurality of the data write entries from a host. A host can cause the cycle switch operation. The ordering of the data write entries may also include waiting a predetermined period of time, determining whether all the primary storage devices of the group of storage devices are ready for the switching operation, and sending a first command for each of the primary storage devices of the group to cycle-switch cause. Sending a command to effect a cycle switch can also cause write entries started after the first command not to complete until a second command is received. Ordering the data write entries may also include sending a second command to all of the primary storage devices after sending the first command to all of the primary storage devices of the group to allow the writes to be completed.

Further, in accordance with the present invention, computer software associating the data write entries with a group of primary storage devices includes executable code that effects a cycle switch operation for the group of primary storage devices, wherein the first plurality of data write entries correspond to a particular cycle at each primary storage device in the primary storage device Group and containing executable code for a second plurality of the write entries after the initialization of the cycle switching operation assigns all of the second plurality of write entries to a cycle that is different from the particular cycle at each primary storage device. The write entries to the group which started after the initialization of the cycle switch operation are not completed until the cycle switch operation has been completed. The computer software may also include executable code that causes each of the group's primary storage devices to initialize the transfer of the first plurality of write entries to a corresponding secondary storage device after completing the cycle switch. The computer software may also include executable code that causes each of the primary storage devices to send a message to the corresponding secondary storage device subsequent to each of the primary storage devices of the group having completed the transfer of the first plurality of write entries to a corresponding secondary storage device , The computer software may also include executable code that provides the first plurality of data write entries to cache slots of the group of the primary storage device. The first plurality of data write entries can come from a host. A host may generate executable code that effects the cycle switch. The executable code that effects the cycle switching operation may also include executable code that waits a predetermined period of time, executable code that determines whether all the primary storage devices of the group of storage devices are ready for switching, and an executable code that sends a first instruction to each of the primary storage devices of the group to effect the cycle switching operation. The executable code by which a command is sent to effect the cycle switch may also cause writes initiated after the first command to not be completed until a second command is received. The computer software may also include executable code that sends a second command to all of the primary storage devices to allow writes to be completed after the first command has been sent to all the primary storage devices of the group.

Brief description of the drawings

1 Fig. 12 is a schematic diagram showing a host, a logical storage device, and a remote data storage device used in conjunction with the system described herein;

2 Fig. 10 is a schematic diagram illustrating a data flow between a host, a local storage device, and a remote data storage device used in conjunction with the system described herein;

3 Fig. 12 illustrates a schematic diagram representing elements for constructing and manipulating blocks of data on a logical storage device according to the system described herein;

4 Figure 11 is a diagram illustrating a data structure for a slot used in conjunction with the system described herein;

5 Fig. 12 illustrates a flowchart illustrating the operation of a host adapter (HA) in response to a write by a host in accordance with the system described herein;

6 Fig. 10 is a flow chart illustrating the transfer process of data from a local storage device to a remote storage device according to the system described herein;

7 Fig. 12 is a schematic diagram illustrating elements for constructing and manipulating blocks of data at a remote storage device in accordance with the system described herein;

8th FIG. 12 is a flow chart illustrating the steps performed by a remote storage device in conjunction with the receipt of a commit indicator from a local storage device according to the system described herein; FIG.

9 FIG. 12 illustrates a flow chart illustrating the storage operation of transmitted data at a remote storage device according to the system described herein; FIG.

10 FIG. 12 is a flow chart illustrating the steps performed in conjunction with a local storage device that increments a sequence number according to the system described herein; FIG.

11 Fig. 12 is a schematic diagram illustrating elements for constructing and manipulating blocks of data on a logical storage device according to an alternative embodiment of the system described herein;

12 FIG. 10 is a flow chart depicting the operation of a host adapter (HA) in response to a write by a host in accordance with an alternative embodiment of the system described herein; FIG.

13 FIG. 12 is a flow chart illustrating the transfer process of data from a local storage device to a remote storage device according to an alternative embodiment of the system described herein; FIG.

14 FIG. 12 is a schematic diagram depicting a plurality of local and remote storage devices with a host according to the system described herein; FIG.

15 show a diagram illustrating a multi-box mode table used in conjunction with the system described herein;

16 FIG. 10 is a flowchart illustrating a modified multi-box mode table according to the system described herein; FIG.

17 FIG. 12 is a flowchart showing the cycle switching operation by a host according to the system described herein; FIG.

18 FIG. 10 is a flowchart illustrating the steps performed in conjunction with a local storage device that increments a sequence number according to the system described herein; FIG.

19 FIG. 12 is a flowchart illustrating the transfer operation of data from a local storage device to a remote storage device according to the system described herein; FIG.

20 FIG. 12 is a flow chart depicting the transfer operation of data from a local storage device to a remote storage device according to an alternative embodiment of the system described herein; FIG.

21 FIG. 12 is a flowchart illustrating delivering an active empty indicator message from a remote storage device to a corresponding local storage device according to the system described herein; FIG.

22 Fig. 12 is a schematic diagram showing a plurality of local and remote storage devices having a plurality of hosts in accordance with the system described herein;

23 Fig. 10 illustrates a flowchart representing processing performed by a remote storage device in conjunction with data retrieval according to the system described herein;

24 FIG. 10 is a flowchart illustrating processing performed by a host in conjunction with data retrieval according to the system described herein. FIG.

Detailed description of the various embodiments

According to 1 shows a diagram 20 a relationship between a host 22 , a local storage device 24 and a remote storage device 26 , The host 22 reads and writes data to and from the local storage device 24 through a host adapter (HA) 28 that is the coupling between the host 22 and the local storage device 24 simplified. Although the diagram 20 just a host 22 and a HA 28 It should be appreciated by those skilled in the art that a variety of HAs may be used and that one or more HAs may also include one or more hosts coupled thereto.

Data from the local storage device 24 become the remote storage device 26 via an RDF link 29 copied to achieve that data at the remote storage device 26 identical to the data on the local storage device 24 are. Although only one link 29 is shown, it is possible additional links between the storage devices 24 . 26 provide and also connecting links between one or both of the storage devices 24 . 26 and also other memory devices (not shown). It should be noted that there is a time delay between the transfer of the data from the local storage device 24 to the remote storage device 26 can occur, leaving the remote storage device 26 at certain times may contain data that is not identical to the data at the local storage device 24 are. Communication using RDF is for example in the U.S. Patent No. 5,742,792 which is incorporated herein by reference.

The local storage device 24 contains a first variety of RDF Adapter Units (RAs) 30a . 30b . 30c and the remote storage device 26 contains a second variety of RAs 32a - 32c , The RAs 30a - 30c . 32a - 32c are to the RDF connector 29 coupled and are similar to the host adapter 28 However, they are used to transfer data between the storage devices 24 . 26 used. The in conjunction with the RAs 30a - 30c . 32a - 32c Software used is explained in more detail below.

The storage devices 24 . 26 may include one or more disks, each containing a different portion of data stored in each of the storage devices 24 . 26 are stored. 1 shows the storage device 24 containing a variety of disks or disk drives 33a . 33b . 33c contains, and wherein the memory device 26 a variety of disks or disk drives 34a . 34b . 34c contains. The RDF functionality that is operated can be applied so that the data for at least a portion of the disks 33a - 33c the local storage device 24 using RDF to at least a portion of the disks 34a - 34c the remote storage device 26 be copied. It is possible that other data of the storage devices 24 . 26 not between the storage devices 24 . 26 are copied and therefore not identical.

Each of the plates 33a - 33c is with a corresponding disk adapter unit (DA) 35a . 35b . 35c coupled the data to a corresponding one of the disks 33a - 33c deliver and data from a corresponding one of the plates 33a - 33c receive. Similarly, a variety of DAs 36a . 36b . 36c the remote storage device 26 used to transfer data to corresponding disks of the disk drives 34a - 34c to deliver and to get data from corresponding disks of the disk drives 34a - 34c to recieve. There is an internal data path between the DAs 35a - 35c . the HA 28 and the RAs 30a - 30c the local storage device 24 , Similarly, an internal data path exists between the DAs 36a - 36c and the RAs 32a - 32c the remote storage device 26 , It should be noted that in other embodiments, it is possible for more than one disk to be serviced by a DA, and for more than one DA to be able to service a disk drive.

The local storage device 24 also contains a global store 37 which can be used to transfer data between the DAs 35a - 35c , the HA 28 and the RAs 30a - 30c to simplify. The memory 37 may contain tasks that are performed by one or more of the DAs 35a - 35c , the HA 28 and the RAs 30a - 30c and may include a cache to store data from one or more of the disks 33a - 33c take. Similarly, the remote storage device includes 26 a global store that may contain tasks that are performed by one or more of the DAs 36a - 36c and the RAs 32a - 32c and a cache to retrieve data from one or more of the disks 34a - 34c take. The use of memory 37 . 38 will be described in more detail below.

The storage space in the local storage device 24 that the plates 33a - 33c may be divided into a plurality of volumes or logical devices. The logical devices may or may not be the physical storage space of the disks 33a - 33c correspond. For example, thus, the plate 33a contain a plurality of logical devices, or alternatively, a single logical device can both plates 33a . 33b span. Similarly, the storage space for the remote storage device 26 which the plates 34a - 34c comprises, subdivided into a plurality of volumes or logical devices, each of the logical devices of one or more of the disks 34a - 34c may or may not correspond.

The provision of an RDF memory map between portions of the local storage device 24 and the remote storage device 26 involves creating a logical device on the remote storage device 26 that is, creating a remote mirror for a logical device on the local storage device 24 , The host 22 reads and writes data from and to the logical device at the local storage device 24 and the RDF map mapping causes modified data from the local storage device 24 to the remote storage device 26 be transmitted using the RAs 30a - 30c . 32a - 32c and the RDF link 29 , In the steady state operation, the logical device includes at the remote storage device 26 Data identical to the data of the logical device at the local storage device 24 are. The logical device on the local storage device 24 by the host 22 is referred to as "R1 Volume" (or just "R1") while the logical device is at the remote storage device 26 containing a copy of the data at the R2 volume called "R2 Volume" (or just "R2"). Thus, the host reads and writes data to and from the R1 volume and RDF automatically handles the copying and renewal of the data from the R1 volume to the R2 volume.

According to 2 is a path of data from the host 22 to the local storage device 24 and the remote storage device 26 illustrated. The data coming from the host 22 to the local storage device 24 are written locally, as indicated by the data element 51 the local storage device 24 is illustrated. The data by the host 22 to the local storage device 24 are also written by the local storage device 24 held in connection with this through the local storage device 24 to the remote storage device 26 over the connecting link 29 be sent.

In the system described here, data provided by the host 22 are written (from, for example, a record, a plurality of records, a track, and so on) to a sequence number. The sequence number may be provided in a suitable data field associated with the write entry. In 2 are the write entries by the host 22 as shown, they are assigned to the sequence number N. All the write entries made by the host 22 are performed, which are assigned to the sequence number N, are in a single data block 52 collected. The block 52 represents a multiplicity of separate write entries by the host 22 which occur at approximately the same time.

In general, the local storage device will collect 24 Blocks from a sequence number while storing a previously accumulated block (having the previous sequence number) to the remote storage device 26 sends. Thus, while the local storage device 24 Write entries from the host 22 which allocates the sequence number N, the write entries that occurred for the earlier sequence number (N-1) occur are through the local storage device 24 to the remote storage device 26 over the connecting link 29 Posted. A data block 54 represents write entries from the host 22 which are assigned the sequence number N-1 not yet to the remote storage device 26 have been sent.

The remote storage device 26 receives data from the block 54 corresponding to the write entries assigned a sequence number N-1, and constructs a new data block 56 of host writes or writes having the sequence number N-1. The data may be transmitted using a suitable RDF protocol which acknowledges data sent over the link 29 were sent. If the remote storage device 26 all the data from the block 54 has received, sends the local storage device 24 an acknowledgment or promise message to the remote storage device 26 to confirm all the data to which the sequence number N-1 is assigned according to the data block 56 , In general, once a data block corresponding to a particular sequence number has been acknowledged, that data block is written to the logical storage device. This is in 2 with a data block 58 which corresponds to write entries associated with the sequence number N-2 (that is, two before the current sequence number used in conjunction with the write entries made by the host 22 to the local storage device 26 be transmitted). In 2 is the data block 58 shown to be a data item 62 which represents a disk memory for the remote storage device 26 , Thus, the remote storage device receives 26 the data block 56 and collects it according to the sequence number N-1, while the data block 58 , which corresponds to the previous sequence number (N-2), to the disk memory of the remote storage device 26 written by the data element 62 is illustrated. In some embodiments, data is for the block 58 marked for writing (but not necessarily written directly) while the data for the block 56 not marked.

Thus, during operation, the host writes 22 Data about the local storage device 24 local to the data element 51 be stored and in the block 52 are collected. Once all the data has been accumulated for a particular sequence number (which has already been described), the local storage device increments 24 the sequence number. Data from the block 54 that correspond to a sequence number that is one less than the current sequence number are read by the local storage device 24 to the remote storage device 26 over the connecting link 29 transfer. The block 58 corresponds to the data for a sequence number generated by the local storage device 24 which sent a message to the remote storage device 26 sends. Data from the block 58 become the disk storage of the remote storage device 26 written down.

It should be noted that the write entries within a particular one of the blocks 52 . 54 . 56 . 58 are not necessarily ordered. However, as more will be described in detail, each write entry for the block 58 corresponding to the sequence number N-2 before the beginning of any write entries for the blocks 54 . 56 started, according to the sequence number N - 1. In addition, each write entry for the blocks 54 . 56 corresponding to the sequence number N-1 before the beginning of any write entries for the block 52 started, which correspond to the sequence number N. Thus, in case of a communication error or failure between the local storage device 24 and the remote storage device 26 the remote storage device 26 just finish writing the last confirmed data block (the block 58 in the example of 2 ) and can ensure that the state of the data at the remote storage device 29 arranged in a sense that the data element 62 contains all the write entries that started before a specific time and does not contain any write entries that started after that time. Thus, R2 contains a timing copy of R1 and it is possible to recover a consistent image from the R2 device.

According to 3 illustrates a diagram 70 Elements used to construct and maintain the blocks 52 . 54 be used. A standard logical device 72 Contains data by the host 22 be written and the data element 51 from 1 match, and also the plates 33a - 33c from 1 , The standard logical device 72 Contains data by the host 22 to the local storage device 24 were written.

Two linked lists of pointers 74 . 76 become in connection with the standard logical device 72 used. The linked lists 74 . 76 correspond to the data, for example, in the memory 37 the local storage device 24 can be stored. The linked list 74 contains a variety of pointers 81 - 85 each one to a slot of a cache memory 88 shows, in conjunction with the local storage device 24 is used. Similarly, the linked list contains 76 a variety of pointers 91 - 95 , each of which accesses a slot of the cache memory 88 shows. In some embodiments, the cache memory may 88 in the store 37 the local storage device 24 be provided. The cache memory 88 contains a variety of cache slots 102 - 104 associated with write entries to the standard logical device 72 can be used and at the same time in conjunction with the linked lists 74 . 76 can be used.

Each of the linked or linked lists 74 . 76 can for one of the data blocks 52 . 54 so that, for example, the linked or linked list 74 the data block 52 for the sequence number N, while the linked list 76 the data block 54 for the sequence number N - 1. If so, data through the host 22 to the local storage device 24 are written, the data becomes the cache memory 88 and in some cases (to be described) will become a suitable pointer to the linked list 74 generated. It should be noted that data from the cache memory 88 until the data is removed to the standard logical device 72 have been mined and no longer on the data by one of the pointers 81 - 85 the linked list 74 are displayed as will be described.

In one embodiment described herein, one of the linked lists becomes 74 . 76 accepted as "active" while the other is assumed to be "inactive". Thus, for example, if the sequence number N is even, the linked list 74 be active while the linked list 76 is inactive. The active one of the linked lists 74 . 75 handles the write entries from the host 22 while the inactive list of linked lists 74 . 76 the data matches that of the local storage device 24 to the remote storage device 26 be transmitted.

While the data is being hosted by the host 22 using the Active One of the Chained Lists 74 . 76 are collected (for the sequence number N), the data, which is the inactive one of the linked lists 74 . 76 correspond (for the earlier sequence number N-1) from the local storage device 24 to the remote storage device 26 transfer. The RAs 30a - 30c use the linked lists 74 . 76 to determine that the data from the local storage device 24 to the remote storage device 26 be transmitted. Once the data is given to a particular one of the pointers in one of the linked lists 74 . 76 correspond to the remote storage device 26 The special One of the pointers may be sent from the appropriate one of the linked lists 74 . 76 be removed. Additionally, data may also be for removal from the cache memory 88 (i.e., the slot may be returned to a pool of unused later slots) provided that the data in the slot is not otherwise needed for some other purpose (e.g. standard logical device 72 to be read out). A mechanism can be used to ensure that data is not cached 88 be removed until all devices no longer use the data. Such a mechanism is for example in the U.S. Patent No. 5,537,568 U.S. Patent Application No. 09 / 850,551, filed Jul. 7, 2001, both of which are incorporated herein by reference.

According to 4 contains a slot 120 similar to the one of the slots 102 - 104 of the cache memory 88 a head section 122 and dates 124 , The header or header 122 Corresponds to additional information provided by the system for managing the slot 120 be used. The data 124 are the corresponding data from the disk that (temporarily) in the slot 120 are stored. The information in the header section 122 include pointer back to disk, timestamp and so on.

The head section 122 also contains a cache stamp 126 used in conjunction with the system described herein. In one embodiment, the cache stamp includes 126 eight bytes. Two of the bytes are a "password" indicating that the slot 120 is used by the system described here. In other embodiments, the password may consist of one byte while the following byte is used for a pad. As will be described, the two bytes of the password (or one byte as the case may be) equal to a certain value indicate that the slot 120 by at least one entry of the linked lists 74 . 76 will be shown. The password, which is equal to a certain value, indicates that on the slot 120 not by an entry of the linked lists 74 . 76 will be shown. The use of the password will be described in more detail.

The cache stamp 126 contains a two-byte field containing the sequence number (for example, N, N-1, N-2, and so on) of the data 124 of the slot 120 displays. As will be described, the sequence number field of the cache stamp 126 used to simplify the processing described herein. The remaining four Bytes of the cache stamp 126 can be used for a pointer, as will be described. Of course, the two bytes are the sequence number and the four bytes of the pointer are valid only if the password is equal to a certain value indicating that the slot 120 by at least one entry in one of the lists 74 . 76 will be shown.

According to 5 illustrates a flowchart 140 Steps through the HA 28 in conjunction with a host 22 which performs a write operation. If, of course, the host 22 performs a write operation, processing is performed to handle the write entry in a normal manner regardless of whether data exists in a part of a R1 / R2 RDF group. For example, if the host 22 Writing data for a portion of the disk writes to the cache slot that may be read out to the disk. The cache slot may either be a new cache slot or may be an existing cache slot that has been created in connection with an earlier read and / or write to the same track.

Processing begins with a first step 142 in which a slot is locked according to a write entry. In the embodiment illustrated here, each corresponds to the slots 102 - 104 of the cache memory 88 a trace of data in the standard logical device 72 , Locking the slot at the step 142 prevents additional processing, so that they take place from the relevant slot during processing by the HA 28 according to the steps of the flowchart 140 ,

Following the step 142 step one step 144 on, in which a value for N, that is set for the sequence number. As will be explained later, the value for the sequence number used in the step 144 is maintained throughout the write operation maintained by the HA 28 is performed while the slot is blocked. As will be explained, the sequence number is assigned to each write to one of the data blocks 52 . 54 to which the write or write entry belongs. Write entries made by the host 22 are executed are assigned to the current sequence number. It is advantageous that a single write operation will maintain the same sequence number throughout.

Subsequent to step S144, a test step follows 146 in which it is determined if the password field of the cache slot is valid. As discussed above, in the system described herein, the password field is set to a predetermined value to indicate that the cache slot is already in one of the linked lists 74 . 76 is included. If in the test step 146 it is determined that the password field is not valid (indicating that the slot is a new slot and that there are no pointers from the list 74 . 76 pointing to the slot), the control proceeds from the step 146 to the step 148 in which the cache stamp of the new slot is set by setting the password to the predetermined value, the sequence number field is set to N, and the pointer field is set to zero. In other embodiments, the pointer field may be adjusted to point to the slot itself.

Following on the step 148 follows a step 152 in which a pointer to the new slot to the active one list of pointer lists 74 . 76 is added. In the embodiment described here, the lists exist 74 . 76 of circular doubly linked lists and the new pointer is added to the circular doubly linked list in a conventional manner. Of course, other suitable data structures may be used to manage the lists 74 . 76 be used. Following on the step 152 follows a step 154 in which flags are set. At the step 154 the RDF_WP flag (the RDF write pending flag) is set to indicate that the slot is to the remote storage device 26 must be transmitted using RDF. In addition, at the step 154 set the IN_CACHE flag to indicate that the slot is the default logical device 172 must be read. Following on the step 154 follows a step 156 in which the host 22 and HA 28 data to be written to the slot. Following on the step 156 follows a step 158 in which the slot is unlocked. Following on the step 158 then the processing is finished.

If in the test step 146 it is determined that the password field of the slot is valid (it is indicated that already on the slot by at least one pointer of the lists 74 . 76 is shown), the control of the step 146 to a test step 162 in which it is then determined whether the sequence number field of the slot is equal to the current sequence number N. It should be noted that there are two valid possibilities for the sequence number field of a slot, with a valid password. It is possible for the sequence number field to be equal to N, as the current sequence number. This occurs when the slot corresponds to an earlier write with the sequence number N. The other possibility for the sequence number field is that it is equal to N-1. This occurs when the slot is in an earlier write to the sequence number N - 1 corresponds. Any other value for the sequence number field is invalid. Thus, for some embodiments, an error / validity checking process may be possible at the step 162 or possibly perform an error / validity check according to a separate step. Such an error may be handled in a suitable manner, which may also include delivering a message to a user.

If at the step 162 it is determined that the value in the sequence number field of the slot is equal to the current sequence number N, no special processing is required and control is then taken from the step 162 to the step 165 described above, wherein the data is written to the slot. In the other case, if the value of the sequence number field is equal to N - 1 (the only other valid value), then control is taken from the step 162 to a step 164 transfer, in which a new slot is obtained. The one at the step 164 The new slot obtained can be used to store the data to be written.

Following on the step 164 follows a step 166 in which the data is copied from the old slot to the new slot that is at the step 164 were obtained. It should be noted that the copied data includes the RDF_WP flag which is at step 154 should have been set in the earlier write when the slot was first created. Following on the step 166 follows a step 168 in which the cache stamp for the new slot is set by setting the password field to the appropriate value, setting the sequence number field to the current sequence number N and setting the pointer field to point to the old slot. Following on the step 168 follows a step 172 in which a pointer is added to the new slot to the active a list of linked lists 74 . 76 , Following on the step 172 follows a step 156 which was explained above, in which the data is written to the slot, which in this case consists of the new slot.

According to 6 illustrates a flowchart 200 the step, in conjunction with the RAs 30a - 30c the inactive is a list of lists 72 . 74 is sampled to the RDF data from the local storage device 24 to the remote storage device 26 transferred to. As explained above, the inactive list of lists shows 72 . 74 on slots corresponding to the N-1 cycle for the R1 device when the N cycle to the R1 device is written by the host using the active list of the lists 72 . 74 ,

Processing begins with a first step 202 in which it is determined if any entries in the inactive list of lists 72 . 74 available. When data is sent, the corresponding entries from the inactive become a list of lists 72 . 74 away. In addition, new write entries to the active list of the lists 72 . 74 delivered and not general to the inactive a list of lists 72 . 74 , It is thus possible (and also desirable, as will be described) that the inactive list of lists 72 . 74 does not contain data at any given time. If at the step 202 is determined that no data should be sent, then the inactive list of lists 72 . 74 continuously polled until data becomes available. Data for transmission is available in connection with a cycle switch (which has already been explained) in which the inactive list of lists 72 . 74 to an active list of lists 72 . 74 and vice versa.

If at the step 202 it is determined that data is available for transmission, the control of the step 202 to a step 204 in which the slot is verified as correct. The processing involved in the step 204 is executed, consists of an optional "health check" which may include a verification that the password field is correct and may contain a verification that the sequence number field is correct. If there are incorrect (unexpected) data in the slot, error processing may be performed which may include informing a user of the error and possible debugging processing.

Following on the step 204 follows a step 212 where data is sent via RDF in a conventional manner. In the embodiment described here, not the entire slot is sent. Rather, only records within the slot that contain the appropriate mirror bit set (indicating records that have changed) are sent to the remote storage device 26 Posted. In other embodiments, however, it may be possible to send the entire slot content, provided that the remote storage device 26 only writes data corresponding to the records having a suitable mirror bit set and ignores other data for a track that may or may not be valid. Following on the step 212 follows a test step 214 in which it is determined whether data that has been sent has been confirmed by the R2 device. If not, the data is retransmitted as indicated by the flow from the step 214 back to the step 212 , In other embodiments, a different more involved Processing can be used to send data and acknowledge receipt. Such processings may include error reporting and alternative processing which, after a certain number of attempts, will send the data which failed.

If in the test step 214 Once it has been determined that the data has been sent successfully, control passes from the step 214 to a step 216 to clear the RDF_WP flag (because the data was sent successfully via RDF). Following on the step 216 follows a test step 218 in which it is determined whether the slot consists of a duplicate slot created in connection with a write to a slot which already contains an existing entry in the inactive list of the lists 72 . 74 having. This option was above in connection with the steps 162 . 164 . 166 . 168 . 172 explained. If at the step 218 it is determined that the slot consists of a duplicate slot, control passes from the step 218 to a step 222 in which the slot is returned to the pool of available slots (to be reused). In addition, the slot may also be aged (or may include any other suitable mechanism used therewith) to realize immediate reuse before other slots, since the data provided in the slot will not be valid for any other purposes are. Following on the step 222 or on the step 218 if the slot does not consist of a duplicate slot, a step follows 224 in which the password field of the slot header section is released, so that when the slot is reused, the test at the step 146 from 5 properly classifies the slot as a new slot.

Following on the step 224 follows a step 226 in which the entry in the inactive list of lists 72 . 74 Will get removed. Following on the step 226 the control will go back to the step 202 described above, which determines whether there are additional entries in the inactive list of lists 72 . 74 are present according to the data to be transmitted.

According to 7 illustrates a diagram 240 the generation and manipulation of data packets 56 . 58 from the remote storage device 26 be used. The data passing through the remote storage device over the link 29 are received, become a cache memory 242 the remote storage device 26 delivered. The cache 242 For example, in the memory 38 the remote storage device 26 be provided. The cache 242 contains a variety of cache slots 244 - 246 each of which is on a track of the standard logical storage device 252 can be pictured. The cache 242 is similar to the cache 88 from 3 and may include that to the standard logical storage device 252 the remote storage device 26 be read out. The standard logical storage device 252 corresponds to the data element 62 which is in 2 shown and the plates 34a - 34c , in the 1 are shown.

The remote storage device 26 also contains a pair of only virtual cache devices 254 . 256 , The only virtual cache devices 254 . 256 correspond device tables, for example, in the memory 38 the remote storage device 26 can be stored. Each track entry of the tables from each of the only virtual cache devices 254 . 256 shows a trace of the standard logical device 252 or points to a slot of the cache memory 242 , The only virtual cache devices are described in co-pending US Patent Application entitled "CACHE-ONLY VIRTUAL DEVICES" filed Mar. 25, 2003, Serial No. 10 / 396,800, which is incorporated herein by reference in its entirety.

The variety of cache slots 244 - 246 can be used in conjunction with write entries to the standard logical device 252 and at the same time in connection with the only virtual cache devices 254 . 256 be used. In one embodiment described herein, each track table entry includes only virtual cache devices 254 . 256 a zero to indicate that the data for that track is on a corresponding track of the standard logical device 252 are stored. Otherwise, an entry in the track table for each of the only virtual cache devices 254 . 256 a pointer to one of the slots 244 - 246 in the cache 242 ,

Each of the only virtual cache devices 254 . 256 corresponds to one of the data packets 56 . 58 , Thus, for example, the only virtual cache device 254 the data packet 56 while the only virtual cache device 256 the data packet 58 can correspond. In one embodiment described herein, one of the only virtual cache devices 254 . 256 while the other is considered one of the only virtual cache devices 254 . 256 can be considered as "inactive". The inactive device of the only virtual cache devices 254 . 256 can match the data provided by the local storage device 24 be received (that is the package 56 ) while the active device is the only virtual cache devices 254 . 256 may correspond to the data corresponding to the standard logical device 252 back to be saved (written).

The data from the logical storage device 24 passing over the link 29 can be received in one of the slots 244 - 246 of the cache 242 to be placed. A corresponding pointer of the inactive device of the only virtual cache devices 254 . 256 can be set to point to the received data. Subsequent data having the same sequence number can be processed in a similar manner. At some point, the local storage device will provide 24 a message that confirms all sent data using the same sequence number. Once the data for a particular sequence number has been acknowledged, the one becomes the only virtual cache device 254 . 256 active and vice versa. At this point, data from the now-active one device becomes the only virtual cache devices 254 . 256 to the standard logical device 252 while the inactive device is the only virtual cache device 254 . 256 is used to receive new data (with a new sequence number) from the local storage device 24 to the remote storage device 26 be sent.

When data from the active device of the only virtual cache devices 254 . 256 can be removed (which will be explained in more detail), the corresponding entry in the active device of the only virtual cache devices 254 . 256 set to zero. Additionally, data may be from the cache 244 also be removed (that is, the slot is returned to the pool of free slots for later use) provided that the data in the slot is not otherwise needed for another purpose (for example, to read the standard logical device 252 ). A mechanism can be used to ensure that data is from the cache 242 can not be removed until all mirrors (including only virtual cache devices 254 . 256 ) no longer use the data. Such a mechanism is for example in the U.S. Patent No. 5,537,568 and U.S. Patent Application No. 09 / 850,551, filed July 7, 2001, both of which are incorporated herein by reference.

In some embodiments discussed herein, the remote storage device may 26 the linked lists 258 . 262 similar to the lists 74 . 76 maintained by the local storage device 24 be used. The lists 258 . 262 may contain information about the slots of the corresponding single cache devices 254 . 256 identify which have not been modified using one of the lists 258 . 262 one of the only virtual cache devices 254 . 256 corresponds and the other of the lists 258 . 262 the other is an apparatus of only virtual cache devices 254 . 256 equivalent. As has already been explained, the lists 258 . 262 to be used for restoring data from the cache only virtual devices 254 . 256 to the standard logical device 252 to simplify.

According to 8th illustrates a flowchart 270 the steps taken by the remote storage device 26 in connection with the processing of data for a sequence number confirmation performed by the local storage device 24 to the remote storage device 26 is sent. As already explained, the local storage device increments 24 periodically the sequence numbers. When this happens, the local storage device stops 24 sending all the data for the previous sequence number and then sending an acknowledgment message for the previous sequence number.

Processing begins with a first step 272 in which the confirmation is received. Following on the step 272 follows a test step 274 in which it is determined whether the active device is the only virtual cache device 254 . 256 the remote storage device 26 is empty. As already explained, the inactive device becomes the only virtual cache device 254 . 256 the remote storage device 26 used data from the local storage device 24 while the active device is the only virtual cache device 254 . 256 for the standard logical device 252 is restored.

If in the test step 274 it is determined that the active device is the only virtual cache device 254 . 256 is not empty, the control of the test step 274 to a step 276 in which the restoration for the active device of the only virtual cache devices 254 . 256 is completed before any further processing is performed. The restoration of data from the active device of the cache only virtual devices 254 . 256 will be described in more detail later. It is advantageous that the active device be the only virtual cache devices 254 . 256 is empty before the acknowledgment is handled and data recovery begins for the next sequence number.

Following on the step 276 or following the step 274 when the active device is the only virtual cache device 254 . 256 is determined to be empty, followed by a step 278 in which the active device is the only virtual cache devices 254 . 256 is made inactive. Following on the step 278 follows a step 282 in which the formerly inactive device is the only virtual cache device 254 . 256 (ie the one that precedes the execution of the step 278 was inactive) is made active. Switching the active and inactive only virtual cache devices 254 . 256 at the steps 278 . 282 forms a preparation for the now inactive (and empty) device of the only virtual cache devices 254 . 256 with the reception of data from the logical storage device 24 to start for the next sequence number.

Following on the step 282 follows a step 284 in which the active device is the only virtual cache devices 254 . 256 to the standard logical device 252 the remote storage device 26 is restored. Restoring the active device of the only virtual cache devices 254 . 256 to the standard logical device 252 will be described in more detail below. It should be noted, however, that in some embodiments, the recovery process is begun but not necessarily completed at the step 284 , Following on the step 284 follows a step 286 in which it is confirmed that what is available from the local storage device 24 to the remote storage device 26 is sent back to the local storage device 24 so that the local storage device 24 informed that the confirmation was successful. Following on the step 286 then the processing is completed.

According to 9 illustrates a flowchart 300 more in detail the steps 276 . 284 from 8th wherein the remote storage device 26 the active device of the only virtual cache devices 254 . 256 restores. Processing begins with a first step 302 in which a pointer is adjusted to the first slot of the active device of the only virtual cache devices 254 . 256 to show. The pointer is used to move through each track table entry of the active virtual cache device only device 254 . 256 pass through (iterate), each of which is processed individually. Following the step 302 follows a test step 304 in which it is determined whether the track is the active device of the only virtual cache devices 254 . 256 which is processed on the standard logical device 252 shows. If so, then there is nothing to restore. Otherwise, the control of the step 304 to a step 306 in which the corresponding slot of the active device of the only virtual cache devices 254 . 256 is locked.

Following on the step 306 follows a test step 308 in which it is determined whether the corresponding slot of the standard logical device 252 already in the cache of the remote storage device 26 is available. If so, control is from the test step 308 to a step 312 in which the slot of the standard logical device is locked or blocked. Following the step 312 follows a step 314 in which the data is from the active device of the only virtual cache devices 254 . 256 with the data in the cache for the standard logical device 252 be mixed. Mixing the data at the step 314 involves overwriting the data for the standard logical device with the new active device data of the cache only virtual devices 254 . 256 , It should be noted that in the embodiments that accommodate the recording of value flags, it may be possible to keep the new records from the active device of the cache only virtual devices 254 . 256 OR to be linked with the recordings of the standard logical device 252 in the cache. That is, if the records are interleaved, then it is only necessary to keep the records from the active cache only virtual device 254 . 256 to be used, and then the records to the cache slot of the standard logical device 252 to deliver. Following on the step 314 follows a step 316 in which the slot of the standard logical device 252 is unlocked. Following the step 316 follows a step 318 in which the slot is the active device of the only virtual cache devices 254 . 256 which is being processed is also unlocked.

If in the test step 308 it is determined that the corresponding slot of the standard logical device 252 is not in the cache, control passes from the test step 308 to a step 322 in which the track entry for the slot of the standard logical device 252 is changed to indicate that the slot of the standard logical device 252 is in the cache (for example, an IN_CACHE flag can be set) and does not need to be read out. As will be explained, in some embodiments, only records of the track that have a suitable mirror bit set need to be read out. Following on the step 322 follows a step 324 , at the flag for the track can be set to indicate that the data for the track is in the cache.

Following the step 324 follows a step 326 in which the slot pointer for the standard logical device 252 is changed to point to the slot in the cache. Following the step 326 follows a test step 328 in which determines if the steps 322 . 324 . 326 executed operations were successful. In some cases, a single operation, called a "compare and relocate" operation, can be used to complete the steps 322 . 324 . 326 perform. If these operations are unsuccessful for any reason, control will be taken from the step 328 back to the step 308 to recheck if the corresponding lane of the standard logical device 252 is in the cache. In the other case, if in the test step 328 it is determined that the previous operations were successful, control of the test step 328 to the step 318 transferred, which was explained above.

Following the step 318 follows a test step 332 in which it is determined whether the active device cache slot is the only virtual cache device 254 . 256 (which is being restored) is still being used. In some cases, it is possible for the active device slot to be the only virtual cache device 254 . 256 is used by another mirror. When it is determined, in the test step 332 in that the slot of the only virtual cache device is not used by another mirror, the control of the test step 332 to a step 334 in which the slot is released for use by other processes (eg, retransmitted back into the pool of available slots, as already explained). Following the step 334 follows a step 336 to point to the next slot, to the next slot of the active cache only virtual device 254 . 256 to process. It should be noted that the step 336 also from the test step 332 is reached when at the step 332 it is determined that the active device is the only virtual cache device 254 . 256 is still used by any other mirror. It should also be noted that the step 336 from the test step 304 can be achieved when at the step 304 it is determined that, for the slot being processed, the active device is the only virtual cache device 254 . 256 to the standard logical device 252 shows. Following the step 336 follows a test step 338 in which it is determined whether multiple slots of the active device of the only virtual cache devices 254 . 256 are processed. If this is not the case, the processing is completed. Otherwise, or in the other case, the control of the test step 338 back to the step 304 transfer.

In another embodiment, it is possible to construct lists of modified slots for the received data block 56 corresponding to the N-1 cycle at the remote storage device 26 like the lists 258 . 262 , in the 7 are shown. When the data is received, the remote storage device constructs 26 a linked list of modified slots. The lists that are constructed may be circular, linear (with zero termination), or designed according to any suitable design. The lists may then be used to identify the active device of the cache only virtual devices 254 . 256 restore.

The flowchart 300 from 9 shows two alternative paths 342 . 344 illustrating the operation of the embodiments using a list of modified slots. At the step 302 For example, a pointer (used for the iteration through the modified slot list) is made to point to the first element of the list. Following the step 302 follows a step 306 that's about the alternative path 342 can be achieved. In the embodiments using the lists of modified slots, the test step becomes 304 not needed because there are no slots on the list to the standard logical device 252 should show.

Following the step 306 processing continues in the manner explained above according to the previous embodiment, except that step 336 a traversal of the modified slots list and not a pointing to the next slot in the COVD. Similarly, in the test at step 338 determines if the pointer is at the end of the list (or back to the beginning in the case of a circular linked list). Even if at the step 338 it is determined that there are more slots that need to be processed, control of the step 338 to the step 306 transfer, as indicated by the alternative path 344 is illustrated. As stated above, in embodiments using a list of modified slots, step 304 be eliminated.

According to 10 illustrates a flowchart 350 the steps involved in connection with the logical storage device 24 be executed while increasing the sequence number. The Processing starts with a first step 352 in which the logical storage device 24 wait at least M seconds before increasing the sequence number. In one embodiment discussed herein, M has the size of thirty, but M may also have some other number. Large values for M increase the amount of data that can be lost when communicating between the storage devices 24 . 26 is interrupted. However, smaller values for M increase the total amount of overhead caused by more frequent incrementing of the sequence number.

Following on the step 352 follows a test step 354 in which it is determined if all the HAs of the local storage device 24 have set a bit indicating that the HAs have completed all their I / Os for an earlier sequence number. As the sequence number changes, each of the HAs perceives the change and sets a bit indicating that all I / Os of the previous sequence number have been completed. For example, if the sequence number changes from N-1 to N, a HA sets the bit when the HA has completed all the I / Os for the sequence number N-1. It should be noted that in some cases, a single I / O for a HA may take a long time and may still be running even after the sequence number has been changed. It should also be noted that for some systems a different mechanism can be used to determine if all the HAs have completed their N-1 I / Os. The alternative mechanism may be to check the device tables in the memory 37 contain.

If in the test step 354 it is determined that the I / Os have been completed from the previous sequence number, the control of the step 354 to a test step 356 in which it is determined whether the active list of lists 74 . 76 is empty. It should be noted that a sequence number switching does not have to be performed until all data corresponding to the inactive list of lists 74 . 76 completely from the local storage device 24 to the remote storage device 26 have been transmitted using the RDF protocol. If once the active list of lists 74 . 76 has been determined to be empty, the control of the step 356 to a step 358 in which a confirmation for the earlier sequence number from the local storage device 24 to the remote storage device 26 is sent. As explained above, the remote storage device operates 26 receiving a confirmation message for a particular sequence number that the remote storage device 26 restores the data according to the sequence number.

Following the step 358 follows a step 362 in which copying the data for the inactive list of lists 74 . 76 is suspended. As already explained, the inactive list of lists is sampled to receive corresponding data from the local storage device 24 to the remote storage device 26 to send. It is useful to suspend copying the data until the sequence count switch is completed. In one embodiment described herein, exposure is provided by sending a message to the RAs 30a - 30c is sent. However, it will be apparent to those skilled in the art that in embodiments using other components to facilitate the transmission of data using the system described herein, the suspension of copying may be provided by sending appropriate messages / commands to the other components.

Following on the step 362 follows a step 364 in which the sequence number is incremented. Following the step 364 follows a step 366 in which the bits for the HAs used in the test step 354 all are cleared so that the bits can be re-set in conjunction with incrementing the sequence number. Following the step 366 follows a test step 372 in which it is determined whether the remote storage device 26 the actuation message, ie at the step 358 is sent, confirmed or acknowledged. The confirmation of the confirmation message or promise message has been discussed above in connection with 8th explained. Once it has been determined that the remote storage device 26 the confirmation message or promise message that is in the step 358 is sent, the control of the step 372 to a step 374 transferred, in which the suspension of copying, which in the step 362 is provided, is deleted, so that the copying process can be resumed. Following the step 374 then the processing is completed. It should be noted that it is possible from the step 374 back to the step 352 to go to start a new cycle to continue incrementing the sequence number.

It is also possible to use COVDs on the R1 device to collect slots associated with the active data and the inactive data packets. In this case, as with the R2 device, one COVD may be assigned to the inactive sequence number and another COVD may be assigned to the active sequence number. This will be described below.

According to 11 illustrates a diagram 400 Elements used to construct and maintain the data packets 52 . 54 be used. A standard logical device 402 Contains data by the host 22 be written and the data element 51 from 2 and the plates 33a - 33c from 1 correspond. The standard logical device 402 Contains data by the host 22 to the logical storage device 24 written down.

There are two only virtual cache devices 404 . 406 in conjunction with the standard logical device 402 used. The only virtual cache devices 404 . 406 corresponding device tables can, for example, in the memory 37 the local storage device 42 get saved. Each track entry of the tables from each of the only virtual cache devices 404 . 406 indicates either a trace of the standard logical device 402 or points to a slot of a cache 408 that is connected to the logical storage device 24 is used. In some embodiments, the cache may be 408 in the store 37 the logical storage device 24 be provided.

The cache 408 contains a variety of cache slots 412 - 414 associated with write entries to the standard logical device 402 can be used and at the same time in connection with the only virtual cache devices 404 . 406 be used. In one embodiment, each track table entry includes only virtual cache devices 404 . 406 a zero to a corresponding track of the standard logical device 402 to show. Otherwise, it contains an entry in the track table for each of the virtual cache devices 404 . 406 a pointer to one of the slots 412 - 414 in the cache 408 ,

Each of the only virtual cache devices 404 . 406 can for any of the data packets 52 . 54 may be used, such that, for example, the only virtual cache device 404 the data packet 52 for the sequence number N, while the only virtual cache device 406 the data packet 54 for the sequence number N - 1. If so, data through the host 22 to the local storage device 24 are written, the data becomes the cache 408 and a suitable pointer to the cache only virtual device 404 is set. It should be noted that the data is not from the cache 408 be removed until the data is transferred to the standard logical device 402 have been read and that the data also by the only virtual cache device 404 be released as described.

In one embodiment described herein, one of the only virtual cache devices 404 . 406 considered "active" while the other is considered "inactive". Thus, for example, if the sequence number N is even, then the only virtual cache device may be 404 be active during the only virtual cache device 406 is inactive. The active device of the only virtual cache devices 404 . 406 handles the write entries from the host 22 while the inactive device is the only virtual cache device 404 . 406 the data matches that of the local storage device 24 to the remote storage device 26 be transmitted.

While the data is being hosted by the host 22 be written using the active device of the only virtual cache devices 404 . 406 (for the sequence number N), the data corresponding to the inactive device of the only virtual cache devices 404 . 406 (for the earlier sequence number N-1) from the local storage device 24 to the remote storage device 26 transfer. In this embodiment and related embodiments, the DAs handle 35a - 35c the local storage device scanning the inactive device of the only virtual cache devices 404 . 406 to copy requests to one or more of the RAs 30a - 30c to send data from the local storage device 24 to the remote storage device 26 to send. Thus, the steps 362 . 374 as described above in connection with suspending and resuming the copy operation, providing messages / commands to the DAs 35a - 35c contain.

Once data is sent to the remote storage device 26 can be sent, the corresponding entry in the inactive device of the only virtual cache devices 404 . 406 set to zero. In addition, data from the cache 408 (i.e., the slot is returned to the pool of slots for later use) if the data in the slot is not otherwise needed for some other purpose (for example, to the standard logical device) 402 to be read out). A mechanism can be used to ensure that the data is not from the cache 408 until all mirrors (including the only virtual cache devices 404 . 406 ) no longer use the data. Such a mechanism is for example in the U.S. Patent No. 5,537,568 , issued July 16, 1996, and in US Patent Application No. 09 / 850,551, filed July 7, 2001, which are incorporated herein by reference.

According to 12 illustrates a flowchart 440 Steps taken by HA 28 in conjunction with a host 22 performing a write operation in conjunction with embodiments wherein two COVDs are used by the R1 device to form the system described herein. Processing begins with a first step 442 in which a slot is locked according to a write entry. In one embodiment described herein, each corresponds to the slots 412 - 414 of the cache 408 a trace of data on the standard logical device 402 , The locking or blocking of the slot in the step 442 prevents additional processes from being executed at the relevant slot during processing by the HA 28 is performed according to the steps of the flowchart 440 ,

Following on the step 442 follows a step 444 in which a value for N, as the sequence number, is set. As with the embodiment which uses lists instead of COVDs on the R1 side, the value for the sequence number that is at step 444 is maintained throughout the write operation maintained by the HA 28 is performed while the slot is blocked. As will be explained, the sequence number is assigned to each write entry to be the one of the data packets 52 . 54 to set to which the write entry belongs. The by the host 22 executed write entries or writes are assigned to the current sequence number. It is advantageous that a single write operation will maintain the same sequence number throughout.

Following on the step 444 follows a test step 446 in which it is determined whether the inactive device is the only virtual cache device 404 . 406 already pointing at the slot, at the step 442 was locked (the slot where an operation takes place). This can occur if a write entry to the same slot was made when the sequence number was one less than the current sequence number. The data corresponding to the write entry for the previous sequence number can not yet go to the remote storage device 26 have been transferred.

If in the test step 446 it is determined that the inactive device is the only virtual cache device 404 . 406 does not point to the slot, the control of the test step 446 to another test step 448 in which it is determined whether the active device is the only virtual cache device 404 . 406 pointing to the slot. It is for the active device of the only virtual cache devices 404 . 406 possible to point to the slot if a previous write entry to the slot has occurred while the sequence number has remained the same as the current sequence number. If in the test step 448 it is determined that the active device is the only virtual cache device 404 . 406 does not point to the slot, the control of the test step 448 to a step 452 in which a new slot for the data is obtained. Following on the step 452 follows a step 454 in which the active device is the only virtual cache devices 404 . 406 is brought to point to the slot.

Following on the step 454 or following the step 448 when the active device is the only virtual cache device 404 . 406 pointing to the slot, a slot follows 456 to be set at the flags. At the step 456 the RDF_WP flag is set to indicate that the slot is to the remote storage device 26 must be transmitted using RDF. In addition, at the step 456 set the IN_CACHE flag to indicate that the slot is the default logical device 402 must be read. It should be noted that in some cases, when the active device is the only virtual cache device 404 . 406 already pointing to the slot (as in the step 448 is determined), it is possible that the RDF_WP and IN_CACHE flags are already set prior to the execution of the step 456 , However, the setting represents the setting of the flags in the step 456 sure that the flags are set correctly, regardless of their previous state.

Following on the step 456 follows a step 458 in which an indirect flag in the track table pointing to the slot is cleared, indicating that the relevant data is provided in the slot and not in a different slot, which is shown indirectly. Following the step 458 follows a step 462 where the data is hosted by the host 22 and by HA 28 written to be slotted to the slot. Following on the step 462 follows a step 464 in which the slot is unlocked. Following on the step 464 the processing is completed.

If in the test step 446 it is determined that the inactive device the only virtual cache devices 404 . 406 pointing to the slot, the control of the step 446 to a step 472 in which a new slot is obtained. The new slot at the step 472 can be obtained for the inactive device of the only virtual cache devices 404 . 406 used to effect the RDF transfer, while the old slot of the active cache only virtual device 404 . 406 can be assigned, as will be described below.

Following on the step 472 follows a step 474 in which data is copied from the old slot to the new slot which is at the step 472 has been obtained. Following on the step 474 follows a step 476 in which the indirect flag (explained above) is set to indicate that the spill table entry for the inactive device is the only virtual cache device 404 . 406 on the old slot shows, however, that the data is present in the new slot pointed to by the old slot. Thus, setting the indirect flag in the step 476 in that the track table is the inactive device of the only virtual cache devices 404 . 406 causes the track record entry to indicate that the data is present in the new slot.

Following on the step 476 follows a step 478 in which the mirror bits for the recordings in the new slot are set. Any local mirror bits that were copied when the data from the old slot to the new slot in the step 474 are deleted, since the purpose of the new slot is merely to effect the RDF transfer for the inactive device of the cache only virtual devices. The old slot is used to handle any local mirrors. Following on the step 478 follows a step 462 in which data is written to the slot. Following on the step 462 follows a step 464 in which the slot is unlocked. Following on the step 464 then the processing is completed.

According to 13 illustrates a flowchart 500 Steps associated with the local storage device 24 be executed, which is a data packet 54 to the remote storage device 26 sends. The sending essentially involves scanning the inactive device of the only virtual cache devices 404 . 406 on tracks written thereon during an earlier iteration when the inactive device is the only virtual cache device 404 . 406 was active. In this embodiment, the DAs key 35a - 35c the local storage device 24 the inactive device of the only virtual cache devices 404 . 406 to transfer the data for transmission to the remote storage device 26 to copy by one or more of the RAs 30a - 30c using the RDF protocol.

Processing begins with a first step 502 in which the first trace of the inactive device of the only virtual cache devices 404 . 406 is shown to begin the process of iteration through all the tracks. Following on the first step 502 follows a test step 504 which determines if the RDF_WP flag is set. As will be explained, the RDF_WP flag is used to indicate that a slot (track) contains data that needs to be transmitted over the RDF link. The RDF_WP flag that is set indicates that at least some data for the slot (lane) is to be transmitted or transmitted using RDF. In the embodiment discussed here, not the entire slot is transferred. Rather, only records within the slot where the appropriate mirror bits are set (indicating that the records have been changed) are transferred to the remote storage device 26 Posted. However, in other embodiments, it may be possible to send the entire slot content, provided that the remote storage device 26 only data is written according to the records where the appropriate mirror bits are set and ignore other data for the track that may or may not be valid.

If in the test step 504 it is determined that the cache slot being processed has a set RDF_WP flag, control is taken from the step 504 to a test step 505 in which it is determined whether the slot contains data or whether the slot consists of an indirect slot pointing to another slot containing the relevant data. In some cases, the slot may not contain data for the portion of the disc that corresponds to the slot. Instead, the slot may consist of an indirect slot pointing to another slot containing the data. If at the step 505 is determined that the slot consists of an indirect slot, the control of the step 505 to a step 506 in which the data (from the slot pointed to by the indirect slot) is obtained. Thus, if the slot consists of a direct slot, the data for transmission is stored in the slot by means of RDF, while if the slot consists of an indirect slot, the data to be transmitted by means of RDF is stored in another slot pointed to by the indirect slot.

Following on the step 506 or the step 505 if the slot is a direct slot, a step follows 507 in which the data which is sent (directly or indirectly from the slot) through one of the DAs 35a - 35c copied to from the local storage device 24 to the remote storage device 26 under Using the RDF protocol. Following on the step 507 follows a test step 508 in which it is determined whether the remote storage device 26 confirmed receipt of the data. If not, control is taken from the step 508 back to the step 507 to resend the data. In other embodiments, different or more involved processing may be used to send data and to confirm receipt thereof. Such processing may include an error report and alternative processing performed after a certain number of attempts have been made to transmit the data, which attempts have failed.

If in the test step 508 Once it has been determined that the data has been sent successfully, control passes from the step 508 to a step 512 to clear the RDF_WP flag (because the data was successfully sent using RDF). Following on the step 512 follows a step 514 in which appropriate mirror flags are cleared to indicate that at least the RDF mirror (R2) no longer needs the data. In one embodiment, each record represents part of a slot (track) and has individual mirror flags indicating which mirrors use the particular record. The R2 device is one of the mirrors for each of the records and it is the flags corresponding to the R2 device that are at the step 514 to be deleted.

Following on the step 514 follows a test step 516 in which it is determined whether any of the recordings of the track being processed have any other mirror flags set (or other mirror devices). If not, the controller proceeds from the step 516 to a step 518 in which the slot is released (that is no longer used). In some embodiments, unused slots are stored in a pool of slots that are available for use. It should be noted that if additional flags are still set, and indeed for some of the recordings of the slot, this may mean that the records have become the standard logical device 402 must be read out or used by any other mirror (including another R2 device). Following on the step 518 or following the step 516 if there are more mirror flags, a step follows 522 in which a pointer used to move through each track entry of the inactive device of the only virtual cache devices 404 . 406 to run through, is led to point to the next lane. Following the step 522 follows a test step 524 in which it is determined whether there are more traces of the inactive device of the only virtual cache devices 404 . 406 are processed. If this is not the case, the processing is completed. Otherwise, control is returned to the test step 504 transferred, which has been explained above. It should be noted that the step 522 also from the test step 504 can be reached if it is determined that the RDF_WP flag for the track being processed is not set.

According to 14 illustrates a diagram 700 a host 702 which is connected to a variety of local storage devices 703 - 705 is coupled. The diagram 700 also shows a variety of remote storage devices 706 - 708 , Although only three local storage devices 703 - 705 and three remote storage devices 706 - 708 in the diagram 700 can be extended to use any number of local and remote storage devices.

Each of the local storage devices 703 - 705 is with a corresponding one of the remote storage devices 706 - 708 coupled so that, for example, the local storage device 703 with the remote storage device 706 coupled, the local storage device 704 with the remote storage device 707 is coupled and the local storage device 705 with the remote storage device 708 is coupled. The local storage devices 703 - 705 and the remote storage devices 706 - 708 can be coupled together using ordered write mechanisms described herein such that, for example, the local storage device 703 to the remote storage device 706 can be coupled using the orderly writing mechanism. As will be further explained, the ordered writing mechanism enables data retrieval using the remote storage device in cases where the local storage device and / or the host stop working and / or lose data.

In some cases, the host may 702 run in a single application that simultaneously has more than one of the local storage devices 703 - 705 used. In such a case, the application may be configured to ensure that the application data is consistent (recoverable) at the local storage devices 703 - 705 if the host 702 stops work at any time and / or if any of the local storage devices 703 - 705 fails. However, since each of the ordered Write connections between the local storage devices 703 - 705 and the remote storage devices 706 - 708 Asynchronous from other connections, there is no assurance that data is consistent (and thus recoverable) for the application at the remote storage devices 706 - 708 , That is, although, for example, the data connection between the local storage device 703 and the remote storage device 706 (a first local / remote pair) is consistent and the data connection between the local storage device 704 the remote storage device 707 (a second local / remote pair 9 is consistent, it is not necessarily the case that the data at the remote storage devices 706 . 707 are always consistent if there is no synchronization between the first and second local / remote pairs.

For applications on the host 702 , at the same time a variety of local storage devices 703 - 705 It is desirable that the data be consistent and recoverable at the remote storage devices 706 - 708 , This can be achieved by using a mechanism in which the host 702 the cycle switching operation on each of the local storage devices 703 - 705 So that controls the data from the application running on the host 702 run, consistent and recoverable at the remote storage devices 706 - 708 , This functionality is provided by a special application on the host 702 which runs a variety of local storage devices 703 - 705 in a multi-box mode, as will be described in more detail below.

According to 15 contains a table 730 a variety of entries 732 - 734 , Each of the entries 732 - 734 corresponds to a single local / remote pair of storage devices, so that, for example, the entry 732 a pair according to the local storage device 703 and the remote storage device 706 can match the entry 733 the pair according to the local storage device 704 and the remote storage device 707 can match and the entry 734 the pair according to the local storage device 705 and the remote storage device 708 can correspond. Each of the entries 732 - 734 includes a plurality of fields, wherein a first field 736a - 736c a serial number of the corresponding local storage device represents a second field 738a - 738c a session number used by the multibox group, a third field 742a - 742c represents the serial number of the corresponding remote storage device of the local / remote pair, and a fourth field 744a - 744c the session number for the multi-box group. The table 730 is from the host 702 designed and maintained in conjunction with operation in the multi-box mode. In addition, the table 730 sent to each of the local storage devices and also sent to the remote storage devices that are part of the multi-box group. The table 730 can be used to facilitate recovery, as explained in greater detail below.

Different local / remote pairs can enter and leave the multi-box mode independently in any sequence at any one time. The host 702 manages the entry and exit of the local storage device / remote storage device (s) thereof into and out of the multi-box mode. This will be described in more detail below.

According to 16 illustrates a flowchart 750 the steps taken by the host 702 in connection with entering or leaving a local / remote pair in or out of the multi-box mode. Processing begins with a first step 752 in which the multi-box mode operation is temporarily suspended. The temporary suspension of multi-box operation in the step 752 is useful to simplify changes made in connection with the entry or exit of a remote / local pair in and out of the multi-box mode. Following on the step 752 follows a step 754 in which a table similar to the table 730 in 15 is modified to either add an entry or omit it as needed. Following on the step 754 follows a step 756 in which the modified table is transmitted to the local storage devices and to the remote storage devices of the multi-box group. Transferring the table at the step 756 simplifies recovery, as will be explained in more detail.

Following on the step 756 follows a step 758 in which a message is sent to the affected local storage device to provide the change. The local storage device may or may not configure itself to run in a multi-box mode, as described in more detail below. As will be explained in detail below, the local storage device handling ordered writes operates differently depending on whether or not it operates as part of a multi-box group. When the local storage device becomes a multi-box group is added, shows the message at the step 758 is sent to the local storage device to be added to a multi-box group so that the local storage device should configure itself to run in the multi-box mode. Alternatively, if the local storage device is removed from a multi-box group, it will show at the step 758 sent message to the local storage device that it has been removed from the multi-box group, so that the local storage device should configure itself so that it does not run in the multi-box mode.

Following the step 758 follows a test step 762 in which it is determined if a local / remote pair has been added to the multi-box group (as opposed to a removal). If so, control is from the test step 762 to a step 764 in which label values are sent to the local storage device which is added. The tag values are provided with the data being transferred from the local storage device to the remote storage device in a manner similar to providing the sequence numbers with the data. The tag values are controlled by the host and are set so that all the local / remote pairs send data having the same tag value during the same cycle. The use of label values is explained in more detail below. Following on the step 764 or following the step 762 If a new local / remote pair has not been added, a step follows 766 in which the multi-box operation is resumed. Following on the step 766 the processing is completed.

According to 17 illustrates a flowchart 780 Steps performed in conjunction with the host management cycle switch for multiple local / remote pairs running as a group in the multi-box mode. As will be discussed, the multi-box mode involves using host sync cycle switches for more than one remote / local pair to maintain data consistency among the remote storage devices. The cycle switching process is coordinated by the host and not generated internally by the logical storage devices. This will be explained in more detail below.

The processing according to the flowchart 780 begins with a test step 782 which determines if M seconds have elapsed. As with non-multi-box operation, cycle shifts do not occur earlier than every M seconds, where M is a number that is selected to optimize the various performance parameters. As the number M is increased, the amount of overhead associated with the shift decreases. However, increasing M will also increase the amount of data that can potentially be lost in connection with a failure or an error. In one embodiment discussed herein, M is chosen to be thirty seconds, although obviously other values for M can be used.

If in the test step 782 If it is determined that M seconds have not yet elapsed, control is returned to the step 782 to continue waiting until M seconds have elapsed. Once in the test step 782 it has been determined that M seconds have elapsed, the control of the step 782 to a step 784 in which the host polls all local storage devices in the multi-box group to determine if all the local / remote pairs are ready for switching. The local / remote pairs ready for switching will be explained in more detail below.

Following on the step 784 follows a test step 786 in which it is determined whether all the local / remote pairs are ready for switching. If not, control is returned to the step 784 to resume the request. In an embodiment described herein, it is only necessary to request the local / remote pairs that were previously not ready for switching because once a local / remote pair is ready for the switch, that pair remains so until the switching operation occurs.

Once in the test step 786 it has been determined that all the local / remote pairs in the multi-box group are ready for switching, the control of the step 786 on a step 788 in which an index variable N is set equal to one. The index variable N is used to iterate through all the local / remote pairs (that is, all the entries 732 - 734 the table 730 from 15 ). Following on the step 788 follows a test step 792 in which it is determined whether the index variable N is greater than the number of local / remote pairs in the multi-box group. If not, control is taken from the step 792 to a step 794 in which an open window for the Nth local storage device of the Nth pair is realized by the host sending a command (that is, a suitable system command) to the Nth local storage device. Opening the window for the Nth local storage device at the step 794 causes the Nth local storage device to suspend writes, so that any writing by a host that is not prior to opening the window in step 794 is not completed until the window is closed (which will be described below). Failure to complete a write operation prevents a second dependent write operation from occurring prior to completing the cycle switch. Any write entries that are in progress that were started before the window was opened can be completed before the window is closed.

Following on the step 794 follows a step 796 in which cycle switching is performed for the Nth local storage device. Performing the cycle switching operation in the step 796 involves sending a command from the host 702 to the Nth local storage device. The processing of the command from the host by the Nth local storage device will be explained in more detail below. A section of the processing involved in the step 796 may include the feature that the host provides new values for the labels associated with data. The tags will be explained in more detail. In an alternative embodiment, the operations involved in the steps 794 . 796 in the form of a single integrated step 797 Running what is illustrated by the box, the steps around 794 . 796 runs around.

Following on the step 796 follows a step 798 in which the index variable N is incremented. Following on the step 798 control is returned to the test step 792 to determine if the index variable N is greater than the number of local / remote pairs.

If in the test step 792 is determined that the index variable N is greater than the number of the local / remote pairs, the control of the test step 792 to a step 802 in which the index variable N is set to one. Following on the step 802 follows a test step 804 in which it is determined whether the index variable N is greater than the number of local / remote pairs. If not, control is taken from the step 804 to a step 806 in which the window for the Nth local storage device is closed. Closing the window according to the step 806 is executed by the host sending a command to the Nth local storage device to cause the Nth local storage device to resume write operations. Thus, any writes that are in progress and those by opening the window at step 794 are now completed and that after the execution of the step 806 , Following on the step 806 the control becomes a step 808 in which the index variable N is incremented. Following on the step 808 control is returned to the test step 804 to determine if the index variable N is greater than the number of local / remote pairs. If so, control is from the test step 804 back to the step 782 to begin processing for the next cycle shift.

According to 18 illustrates a flowchart 830 Steps performed by a local storage device in conjunction with the cycle switching operation. The flowchart 830 from 18 replaces the flowchart 350 from 10 in cases where the local storage device supports both the multi-box mode and the non-multi-box mode. That is flowchart 830 illustrates steps performed similar to those of the flowchart 350 from 10 to support the non-multi-box mode and also includes steps to support the multi-box mode.

The processing begins with a first test step 832 in which it is determined whether the local storage device is in the multi-box mode. It should be noted that the flowchart 750 from 16 one step 758 Fig. 2 shows where the host sends a message to the local storage device. The message at the step 758 is sent, the local storage device indicates whether or not the local storage device is in the multi-box mode. After receiving the message sent by the host at the step 758 is sent, the local storage device sets an internal variable to indicate whether or not the local storage device is in the multi-box mode. The internal variable can be at the test step 832 being checked.

If in the test step 832 it is determined that the local storage device is not in the multi-box mode, the control of the test step 832 to a step 834 to wait for M seconds to cycle. If the local storage device is not operating in the multi-box mode, then the local storage device controls its own cycle switching operation and thus performs the step 834 to wait M seconds before initiating the next cycle switch.

Following on the step 834 or following the step 832 When the local storage device is in the multi-box mode, a test step follows 836 in which it is determined whether all of the HAs of the local storage device have set a bit indicating that the HA completes all I / Os for an earlier sequence number to have. As the sequence number changes, each HAs perceives the change and sets a bit indicating that all I / Os of the previous sequence number have been completed. For example, if the sequence number changes from N-1 to N, a HA sets a bit if the HA has completed all the I / Os for the sequence number N-1. It should be noted that in some cases a single I / O for a HA may take a long time and may still be running, even after the sequence number has changed. It should also be noted that some systems may use a different mechanism to determine if all HAs have completed their N-1 I / Os. This other or different mechanism may include tester tables. Once in the test step 836 it has been determined that all HAs have set the appropriate bit, control is taken from the test step 836 to a step 888 in which it is determined whether an inactive data packet for the local storage device is empty. If in the test step 888 it is determined that the inactive packet is empty, control is taken from the step 888 to a step 899 in which the copying of the data from the local storage device to the remote storage device is suspended. It is advantageous to suspend copying the data until the sequence count switch is completed.

Following on the step 899 follow a test step 892 to determine if the local storage device is in the multi-box mode. If in the test step 892 is determined that the local storage device is in the multi-box mode, the control of the test step 892 to a test step 894 to determine if the active packet (chunk) of the corresponding remote storage device is empty. As explained more in detail below, the remote storage device sends a message to the local storage device when it has emptied its active data packet. In response to the message, the local storage device sets an internal variable that is at the test step 894 is checked.

Once in the test step 894 has been determined that the active data packet of the remote storage device is empty, the control of the test step 894 to a step 896 in which an internal variable is set at a local storage device, indicating that the local storage device is ready for switching cycles. As above in connection with the flow chart 780 from 17 The host performs requests at each local storage device to determine if each of the local storage devices is ready for the switching process. In response to the request provided by the host, the local storage device checks the internal variable involved in the step 896 has been set and returns the result to the host.

Following on the step 896 follows a test step 898 in which the local storage device waits to receive a command from the host to perform the cycle switching operation. As above in connection with the flow chart 780 from 17 has been explained, the host provides a command in conjunction with switching cycles to the local storage device when the local storage device is operating in the multi-box mode. Thus, the local storage device waits for the command at the step 898 which is reached only when the local storage device is operating in the multi-box mode.

Once the local storage device has received the switching command from the host, control is taken from the step 898 to a step 902 to send a confirmation message to the remote storage device. It should be noted that the step 902 also from the test step 892 is reached when in the test step 892 it is determined that the logical storage device is not in the multi-box mode. At the step 902 the local storage device sends a confirmation message to the remote storage device. In response to receiving a confirmation message for a particular sequence number, the remote storage device begins to recover the data according to the sequence number, as discussed above.

Following on the step 902 from follows a step 906 in which the sequence number is incremented and a new value for the tag (from the host) is stored. The sequence number has been explained above. The label or identifier is a label which is added to the local storage device at the step 764 is delivered and also at the step 796 as explained above. The tag or tag is used to simplify data recovery, as already stated.

Following on the step 906 follows a step 907 in which the completion of the cycle switching operation from the local storage device to the host is confirmed by sending a message from the local storage device to the host. In some embodiments, it is possible to do the step 907 for the execution depending on whether the local storage device is in the multi-box mode or not, because if the local storage device is not in the multi-box mode, the host is not is necessarily interested in when cycle shifts occur.

Following on the step 907 follows a step 908 in which the bits for the HAs used in the test step 836 are cleared so that the bits can be re-set in conjunction with incrementing the sequence number. Following on the step 908 follows a test step 912 in which it is determined whether the remote storage device has acknowledged the acknowledgment message. It should be noted that if the local / remote pair is operating in the multi-box mode and the active data packet of the remote storage device is in step 894 has been set as empty or determined, then the remote storage device should acknowledge the acknowledgment message almost immediately because the remote storage device is immediately ready for the cycle switch due to the fact that its active packet is already empty.

If in the test step 912 Once it has been determined that the acknowledgment message has been acknowledged by the remote storage device, control is taken from step 912 to a step 914 transferred, in which the suspension of the copying process, what in the step 899 has been realized, so that the copying operation can be resumed from the local storage device to the remote storage device. Following the step 914 then the processing is completed.

According to 19 illustrates a flowchart 940 Steps performed in conjunction with the RAs scanning inactive buffer stages to send RDF data from the local storage device to the remote storage device. The flowchart 940 from 19 is similar to the flowchart 200 from 6 and similar steps are given the same reference number. However, the flowchart contains 940 two additional steps 942 . 944 in the flowchart 200 from 6 can not be found. The additional steps 942 . 944 are used to simplify multi-box processing. After data at the step 212 have been sent, the control of the step 212 to a test step 942 in which it is determined whether the data sent is the last data in the inactive packet of the local storage device. If not, control is taken from the step 942 to the step 214 Transfer and processing continues, as discussed above in connection with the flowchart 200 from 6 has been explained. If, in the other case, in the test step 942 it is determined that the transmitted data is the last data of the packet, the control of the step 942 to the step 944 to send a special message from the local storage device to the remote storage device indicating that the last data has been sent. Following on the step 944 the control is on one step 214 Transfer and processing continues as described in connection with the flowchart 200 from 6 was explained above. In some embodiments, the steps 942 . 944 be performed by a separate process (and / or separate hardware device) that is different from the process and / or hardware device in which the data is being transferred.

According to 20 illustrates a flowchart 950 Steps performed in conjunction with the RAs scanning inactive buffer stages to transfer RDF data from the on-premises storage device to the remote storage device. The flowchart 950 from 20 is similar to the flowchart 500 from 13 and similar steps are provided with like reference numerals. However, the flowchart contains 950 an extra step 952 that in the flowchart 500 from 13 can not be found. The extra step 952 is used to simplify multi-box processing and is similar to the extra step 944 of the flowchart 940 from 19 , After it has been determined that in the test step 524 no more slots remain to be sent from the local storage device to the remote storage device, the control of the step 524 on the step 952 to send a special message from the local storage device to the remote storage device indicating that the last data for the packet has been sent. Following the step 952 the processing is completed.

According to 21 illustrates a flowchart 960 Steps performed on the remote storage device in conjunction with providing an indication that the active package of the remote storage device is empty. The flowchart 960 is similar to the flowchart 300 from 9 with the exception that the flowchart 960 a new step 962 which is executed after the active package of the remote storage device has been stored again. At the step 962 the remote storage device sends a message to the local storage device indicating that the active packet of the remote storage device is empty. After receiving the message at the step 962 is sent, the local storage device sets an internal variable indicating that the inactive buffer level of the remote storage device is empty. The local variable is associated with the test step 894 of the flowchart 830 from 18 tested, as explained above.

According to 22 illustrates a diagram 980 the host 702 , the local storage devices 703 - 705 and the remote storage devices 706 - 708 that in the diagram 700 from 14 are shown. The diagram 980 also contains a first alternative host 982 who is at the host 702 and the local storage devices 703 - 705 is coupled. The diagram 980 also contains a second alternative host 984 to the remote storage devices 706 - 708 is coupled. The alternative hosts 982 . 984 can also be used for data recovery or recovery, as described in more detail below.

If recovery of data at the remote site is required, recovery may be through the host 702 be executed, or by the host 982 provided that the links between the local storage devices 703 - 705 and the remote storage devices 706 - 708 continue to operate. If the links are not in operation, data recovery can be done using the second alternate host 984 be executed, which to the remote storage devices 706 - 708 is coupled. The second alternative host 984 may be in the same location as one or more of the remote storage devices 706 - 708 be provided. Alternatively, the second alternative host 984 even from all the remote storage devices 706 - 708 lie away. The table 730 which is passed through the system is accessed in conjunction with the data recovery to determine the members of the multi-box group.

According to 23 illustrates a flowchart 1000 Steps taken by each of the remote storage devices 706 - 708 in connection with the data recovery operation. The steps of the flowchart 1000 can pass through any of the remote storage devices 706 - 708 after receiving a signal or message indicating that data recovery or retrieval is required. In some embodiments, it may be possible for a remote storage device to automatically determine that data recovery is required using, for example, conventional criteria such as the length of time since the last write.

Processing begins with a first step 1002 in which the remote storage device terminates the re-storage of the active packet, as previously explained. Following on the step 1002 follows a test step 1004 in which it is determined whether the inactive packet of the remote storage device is complete (that is, all the data has been written thereto). It should be noted that the remote storage device may determine whether the inactive data packet is complete, using the message provided by the local storage device in the steps 944 . 952 is sent, as explained above. That is, when the local storage device receives the message at the step 944 or at the step 952 then the remote storage device may use the receipt of that message to confirm that the inactive data packet is complete.

If in the test step 1004 is determined that the inactive data packet of the remote storage device is not complete, the control of the test step 1004 to a step 1006 in which the data is deleted from the inactive data packet. No data recovery or retrieval is then performed using incomplete inactive data packets because the data there may be inconsistent with the corresponding active data packets. As a result, data recovery or retrieval is performed using the active data packets, and in some cases, the inactive data packets are complete. Following on the step 1006 then the processing is completed.

If in the test step 1004 it is determined that the inactive data packet is complete, the control of the step 1004 to the step 1008 in which the remote storage device is waiting for an intervention by the host. If an inactive data packet exists, one of the hosts must be 702 . 982 . 984 as appropriate, check the state of all the remote storage devices in the multi-box group to determine how to perform the recovery or recovery. This will be described in more detail below.

Following on the step 1008 follows a test step 1012 in which it is determined if the host has sent a command to all storage devices to wipe out the inactive data packet. If this is the case, the control of the step 1012 to the step 1006 to delete the inactive data packet. Following on the step 1006 then the processing is completed.

If in the test step 1002 it is determined that the host has issued a command to restore the entire inactive data packet, the control of the step 1012 to a step 1014 in which the inactive data packet is re-stored to the remote storage device. The restoring of the inactive data packet in the remote storage device involves making the inactive data packet an active data packet and then writing the active data packet to the disk, as already described. Following on the step 1014 then the processing is completed.

According to 24 illustrates a flowchart 1030 Steps in conjunction with one of the hosts 702 . 982 . 984 when determining whether erasure or restocking of each of the inactive data packets from each of the remote storage devices is to be performed. The one of the hosts 702 . 982 . 984 The re-storage communicates with the remote storage devices 706 - 708 to provide instructions there and to receive information from them using the labels or identifiers assigned by the host, as previously explained.

Processing begins with a first step 1032 in which it is determined whether any of the remote storage devices contains a complete inactive data block. If this is not the case, then no further processing is performed and the remote storage devices delete the incomplete data packets for themselves, as already explained above, without intervention of the host. Otherwise, the control of the test step 1032 to a test step 1034 in which the host determines if all the remote storage devices have inactive data packets completely. If so, control is from the test step 1034 to a test step 1036 in which it is determined whether all of the complete inactive data packets from all the remote storage devices have the same label number. As already explained, the tags are assigned by the host and are used by the system to identify data in a manner similar to the sequence number except that the tags or tags are controlled by the host. so that they have the same value for the same cycle.

If in the test step 1036 it is determined that all the remote storage devices have the same label or identifier for the inactive data packets, control is taken from the step 1036 to a step 1038 transferred, in which all the inactive data packets are stored again. Performing the step 1038 Ensures that all the remote storage devices have data from the same cycle. Following on the step 1038 then the processing is completed.

If in the test step 1034 it is determined that all of the inactive data packets are incomplete or if it is determined that at the step 1036 if all the inactive data packets that are complete do not have the same identifier or tag, the control becomes one step 1042 in which the host provides a command to the remote storage devices to re-store the complete inactive data packets having the lower tag count. For purposes of explanation, assume that the label numbers or label numbers are incremented so that a lower label number represents older data. By way of example, if a first remote storage device had a complete inactive record having a tag value of three and a second remote storage device had a complete inactive record or data packet with a tag value of four, the step would become 1042 the first remote memory device cause (but not the second) to re-store its inactive data packet. Following on the step 1042 follows a step 1044 in which the host provides commands to the remote storage devices to clear the entire inactive buffer stages having a higher tag count (e.g., the second remote storage device in the earlier example). Following on the step 1044 then the processing is completed.

Following the execution of the step 1044 Each of the remote storage devices contains data associated with the same tag value in the form of data for other of the remote storage devices. Consequently, then the recovered or retrieved data should be at the remote storage devices 706 - 708 be consistent.

Although the present invention has been illustrated in conjunction with various embodiments, embodiments modified for those skilled in the art will be apparent. Accordingly, the scope of the present invention is apparent from the following claims.

Claims (3)

Procedure for ordering data write entries, with the following steps: at least one of a group of primary storage devices receives a first plurality of data write entries that are initiated a first time; a transmit cycle switch is caused to a new send cycle for the at least one of the groups of primary storage devices, the first plurality of data write entries being associated with a particular send cycle for the at least one of the primary storage devices in the group; the at least one of the group of primary storage devices receives a second plurality of write entries after initialization of the transmit cycle switch, wherein all write entries of the second plurality of write entries are associated with a transmit cycle from the particular transmit cycle for the at least one of the primary storage devices is different and causes the second plurality of write entries after said first time; wherein the write entries to the group which started after the transmission cycle switching has been initialized are not completed until after the completion of the transmission cycle switching operation; and after completion of the transmit cycle switching operation, the at least one of the primary storage devices of the group initiates transfer of the first plurality of write entries to a corresponding secondary storage device and wherein causing the transmission cycle switching process includes: Waiting for a predetermined period of time; Determining if all the primary storage devices of the group of storage devices are ready for the switching operation; and Sending a first command to each of the primary storage devices of the group to effect a send cycle switch. The method of claim 1, wherein a host causes the transmission cycle switching operation. The method of claim 1, wherein each transmit cycle is associated with a particular tag value provided with each data packet.

Images