WO2007120437A2 - Metadata management system for an information dispersed storage system - Google Patents
Metadata management system for an information dispersed storage system Download PDFInfo
- Publication number
- WO2007120437A2 WO2007120437A2 PCT/US2007/007285 US2007007285W WO2007120437A2 WO 2007120437 A2 WO2007120437 A2 WO 2007120437A2 US 2007007285 W US2007007285 W US 2007007285W WO 2007120437 A2 WO2007120437 A2 WO 2007120437A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- metadata
- storage
- grid
- subsets
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
Definitions
- the present invention relates to a metadata management system and method for an information dispersal storage system which disperses information into a plurality of subsets or slices for storage in separate storage nodes coupled to a communication network and more particularly, to a metadata management system and method that can store and index metadata attributes of files, such as data source names, file size, last modification date and other data related information, of all file slices in all of the storage nodes coupled to the communication network in order to keep track of dispersed information and is also useful for dispersing data.
- metadata attributes of files such as data source names, file size, last modification date and other data related information
- the data may be stored across more than one storage device, such as a hard drive, or removable media, such as a magnetic tape or a so called "memory stick" as set forth in U.S. Pat. No. 6,128,277, hereby incorporated by reference, as well as for reasons relating to performance improvements or capacity limitations.
- a hard drive or removable media, such as a magnetic tape or a so called "memory stick" as set forth in U.S. Pat. No. 6,128,277, hereby incorporated by reference, as well as for reasons relating to performance improvements or capacity limitations.
- recent data in a database might be stored on a hard drive while older data that is less often used might be stored on a magnetic tape.
- Another example is storing data from a single file that would be too large to fit on a single hard drive on two hard drives.
- the data subset stored on each data storage device does not contain all of the original data, but does contain a generally continuous portion of the data that can be used to provide some usable information. For example, if the original data to be stored was the string of characters in the following sentence:
- the quick fox jumped (Stored on the first storage device) over the lazy brown dog. (Stored on the second storage device)
- each device is not a complete copy of the original data, but each of the data subsets stored on each device provides some usable information.
- bit pattern of data storage on a device is structured with additional values to represent file types, file systems and storage structures, such as hard drive sectors or memory segments.
- file types such as a hard drive
- storage structures such as hard drive sectors or memory segments.
- the techniques used to structure data in particular file types using particular file systems and particular storage structures are well known and allow individuals familiar with these techniques to identify the source data from the bit pattern on a physical media.
- the present invention relates to an information dispersal sytem in which original data to be stored is separated into a number of subsets or data "slices" in such a manner that the data in each subset is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data slices.
- These data slices are stored on separate storage devices (i.e., separate dataspaces) as a way of increasing privacy and security.
- a metadata management system stores and indexes user files across all of the storage nodes.
- the metadata management system includes a number of applications that run on the servers supporting these storage nodes and are responsible for controlling the metadata.
- Metadata is the information about the data, the data slices or data subsets and the way in which these data subsets are dispersed among different storage nodes running over the network. More particularly, metadata includes data source names, their size, last modification date, authentication information etc. This information is required to keep track of dispersed data subsets among all the nodes in the system. Every time new data subsets are stored and old ones are removed from the storage nodes, the metadata is updated.
- the metadata management system stores metadata for dispersed data where:
- the metadata is in a separate dataspace from the dispersed data. Accordingly, the metadata management system is able to manage the metadata in a manner that is computationally efficient relative to known systems in order to enable broad use of the invention using the types of computers generally used by businesses, consumers and other organizations currently.
- FIG. 1 is a block diagram of an exemplary data storage system in accordance with the present invention which illustrates how the original data is sliced into data subsets, coded and transmitted to a separate digital data storage device or node.
- FIG. 2 is similar to FIG. 1 but illustrates how the data subsets from all of the exemplary six nodes are retrieved and decoded to recreate the original data set.
- FIG. 3 is similar to FIG. 2 but illustrates a condition of a failure of one of the six digital data storage devices.
- FIG. 4 is similar FIG. 3 but for the condition of a failure of three of the six digital data storage devices.
- FIG. 5 is an examplary tablein accordance with the present invention that can be used to recreate data which has been stored on the exemplary six digital data storage devices.
- FIG. 6 is an exemplary table that lists the decode equations for an exemplary six node storage data storage system for a condition of two node outages
- FIG. 7 is is similar to FIG. 6 but for a condition with three node outages
- FIG. 8 is a table that lists all possible storage node outage states for an exemplary data storage system with nine storage nodes for a condition with two node outages.
- FIG. 9 is an exemplary diagram in accordance with the present invention which illustrates the various functional elements of a metadata management system for use with an information dispersal storage system in accordance with the present invention.
- FIG. 10 is an exemplary flow chart that shows the process for maintaining metadata for data stored on the dispersed data storage grid.
- FIG. 11 shows the essential metadata components that are used during user transactions and during user file set lookup.
- FIGS. 12 A and 12 B illustrate the operation of the system.
- the present invention relates to a billing system for an information dispersal storage systen or data storage system.
- the information dispersal storage system is illustrated and descrided in connection with FIGS. 1-8.
- a metadata management system for managing information dispersal on a grid which includes a plurality of storage nodes is illustrated in FIGS. 9-12.
- the original data is separated into a number of data “slices” or subsets.
- the amount of data in each slice is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data subsets.
- the system in accordance with the present invention "slices" the original data into data subsets and uses a coding algorithm on the data subsets to create coded data subsets.
- Each data subset and its corresponding coded subset may be transmitted separately across a communications network and stored in a separate storage node in an array of storage nodes.
- data subsets and coded subsets are retrieved from some or all of the storage nodes or communication channels, depending on the availability and performance of each storage node and each communication channel.
- the original data is recreated by applying a series of decoding algorithms to the retrieved data and coded data.
- unauthorized access to one or more data subsets only provides reduced or unusable information about the source data.
- the system codes and decodes data subsets in a manner that is computationally efficient relative to known systems in order to enable broad use of this method using the types of computers generally used by businesses, consumers and other organizations currently.
- the string is broken into segments that are n characters each, where n is chosen to provide the desired reliability and security characteristics while maintaining the desired level of computational efficiency — typically n would be selected to be below 100. In one embodiment, n may be chosen to be greater than four (4) so that each subset of the data contains less than, for example, 1 A of the original data, thus decreasing the recognizablity of each data subset.
- the coded data values are created by adding data values from a subset of the other data values in the original data set.
- c[x] is the xth coded data value in the segment array of coded data values d[x+l] is the value in the position 1 greater than x in a array of data values d[x+2] is the value in the position 2 greater than x in a array of data values d[x+4] is the value in the position 4 greater than x in a array of data values
- modO is function that performs a modulo operation over the number space 0 to n-1
- cA for example, is equal to B+C+E and represents the coded value that will be communicated and /or stored along with the data value, A.
- the original data set 20, consisting of the exemplary data ABCDEF is sliced into, for example, six (6) data subsets A, B, C, D, E and F.
- the data subsets A, B, C, D, E and F are also coded as discussed below forming coded data subsets cA, cB, cC, cD, cE and cF.
- the data subsets A, B, C, D, E and F and the coded data subsets cA, cB, cC, cD, cE and cF are formed into a plurality of slices 22, 24, 26, 28,30 and 32 as shown, for example, in FIG. 1.
- Each slice 22, 24, 26, 28, 30 and 32 contains a different data value A, B, C, D, E and F and a different coded subset cA, cB, cC, cD, cE and cF.
- the slices 22, 24, 26, 28, 30 and 32 may be transmitted across a communications network, such as the Internet, in a series of data transmissions to a series and each stored in a different digital data storage device or storage node 34, 36, 38, 40, 42 and 44.
- each storage node 34, 36, 38, 40, 42 and 44 are transmitted across a communications network, such as the Internet, to a receiving computer (not shown).
- the receiving computer receives the slices 22, 24, 26, 28, 30 and 32, each of which contains a different data value A, B, C, D, E and F and a different coded value cA, cB, cC, cD, cE and cF.
- FIG. 3 illustrates a condition in which the present invention recreates the original data set when one data slice 22, 24, 26, 28, 30 and 32, for example, the data slice 22 containing the data value A and the coded value cA are not available.
- the original data value A can be obtained as follows:
- cC is a coded value
- D and E are original data values, available from the slices 26, 28 and 30, which are assumed to be available from the nodes 38, 40 and 42, respectively.
- the missing data value can be determined by reversing the coding equation that summed a portion of the data values to create a coded value by subtracting the known data values from a known coded value.
- the data value of the A could be determined as follows:
- FIG. 4 illustrates a condition in which three (3) of the six (6) nodes 34, 36 and 42 which contain the original data values A, B and E and their corresponding coded values cA, cB and cE are not available. These missing data values A, B and E and corresponding in FIG. 4 can be restored by using the following sequence of equations:
- FIG. 5 contains a table that can be used to determine how to recreate the missing data.
- This table lists the 40 different outage scenarios where 1, 2, or 3 out of six storage nodes are be not available or performing slow enough as to be considered not available.
- an 'X' in a row designates that data and coded values from that node are not available.
- the 'Type' column designates the number of nodes not available.
- An 'Offset' value for each outage scenario is also indicated. The offset is the difference the spatial position of a particular outage scenario and the first outage scenario of that Type.
- the data values can be represented by the array d[x], where x is the node number where that data value is stored.
- the coded values can be represented by the array c[x].
- c3dQ is a function in pseudo computer software code as follows: c3d(coded_data_pos, known_data_a_pos, known_data_b_pos, offset)
- n_mod() is the function defined previously.
- the equations in the table in FIG. 6 can be used.
- the Outage Type Num' refers to the corresponding outage 'Type' from FIG. 5.
- the 'Decode Operation' in FIG. 6 refers to the order in which the decode operations are performed.
- the 'Decoded Data' column in FIG. 6 provides the specific decode operations which produces each missing data value.
- n 9
- the number of storage nodes in the grid c[x] is the xth coded data value in the segment array of coded data values d[x+l] is the value in the position 1 greater than x in a array of data values d[x+2] is the value in the position 2 greater than x in a array of data values
- n modO is function that performs a mod over the number space 0 to n-1
- FIG. 8 lists all possible storage grid cases for a 9 storage node grid with 2 storage node outages. Although there are 36 outage cases on a 9 node storage grid with 2 storage node outages, these can be grouped into 4 Types as shown in FIG. 8. Each of these 4 Types represent a particular spatial arrangement of the 2 outages, such as the 2 storage node outages being spatially next to each other in the grid (Type 1) or the 2 storage node outages being separated by one operating storage node (Type 2). The offset listed in FIG.
- FIG. 8 shows the spatial relationship of each outage case within the same Type as they relate to the first outage case of that Type listed in that table.
- the first instance of a Type 1 outage in FIG. 8 is the outage case where NodeO and Nodel are out.
- This first instance of a Type 1 outage is then assigned the Offset value of 0.
- the second instance of a Type 1 outage in FIG. 8 is the outage case where Nodel and Node2 are out. Therefore, this second instance of a Type 1 outage is assigned the Offset value of 1 since the two storage nodes outages occur at storage nodes that are 1 greater than the location of the storage node outages in the first case of Type 1 in FIG. 8.
- This type of validity testing can then be used to test if all original data can be obtained in all other instances where 2 storage nodes on a 9 node storage grid are not operating. Next, all instances where 1 storage node is not operating on a 9 node storage grid are tested to verify whether that candidate coding algorithm is valid. If the validity testing shows that all original data can be obtained in every instance of 2 storage nodes not operating on a 9 node storage grid and every instance of 1 storage node not operating on a 9 node storage grid, then that coding algorithm would be valid to store data on a 9 node storage grid and then to retrieve all original data from that grid if up to 2 storage nodes were not operating.
- a metadata management system is used to manage dispersal and storage of information that is dispersed and stored in several storage nodes coupled to a common communication network forming a grid, for example, as discussed above in connection with FIGS. 1-8.
- metadata attributes of the transactions on the grid are stored in separate dataspace from the dispersed data.
- the information dispersal system "slices" the original data into data subsets and uses a coding algorithm on the data subsets to create coded data subsets.
- data subsets and coded subsets are retrieved from some or all of the storage nodes or communication channels, depending on the availability and performance of each storage node and each communication channel.
- unauthorized access to one or more data subsets only provides reduced or unusable information about the source data. For example as illustrated in FIG.
- each slice 22, 24, 26, 28, 30 and 32 contains a different data value A, B, C, D, E and F and a different "coded subset" (Coded subsets are generated by algorithms and are stored with the data slices to allow for restoration when restoration is done using part of the original subsets) cA, cB, cC, cD, cE and cF.
- the slices 22, 24, 26, 28, 30 and 32 may be transmitted across a communications network, such as the Internet, in a series of data transmissions to a series and each stored in a different digital data storage device or storage node 34, 36, 38, 40, 42 and 44 (i.e., dataspace).
- Each data slice and its corresponding coded subset may be transmitted separately across a communications network and stored in a separate storage node in an array of storage nodes.
- a "file stripe” is the set of data and/or coded subsets corresponding to a particular file. Each file stripe may be stored on a different set of data storage devices or storage nodes 57 within the overall grid as available storage resources or storage nodes may change over time as different files are stored on the grid.
- a "dataspace” is a portion of a storage grid 49 that contains the data of a specific client 64.
- a grid client may also utilize more than one dataspace.
- the dataspaces table 106 in FIG. 11 shows all dataspaces associated with a particular client. Typically, particular grid clients are not able to view the dataspaces of other grid clients in order to provide data security and privacy.
- FIG. 9 shows the different components of a storage grid, generally identified with the reference numeral 49.
- the grid 49 includes associated storage nodes 54 associated with a specific grid client 64 as well as other storage nodes 56 associated with other grid clients (collectively or individually “the storage nodes 57"), connected to a communication network, such as the Internet.
- the grid 49 also includes applications for managing client backups and restorations in terms of dataspaces and their associated collections.
- a "director" is an application running on the grid 49.
- the director serves various purposes, such as:
- the applications on the grid form a metadata management system and include a primary director 58, secondary directors 60 and other directors 62.
- Each dataspace is always associated at any given time with one and only one primary director 58. Every time a grid client 64 attempts any dataspace operation (save/retrieve), the grid client 64 must reconcile the operation with the primary director 58 associated with that dataspace.
- the primary director 58 manages exclusive locks for each dataspace. Every primary director 58 has at least one or more secondary directors 60.
- any dataspace metadata updates are synchronously copied by the dataspace's primary director 58 and to all of its secondary or backup directors 60 before returning acknowledgement status back to the requesting grid client. 64.
- all other directors 62 on the Grid may also asynchronously receive a copy of the metadata update. In such a configuration, all dataspace metadata is effectively copied across the entire grid 49.
- a primary director 58 and its associated secondary directors 60 are also referred to as associated directors 60.
- the secondary directors 60 ensure that any acknowledged metadata management updates are not lost in the event that a primary director 58 fails in the midst of a grid client 64 dataspace update operation.
- the associated directors 66 and other directors 62 do not track which slices are stored on each storage node 57, but rather keep track of the associated storage nodes 57 associated with each grid client 64. Once the specific nodes are known for each client, it is necessary to contact the various storage nodes 57 in order to determine the slices associated with each grid client 64,
- the primary director 58 controls the majority of grid metadata; the storage nodes 57 serve the following responsibilities:
- the storage nodes 57 store the user slices in a file-system that mirrors the user's file-system structure on the Client machine(s).
- the storage node 57 stores a list of per-user files on the storage node 57 in a database.
- the storage node 57 associates minimal metadata attributes, such as Slice hash signatures (e.g., MD5s) with each slice "row" in the database.
- Slice hash signatures e.g., MD5s
- the Grid identifies each storage node 57 with a unique storage volume serial number (volumelD) and as such can identify the storage volume even when it is spread across multiple servers.
- volumelD storage volume serial number
- data subsets and coded subsets are retrieved from some or all of the storage nodes 57 or communication channels, depending on the availability and performance of each storage node 57 and each communication channel.
- Each primary director 58 keeps a list of all storage nodes 57 on the grid 49 and therefore all the nodes available at each site.
- FIG. 10 describes a flow of data and a top level view of what happens when a client interacts with the storage system.
- FIG. 11 illustrates the key metadata tables that are used to keep track of user info in the process.
- a grid client 64 starts with logging in to a a director application running on a server on the grid.
- the director application returns to the grid client 64 in step 72, a DataspaceDirectorMap 92 (FIG.l 1).
- the director application includes an AccountDataspaceMap 93; a look up table which looks up the grid client's AccountID in order to determine the DataspaceED.
- the DataspaceED is then used to determine the grid client's primary director (i.e. DirectorAppBD) from the DataspaceDirectorMap 92.
- the grid client 64 can request a Dataspace VolumeMap 94 (FIG. 11) and use the DataspaceID to determine the storage nodes associated with that grid client 64 (i.e.VoIumelD).
- the primary director 58 sets up a TransactionContextDD for the grid client 64 in a Transactions table 102 (FIG. 11).
- the TransactionContextID is unique for each transaction (i.e for each running instance or session of the grid client 64).
- the Dataspace ID from the DataspaceDirectorMap 92 is used to create a unique transaction ID in a TransactionContexts table 96.
- the transaction ID stored in a Transaction table 102 along with the TransactionContextID in order to keep track of all transactions by all of the grid clients for each session of a grid client with the grid 49.
- TransactionContextld metadata attribute is a different attribute than TransactionID in that a client can be involved with more than one active transactions (not commited) but at all times only one "Transaction context Id" is associated with one running instance of the client.
- the primary director 58 maintains a list of the storage nodes 57 associated with each grid client 64. This list is maintained as a TransactionContexts table 96 which maintains the identities of the storade nodes (i.e. DataspacelD) and the identity of the grid client 64 (i.e. ID).
- the primary director 58 contains the "Application" metadata (i.e. Applications table 104) used by the grid client 64 to communicate with the primary director 58.
- the Applications table 64 is used to record the type of transaction (AppTypelD) , for example add or remove data slices and the storage nodes 57 associated with the transaction (i.e. SitelD).
- the grid client 64 files metadata with the primary director 58 regarding the intended transaction, such as the name and size of the file as well as its creation date and modification date, for example.
- the metadata may also include other metadata attributes, such as the various fields illustrated in the TransactionsDatasources table 98.(FIG. 11)
- the Transaction Datasources metadata table 98 is used to keep control over the transactions until the transactions are completed.
- the grid client 64 connects to the storage nodes in step 74 in preparation for transfer of the file slices. Before any information is exchanged , the grid client 64 registers the metadata in its Datasources table 100 in step 76 in order to fill in the data fields in the Transaction Datasources table 98.
- step 78 the data slices and coded subsets are created in the manner discussed above by an application running on the grid client 64. Any data scrambling, compression and/or encryption of the data may be done before or after the data has been dispersed into slices.
- the data slices are then uploaded to the storage nodes 57 in step 80.
- the grid client 64 uses the transaction metadata (i.e. data from Transaction Datasources table 98) to update the file metadata (i.e. DataSources table 100).
- the datasource information from the Transaction Datasources table 98 is moved to the Datasource table 100 and removed from the Transaction Datasources table 98 in steps 84 , 86 and 88. This process is "atomic" in nature, that is, no change is recorded if at any instance the transaction fails.
- the Datasources table 100 includes revision numbers to maintain the integrity of the user's file set.
- FIGS. 12 A and 12B illustrate the operation of the metadata management system 50.
- the example assumes that the client wants to save a file named "Myfile.txt" on thegrid 49.
- Step 1 The grid client connects to the director application running on the grid 49. Since the director application is not the primary director 58 for this grid client 64, the director application authenticates the grid client and returns the DataspaceDirectorMap 92. Basically, the director uses the AccountID to find its DataspaceED and return the corresponding DirectorAppID (primary director ID for this client).
- Step 2 Once the grid client 64 has the DataspaceDirectorMap 92, it now knows which director is its primary director. The grid client 64 then connects to this director application and the primary director creates a TransactionContextID , as explained above, which is unique for the grid client session. The primary director 58 also sends the grid client 64 its DataspaceVolumeMap 94 (i.e. the number of storage nodes 57 in which the grid client 64 needs toa connection). The grid client 64 sends the file metadata to the director (i.e. fields required in the Transaction Datasources table).
- Step 3 By way of an application running on the client, the data slices and coded subsets of "Myfile.txt" are created using storage algorithms as discussed above.
- the grid client 64 now connects to the various storage nodes 57 on the grid 49, as per the DataspaceVolumeMap 94.
- the grid client now pushes its data and coded subsets to the various storage nodes 57 on the grid 49.
- Step 4 When the grid client 64 is finished saving its file slices on the various storage nodes 57, the grid client 64 notifies the primary director application 58 to remove this transaction from the TransactionDatasources Table 98 and add it to the Datasources Table 100.
- the system is configured so that the grid clent 64 is not able retrieve any file that is not on the Datasources Table 100. As such, adding the file Metadata on the Datasources table 100 completes the file save/backup operation.
- the primary director 58 is an application that decides when a transaction begins or ends. A transaction begins before a primary director 58 sends the storage node 57 metadata to the grid client 64 and it ends after writing the information about the data sources on the Datasources table 100. This configuration insures completeness. As such, if a primary director 58 reports a transaction as having completed, then any application viewing that transaction will know that all the other storage nodes have been appropriately updated for the transaction. This concept of "Atomic Transactions" is important to maintain the integrity of the storage system.
- the storage system is left in a state of disarray, at least for the Dataspace table 100 of the grid client 64 in question. Otherwise, if transactions are interrupted for any reason (e.g., simply by powering off a client PC in the middle of a backup process) and are otherwise left in an incomplete state, the system's overall data integrity would become compromised rather quickly.
Abstract
Briefly, the present invention relates to an information dispersal sytem in which original data to be stored is separated into a number of data 'slices' in such a manner that the data in each subset is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data subsets. These data subsets are stored on separate storage devices as a way of increasing privacy and security. In accordance with an important aspect of the invention, a metadata management system stores and indexes user files across all of the storage nodes. A number of applications run on the servers supporting these storage nodes and are responsible for controlling the metadata. Metadata is the information about the data, the data slices or data subsets and the way in which these data subsets are dispersed among different storage nodes running over the network. As used herein, metadata includes data source names, their size, last modification date, authentication information etc. This information is required to keep track of dispersed data subsets among all the nodes in the system. Every time new data subsets are stored and old ones are removed from the storage nodes, the metadata is updated. In accordance with an important aspect of the invention, the metadata management system stores metadata for dispersed data where: the dispersed data is in several pieces; the metadata is in a separate dataspace from the dispersed data. Accordingly, the metadata management system is able to manage the metadata in a manner that is computationally efficient relative to known systems in order to enable broad use of the invention using the types of computers generally used by businesses, consumers and other organizations currently.
Description
METADATA MANAGEMENT SYSTEM FOR AN INFORMATION DISPERSED
STORAGE SYSTEM
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation-in-part of commonly owned co-pending US Application No. 11/241,555, filed on September 30, 2005.
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to a metadata management system and method for an information dispersal storage system which disperses information into a plurality of subsets or slices for storage in separate storage nodes coupled to a communication network and more particularly, to a metadata management system and method that can store and index metadata attributes of files, such as data source names, file size, last modification date and other data related information, of all file slices in all of the storage nodes coupled to the communication network in order to keep track of dispersed information and is also useful for dispersing data.
2. Description of the Prior Art
[0002] Various data storage systems are known for storing data. Normally such data storage systems store all of the data associated with a particular data set, for example, all the data of a particular user or all the data associated with a particular software
application or all the data in a particular file, in a single dataspace (i.e single digital data storage device). Critical data is known to be initially stored on redundant digital data storage devices. Thus, if there is a failure of one digital data storage device, a complete copy of the data is available on the other digital data storage device. Examples of such systemswith redundant digital data storage devices are disclosed in US Patent Nos.: 5,890,156; 6,058,454; and 6,418,539, hereby incorporated by reference. Although such redundant digital data storage systems are relatively reliable, there are other problems with such systems. First, such systems essentially double or further increase the cost of digital data storage. Second, all of the data in such redundant digital data storage systems is in one place making the data vulnerable to unauthorized access.
[0003] In order to improve the security and thus the reliability of the data storage system, the data may be stored across more than one storage device, such as a hard drive, or removable media, such as a magnetic tape or a so called "memory stick" as set forth in U.S. Pat. No. 6,128,277, hereby incorporated by reference, as well as for reasons relating to performance improvements or capacity limitations. For example, recent data in a database might be stored on a hard drive while older data that is less often used might be stored on a magnetic tape. Another example is storing data from a single file that would be too large to fit on a single hard drive on two hard drives. In each of these cases, the data subset stored on each data storage device does not contain all of the original data, but does contain a generally continuous portion of the data that can be used to provide some usable information. For example, if the original data to be stored was the string of characters in the following sentence:
The quick brown fox jumped over the lazy dog.
and that data was stored on two different data storage devices, then either one or both of those devices would contain usable information. If, for example, the first 20 characters of that 45 character string was stored on one data storage device the remaining 25 characters were stored on a second data storage device, then the sentence be stored as follows:
The quick fox jumped (Stored on the first storage device)
over the lazy brown dog. (Stored on the second storage device)
[0004} In each case, the data stored on each device is not a complete copy of the original data, but each of the data subsets stored on each device provides some usable information.
[0005] Typically, the actual bit pattern of data storage on a device, such as a hard drive, is structured with additional values to represent file types, file systems and storage structures, such as hard drive sectors or memory segments. The techniques used to structure data in particular file types using particular file systems and particular storage structures are well known and allow individuals familiar with these techniques to identify the source data from the bit pattern on a physical media.
[0006] In order to make sure that stored data is only available only to authorized users, data is often stored in an encrypted form using one of several known encryption techniques, such as DES, AES or several others. These encryption techniques store data is some coded form that requires a mathematical key that is ideally known only to authorized users or authorized processes. Although these encryption techniques are difficult to "break" , instances of encryption techniques being broken are known making the data on such data storage systems vulnerable to unauthorized access.
[0007] In addition to securing data using encryption, several methods for improving the security of data storage using information dispersal algorithms have been developed, for example as disclosed in US Patent No. 6,826,711 and US Patent Application Publication No. US 2005/0144382, hereby incorporated by reference. Such information dispersal algorithms are used to "slice" the original data into multiple data subsets and distribute these subsets to different storage nodes (i.e different digital data storage devices). Individually, each data subset or slice does not contain enough information to recreate the original data; however, when threshold number of subsets (i.e.less than the original number of subsets) are available, all the original data can be exactly created.
[0008] The use of such information dispersal algorithms in data storage systems is also described in various trade publications. For example, "How to Share a Secret", by A. Shamir, Communications of the ACM, Vol. 22, No. 11, November, 1979, describes a scheme for sharing a secret, such as a cryptographic key, based on polynomial
interpolation. Another trade publication, "Efficient Dispersal of Information for Security, Load Balancing, and Fault Tolerance", by M. Rabin, Journal of the Association for Computing Machinery, Vol. 36, No. 2, April 1989, pgs. 335-348, also describes a method for information dispersal using an information dispersal algorithm. Unfortunately, these methods and other known information dispersal methods are computationally intensive and are thus not applicable for general storage of large amounts of data using the kinds of computers in broad use by businesses, consumers and other organizations today. Thus there is a need for a data storage system that is able to reliably and securely protect data that does not require the use of computation intensive algorithms.
SUMMARY OF THE INVENTION
Briefly, the present invention relates to an information dispersal sytem in which original data to be stored is separated into a number of subsets or data "slices" in such a manner that the data in each subset is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data slices. These data slices are stored on separate storage devices (i.e., separate dataspaces) as a way of increasing privacy and security. In accordance with an important aspect of the invention, a metadata management system stores and indexes user files across all of the storage nodes. The metadata management system includes a number of applications that run on the servers supporting these storage nodes and are responsible for controlling the metadata. Metadata is the information about the data, the data slices or data subsets and the way in which these data subsets are dispersed among different storage nodes running over the network. More particularly, metadata includes data source names, their size, last modification date, authentication information etc. This information is required to keep track of dispersed data subsets among all the nodes in the system. Every time new data subsets are stored and old ones are removed from the storage nodes, the metadata is updated. In accordance with an important aspect of the invention, the metadata management system stores metadata for dispersed data where:
• The dispersed data is in several pieces.
• The metadata is in a separate dataspace from the dispersed data.
Accordingly, the metadata management system is able to manage the metadata in a manner that is computationally efficient relative to known systems in order to enable broad use of the invention using the types of computers generally used by businesses, consumers and other organizations currently.
DESCRIPTION OF THE DRAWING
[0009] These and other advantages of the present invention will be readily understood with reference to the following drawing and attached specification wherein:
[0010] FIG. 1 is a block diagram of an exemplary data storage system in accordance with the present invention which illustrates how the original data is sliced into data subsets, coded and transmitted to a separate digital data storage device or node.
[0011] FIG. 2 is similar to FIG. 1 but illustrates how the data subsets from all of the exemplary six nodes are retrieved and decoded to recreate the original data set.
[0012] FIG. 3 is similar to FIG. 2 but illustrates a condition of a failure of one of the six digital data storage devices.
[0013] FIG. 4 is similar FIG. 3 but for the condition of a failure of three of the six digital data storage devices.
[0014] FIG. 5 is an examplary tablein accordance with the present invention that can be used to recreate data which has been stored on the exemplary six digital data storage devices.
[0015] FIG. 6 is an exemplary table that lists the decode equations for an exemplary six node storage data storage system for a condition of two node outages
[0016] FIG. 7 is is similar to FIG. 6 but for a condition with three node outages
[0017] FIG. 8 is a table that lists all possible storage node outage states for an exemplary data storage system with nine storage nodes for a condition with two node outages.
[0018] FIG. 9 is an exemplary diagram in accordance with the present invention which illustrates the various functional elements of a metadata management system for use with an information dispersal storage system in accordance with the present invention.
[0019] FIG. 10 is an exemplary flow chart that shows the process for maintaining metadata for data stored on the dispersed data storage grid.
[0020] FIG. 11 shows the essential metadata components that are used during user transactions and during user file set lookup.
[0021] FIGS. 12 A and 12 B illustrate the operation of the system.
DETAILED DESCRIPTION
[0022] The present invention relates to a billing system for an information dispersal storage systen or data storage system. The information dispersal storage system is illustrated and descrided in connection with FIGS. 1-8. A metadata management system for managing information dispersal on a grid which includes a plurality of storage nodes is illustrated in FIGS. 9-12.
Information Dispersal Storage System
[0023] In order to protect the security of the original data, the original data is separated into a number of data "slices" or subsets. The amount of data in each slice is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data subsets. In particular, the system in accordance with the present invention "slices" the original data into data subsets and uses a coding algorithm on the data subsets to create coded data subsets. Each data subset and its corresponding coded subset may be transmitted separately across a communications network and stored in a separate storage node in an array of storage nodes. In order to recreate the original data, data subsets and coded subsets are retrieved from some or all of the storage nodes or communication channels, depending on the availability and performance of each storage node and each communication channel. The original data is recreated by applying a series of decoding algorithms to the retrieved data and coded data.
[0024] As with other known data storage systems based upon information dispersal methods, unauthorized access to one or more data subsets only provides reduced or unusable information about the source data. In accordance with an important aspect of the invention, the system codes and decodes data subsets in a manner that is computationally efficient relative to known systems in order to enable broad use of this method using the types of computers generally used by businesses, consumers and other organizations currently.
[0025] In order to understand the invention, consider a string of N characters do, di, ..., dfl which could comprise a file or a system of files. A typical computer file system may contain gigabytes of data which would mean N would contain trillions of characters. The following example considers a much smaller string where the data string length, N, equals the number of storage nodes, n. To store larger data strings, these methods can be applied repeatedly. These methods can also be applied repeatedly to store computer files or entire file systems.
[0026] For this example, assume that the string contains the characters, O L I V E R where the string contains ASCII character codes as follows: do = O = 79 di = L = 76 d2, = I = 73 d3, = V = 86 dU, = E = 69 ds = R = 82
[0027] The string is broken into segments that are n characters each, where n is chosen to provide the desired reliability and security characteristics while maintaining the desired level of computational efficiency — typically n would be selected to be below 100. In one embodiment, n may be chosen to be greater than four (4) so that each subset of the data contains less than, for example, 1A of the original data, thus decreasing the recognizablity of each data subset.
[0028] In an alternate embodiment, n is selected to be six (6), so that the first original data set is separated into six (6) different data subsets as follows:
A=(I0, B=di, C=(I2, D=d3, E=(I4, F=d5
[0029] For example, where the original data is the starting string of ASCII values for the characters of the text O L I V E R, the values in the data subsets would be those listed below:
A = 79
B = 76
C = 73
D = 86
E = 69
F = 82
[0030] In this embodiment, the coded data values are created by adding data values from a subset of the other data values in the original data set. For example, the coded values can be created by adding the following data values: c[x] = d[n_mod(x+l)] + d[n_mod(x+2)] + d[n_mod(x+4)]
where:
c[x] is the xth coded data value in the segment array of coded data values d[x+l] is the value in the position 1 greater than x in a array of data values d[x+2] is the value in the position 2 greater than x in a array of data values d[x+4] is the value in the position 4 greater than x in a array of data values n modO is function that performs a modulo operation over the number space 0 to n-1
Using this equation, the following coded values are created:
cA, cB, cC, cD, cE, cF
where cA, for example, is equal to B+C+E and represents the coded value that will be communicated and /or stored along with the data value, A.
[0031] For example, where the original data is the starting string of ASCII values for the characters of the text O L I V E R, the values in the coded data subsets would be those listed below: cA = 218 cB = 241 cC = 234 cD = 227 cE = 234 cF = 241
[0032] The original data set 20, consisting of the exemplary data ABCDEF is sliced into, for example, six (6) data subsets A, B, C, D, E and F. The data subsets A, B, C, D, E and F are also coded as discussed below forming coded data subsets cA, cB, cC, cD, cE and cF. The data subsets A, B, C, D, E and F and the coded data subsets cA, cB, cC, cD, cE and cF are formed into a plurality of slices 22, 24, 26, 28,30 and 32 as shown, for example, in FIG. 1. Each slice 22, 24, 26, 28, 30 and 32, contains a different data value A, B, C, D, E and F and a different coded subset cA, cB, cC, cD, cE and cF. The slices 22, 24, 26, 28, 30 and 32 may be transmitted across a communications network, such as the Internet, in a series of data transmissions to a series and each stored in a different digital data storage device or storage node 34, 36, 38, 40, 42 and 44.
[0033] In order to retrieve the original data (or receive it in the case where the data is just transmitted, not stored), the data can reconstructed as shown in FIG. 2. Data values from each storage node 34, 36, 38, 40, 42 and 44 are transmitted across a communications network, such as the Internet, to a receiving computer (not shown). As shown in FIG. 2, the receiving computer receives the slices 22, 24, 26, 28, 30 and 32, each of which contains a different data value A, B, C, D, E and F and a different coded value cA, cB, cC, cD, cE and cF.
[0034] For a variety of reasons, such as the outage or slow performance of a storage node 34, 36, 38, 40, 42 and 44 or a communications connection, not all data slices 22, 24, 26, 28, 30 and 32 will always be available each time data is recreated. FIG. 3 illustrates a condition in which the present invention recreates the original data set when
one data slice 22, 24, 26, 28, 30 and 32, for example, the data slice 22 containing the data value A and the coded value cA are not available. In this case, the original data value A can be obtained as follows:
A = CC - D - E
where cC is a coded value and D and E are original data values, available from the slices 26, 28 and 30, which are assumed to be available from the nodes 38, 40 and 42, respectively. In this case the missing data value can be determined by reversing the coding equation that summed a portion of the data values to create a coded value by subtracting the known data values from a known coded value.
[0035] For example, where the original data is the starting string of ASCII values for the characters of the text O L I V E R, the data value of the A could be determined as follows:
A = 234 - 86 - 69
Therefore A = 79 which is the ASCII value for the character, O.
[0036] In other cases, determining the original data values requires a more detailed decoding equation. For example, FIG. 4 illustrates a condition in which three (3) of the six (6) nodes 34, 36 and 42 which contain the original data values A, B and E and their corresponding coded values cA, cB and cE are not available. These missing data values A, B and E and corresponding in FIG. 4 can be restored by using the following sequence of equations:
1. B = (cD - F + cF -cC) I l
2. E = CD - F - B
3. A = CF - B - D
[0037] These equations are performed in the order listed in order for the data values required for each equation to be available when the specific equation is performed.
[0038] For example, where the original data is the starting string of ASCII values for the characters of the text O L I V E R, the data values of the B, E and A could be determined as follows:
1. B = (227 - 82 + 241 - 234) / 2 B = 76
2. E = 227 - 82 - 76 E = 69
3. A = 241 - 76 - 86
A = 79
[0039] In order to generalize the method for the recreation of all original data ABCDEF when n=6 and up to three slices 22, 24, 26, 28 30 and 32 are not available at the time of the recreation, FIG. 5 contains a table that can be used to determine how to recreate the missing data. This table lists the 40 different outage scenarios where 1, 2, or 3 out of six storage nodes are be not available or performing slow enough as to be considered not available. In the table in FIG. 5, an 'X' in a row designates that data and coded values from that node are not available. The 'Type' column designates the number of nodes not available. An 'Offset' value for each outage scenario is also indicated. The offset is the difference the spatial position of a particular outage scenario and the first outage scenario of that Type.
[0040] The data values can be represented by the array d[x], where x is the node number where that data value is stored. The coded values can be represented by the array c[x].
[0041] In order to reconstruct missing data in an outage scenario where one node is not available in a storage array where n=6, the follow equation can be used: d[0 + offset] = c3d(2, 3, 4, offset)
where c3dQ is a function in pseudo computer software code as follows:
c3d(coded_data_pos, known_data_a_pos, known_data_b_pos, offset)
return unknown_data
}
where n_mod() is the function defined previously. [
[0042] In order to reconstruct missing data in an outage scenario where two nodes are not available in a storage array where n=6, the equations in the table in FIG. 6 can be used. In FIG. 6, the Outage Type Num' refers to the corresponding outage 'Type' from FIG. 5. The 'Decode Operation' in FIG. 6 refers to the order in which the decode operations are performed. The 'Decoded Data' column in FIG. 6 provides the specific decode operations which produces each missing data value.
[0043] In order to reconstruct missing data in an outage scenario where three nodes are not available in a storage array where n=6, the equations in the table in FIG. 7 can be used. Note that in figure 7, the structure of the decode equation for the first decode for outage type = 3 is a different structure than the other decode equations where n=6.
[0044] The example equations listed above are typical of the type of coding and decoding equations that create efficient computing processes using this method, but they only represent one of many examples of how this method can be used to create efficient information distribution systems. In the example above of distributing original data on a storage array of 6 nodes where at least 3 are required to recreate all the data, the computational overhead of creating the coded data is only two addition operations per byte. When data is decoded, no additional operations are required if all storage nodes
and Communications channels are available. If one or two of the storage nodes or communications channels are not available when n=6, then only two additional addition/subtraction operations are required to decode each missing data value. If three storage nodes or communications channels are missing when n=6, then just addition/subtraction operations are required for each missing byte in 11 of 12 instances - in that twelfth instance, only 4 computational operations are required (3 addition/subtractions and one division by an integer). This method is more computationally efficient that known methods, such as those described by Rabin and Shamir.
[0045] This method of selecting a computationally efficient method for secure, distributed data storage by creating coded values to store at storage nodes that also store data subsets can be used to create data storage arrays generally for configurations where n=4 or greater. In each case decoding equations such as those detailed above can be used to recreate missing data in a computationally efficient manner.
[0046] Coding and decoding algorithms for varying grid sizes which tolerate varying numbers of storage node outages without original data loss can also be created using these methods. For example, to create a 9 node grid that can tolerate the loss of 2 nodes, a candidate coding algorithm is selected that uses a mathematical function that incorporates at least two other nodes, such as: c[x] = d[n_mod(x+l)] + d[n_mod(x+2)]
where:
n = 9, the number of storage nodes in the grid c[x] is the xth coded data value in the segment array of coded data values d[x+l] is the value in the position 1 greater than x in a array of data values d[x+2] is the value in the position 2 greater than x in a array of data values n modO is function that performs a mod over the number space 0 to n-1
[0047] In this example embodiment, n=9, the first data segment is separated into different data subsets as follows:
A=do, B=(I1, C=d2, D=d3, E=(I4, F=d5, G=(I6, H=d7, I=d8
[0048] Using this candidate coding algorithm equation above, the following coded values are created: cA, cB, cC, cD, cE, cF, cG, cH, cl
[0049] The candidate coding algorithm is then tested against all possible grid outage states of up to the desired number of storage node outages that can be tolerated with complete data restoration of all original data. FIG. 8 lists all possible storage grid cases for a 9 storage node grid with 2 storage node outages. Although there are 36 outage cases on a 9 node storage grid with 2 storage node outages, these can be grouped into 4 Types as shown in FIG. 8. Each of these 4 Types represent a particular spatial arrangement of the 2 outages, such as the 2 storage node outages being spatially next to each other in the grid (Type 1) or the 2 storage node outages being separated by one operating storage node (Type 2). The offset listed in FIG. 8 shows the spatial relationship of each outage case within the same Type as they relate to the first outage case of that Type listed in that table. For example, the first instance of a Type 1 outage in FIG. 8 is the outage case where NodeO and Nodel are out. This first instance of a Type 1 outage is then assigned the Offset value of 0. The second instance of a Type 1 outage in FIG. 8 is the outage case where Nodel and Node2 are out. Therefore, this second instance of a Type 1 outage is assigned the Offset value of 1 since the two storage nodes outages occur at storage nodes that are 1 greater than the location of the storage node outages in the first case of Type 1 in FIG. 8.
[0050] The validity of the candidate coding algorithm can them be tested by determining if there is a decoding equation or set of decoding equations that can be used to recreate all the original data in each outage Type and thus each outage case. For example, in the first outage case in FIG. 8, NodeO and Nodel are out. This means that the data values A and B are not directly available on the storage grid. However, A can be recreated from cH as follows: cH = I + A A = cH - 1
[0051] The missing data value B can then be created from cl as follows:
Cl = A + B
B = Cl - A
[0052] This type of validity testing can then be used to test if all original data can be obtained in all other instances where 2 storage nodes on a 9 node storage grid are not operating. Next, all instances where 1 storage node is not operating on a 9 node storage grid are tested to verify whether that candidate coding algorithm is valid. If the validity testing shows that all original data can be obtained in every instance of 2 storage nodes not operating on a 9 node storage grid and every instance of 1 storage node not operating on a 9 node storage grid, then that coding algorithm would be valid to store data on a 9 node storage grid and then to retrieve all original data from that grid if up to 2 storage nodes were not operating.
[0053] These types of coding and decoding algorithms can be used by those practiced in the art of software development to create storage grids with varying numbers of storage nodes with varying numbers of storage node outages that can be tolerated by the storage grid while perfectly restoring all original data.
Metadata Management System For Information Dispersal Storage System [0054] In accordance with an important aspect of the invention, a metadata management system is used to manage dispersal and storage of information that is dispersed and stored in several storage nodes coupled to a common communication network forming a grid, for example, as discussed above in connection with FIGS. 1-8. In order to enhance the reliability of the information dispersal system, metadata attributes of the transactions on the grid are stored in separate dataspace from the dispersed data.
[0055] As discussed above, the information dispersal system "slices" the original data into data subsets and uses a coding algorithm on the data subsets to create coded data subsets. In order to recreate the original data, data subsets and coded subsets are retrieved from some or all of the storage nodes or communication channels, depending on the availability and performance of each storage node and each communication channel. As with other known data storage systems based upon information dispersal methods, unauthorized access to one or more data subsets only provides reduced or unusable information about the source data. For example as illustrated in FIG. 1, each slice 22, 24,
26, 28, 30 and 32, contains a different data value A, B, C, D, E and F and a different "coded subset" (Coded subsets are generated by algorithms and are stored with the data slices to allow for restoration when restoration is done using part of the original subsets) cA, cB, cC, cD, cE and cF. The slices 22, 24, 26, 28, 30 and 32 may be transmitted across a communications network, such as the Internet, in a series of data transmissions to a series and each stored in a different digital data storage device or storage node 34, 36, 38, 40, 42 and 44 (i.e., dataspace). Each data slice and its corresponding coded subset may be transmitted separately across a communications network and stored in a separate storage node in an array of storage nodes.
[0056] A "file stripe" is the set of data and/or coded subsets corresponding to a particular file. Each file stripe may be stored on a different set of data storage devices or storage nodes 57 within the overall grid as available storage resources or storage nodes may change over time as different files are stored on the grid.
10057] A "dataspace" is a portion of a storage grid 49 that contains the data of a specific client 64. A grid client may also utilize more than one dataspace. The dataspaces table 106 in FIG. 11 shows all dataspaces associated with a particular client. Typically, particular grid clients are not able to view the dataspaces of other grid clients in order to provide data security and privacy.
[0058] FIG. 9 shows the different components of a storage grid, generally identified with the reference numeral 49. The grid 49 includes associated storage nodes 54 associated with a specific grid client 64 as well as other storage nodes 56 associated with other grid clients (collectively or individually "the storage nodes 57"), connected to a communication network, such as the Internet. The grid 49 also includes applications for managing client backups and restorations in terms of dataspaces and their associated collections.
[0059] In general, a "director" is an application running on the grid 49. The director serves various purposes, such as:
1. Provide a centralized-but-duplicatable point of User-Client login. The Director is the only grid application that stores User-login information.
2. Autonomously provide a per-User list of stored files. All User-Client's can acquire the entire list of files stored on the Grid for each user by talking to one and only one director. This file-list metadata is duplicated across one Primary Directory to several Backup Directors.
3. Track which Sites contain User Slices.
4. Manager Authentication Certificates for other Node personalities.
[0060] The applications on the grid form a metadata management system and include a primary director 58, secondary directors 60 and other directors 62. Each dataspace is always associated at any given time with one and only one primary director 58. Every time a grid client 64 attempts any dataspace operation (save/retrieve), the grid client 64 must reconcile the operation with the primary director 58 associated with that dataspace. Among other things, the primary director 58 manages exclusive locks for each dataspace. Every primary director 58 has at least one or more secondary directors 60. In order to enhance reliability of the system, any dataspace metadata updates (especially lock updates) are synchronously copied by the dataspace's primary director 58 and to all of its secondary or backup directors 60 before returning acknowledgement status back to the requesting grid client. 64. In addition, for additional reliability, all other directors 62 on the Grid may also asynchronously receive a copy of the metadata update. In such a configuration, all dataspace metadata is effectively copied across the entire grid 49.
[0061] As used herein, a primary director 58 and its associated secondary directors 60 are also referred to as associated directors 60. The secondary directors 60 ensure that any acknowledged metadata management updates are not lost in the event that a primary director 58 fails in the midst of a grid client 64 dataspace update operation. There exists a trade-off between the number of secondary directors 60 and the metadata access performance of the grid 49. In general, the greater the number of secondary directors 60, the higher the reliability of metadata updates, but the slower the metadata update response time.
[0062] The associated directors 66 and other directors 62 do not track which slices are stored on each storage node 57, but rather keep track of the associated storage nodes 57 associated with each grid client 64. Once the specific nodes are known for each client, it
is necessary to contact the various storage nodes 57 in order to determine the slices associated with each grid client 64,
[0063] While the primary director 58 controls the majority of grid metadata; the storage nodes 57 serve the following responsibilities:
1. Store the user's slices. The storage nodes 57 store the user slices in a file-system that mirrors the user's file-system structure on the Client machine(s).
2. Store a list of per-user files on the storage node 57 in a database. The storage node 57 associates minimal metadata attributes, such as Slice hash signatures (e.g., MD5s) with each slice "row" in the database.
[0064] The Grid identifies each storage node 57 with a unique storage volume serial number (volumelD) and as such can identify the storage volume even when it is spread across multiple servers. In order to recreate the original data, data subsets and coded subsets are retrieved from some or all of the storage nodes 57 or communication channels, depending on the availability and performance of each storage node 57 and each communication channel. Each primary director 58 keeps a list of all storage nodes 57 on the grid 49 and therefore all the nodes available at each site.
[0065] Following is the list of key metadata attributes used during backup/restore processes:
[0066] FIG. 10 describes a flow of data and a top level view of what happens when a client interacts with the storage system. FIG. 11 illustrates the key metadata tables that are used to keep track of user info in the process.
[0067] Referring to FIG. 10, initially in step 70, a grid client 64 starts with logging in to a a director application running on a server on the grid. After a successful log in, the director application returns to the grid client 64 in step 72, a DataspaceDirectorMap 92 (FIG.l 1). The director application includes an AccountDataspaceMap 93; a look up table which looks up the grid client's AccountID in order to determine the DataspaceED. The DataspaceED is then used to determine the grid client's primary director (i.e. DirectorAppBD) from the DataspaceDirectorMap 92.
[0068] Once the grid client 64 knows its primary director 58., the grid client 64 can request a Dataspace VolumeMap 94 (FIG. 11) and use the DataspaceID to determine the storage nodes associated with that grid client 64 (i.e.VoIumelD). The primary director 58 sets up a TransactionContextDD for the grid client 64 in a Transactions table 102 (FIG. 11). The TransactionContextID is unique for each transaction (i.e for each running instance or session of the grid client 64). In particular, the Dataspace ID from the DataspaceDirectorMap 92 is used to create a unique transaction ID in a TransactionContexts table 96. The transaction ID stored in a Transaction table 102 along
with the TransactionContextID in order to keep track of all transactions by all of the grid clients for each session of a grid client with the grid 49.
[0069J The "TransactionContextld" metadata attribute is a different attribute than TransactionID in that a client can be involved with more than one active transactions (not commited) but at all times only one "Transaction context Id" is associated with one running instance of the client. These metadata attributes allow management of concurrent transactions by different grid clients.
[0070] As mentioned above, the primary director 58 maintains a list of the storage nodes 57 associated with each grid client 64. This list is maintained as a TransactionContexts table 96 which maintains the identities of the storade nodes (i.e. DataspacelD) and the identity of the grid client 64 (i.e. ID). The primary director 58 contains the "Application" metadata ( i.e. Applications table 104) used by the grid client 64 to communicate with the primary director 58. The Applications table 64 is used to record the type of transaction (AppTypelD) , for example add or remove data slices and the storage nodes 57 associated with the transaction (i.e. SitelD).
[0071 J Before any data transfers begins, the grid client 64 files metadata with the primary director 58 regarding the intended transaction, such as the name and size of the file as well as its creation date and modification date, for example. The metadata may also include other metadata attributes, such as the various fields illustrated in the TransactionsDatasources table 98.(FIG. 11) The Transaction Datasources metadata table 98 is used to keep control over the transactions until the transactions are completed.
[0072] After the above information is exchanged between the grid client 64 and the primary director 58, the grid client 64 connects to the storage nodes in step 74 in preparation for transfer of the file slices. Before any information is exchanged , the grid client 64 registers the metadata in its Datasources table 100 in step 76 in order to fill in the data fields in the Transaction Datasources table 98.
[0073] Next in step 78, the data slices and coded subsets are created in the manner discussed above by an application running on the grid client 64. Any data scrambling, compression and/or encryption of the data may be done before or after the data has been dispersed into slices. The data slices are then uploaded to the storage nodes 57 in step 80.
[0074] Once the upload starts, the grid client 64 uses the transaction metadata (i.e. data from Transaction Datasources table 98) to update the file metadata (i.e. DataSources table 100). Once the upload is complete, only then the datasource information from the Transaction Datasources table 98 is moved to the Datasource table 100 and removed from the Transaction Datasources table 98 in steps 84 , 86 and 88. This process is "atomic" in nature, that is, no change is recorded if at any instance the transaction fails. The Datasources table 100 includes revision numbers to maintain the integrity of the user's file set.
[0075] A simple example, as illustrated in FIGS. 12 A and 12B, illustrates the operation of the metadata management system 50. The example assumes that the client wants to save a file named "Myfile.txt" on thegrid 49.
[0076] Step 1 : The grid client connects to the director application running on the grid 49. Since the director application is not the primary director 58 for this grid client 64, the director application authenticates the grid client and returns the DataspaceDirectorMap 92. Basically, the director uses the AccountID to find its DataspaceED and return the corresponding DirectorAppID (primary director ID for this client).
[0077] Step 2: Once the grid client 64 has the DataspaceDirectorMap 92, it now knows which director is its primary director. The grid client 64 then connects to this director application and the primary director creates a TransactionContextID , as explained above, which is unique for the grid client session. The primary director 58 also sends the grid client 64 its DataspaceVolumeMap 94 (i.e. the number of storage nodes 57 in which the grid client 64 needs toa connection). The grid client 64 sends the file metadata to the director (i.e. fields required in the Transaction Datasources table).
[0078] Step 3: By way of an application running on the client, the data slices and coded subsets of "Myfile.txt" are created using storage algorithms as discussed above. The grid client 64 now connects to the various storage nodes 57 on the grid 49, as per the DataspaceVolumeMap 94. The grid client now pushes its data and coded subsets to the various storage nodes 57 on the grid 49.
[0079] Step 4: When the grid client 64 is finished saving its file slices on the various storage nodes 57, the grid client 64 notifies the primary director application 58 to remove
this transaction from the TransactionDatasources Table 98 and add it to the Datasources Table 100. The system is configured so that the grid clent 64 is not able retrieve any file that is not on the Datasources Table 100. As such, adding the file Metadata on the Datasources table 100 completes the file save/backup operation.
[0080] As should be clear from the above, the primary director 58 is an application that decides when a transaction begins or ends. A transaction begins before a primary director 58 sends the storage node 57 metadata to the grid client 64 and it ends after writing the information about the data sources on the Datasources table 100. This configuration insures completeness. As such, if a primary director 58 reports a transaction as having completed, then any application viewing that transaction will know that all the other storage nodes have been appropriately updated for the transaction. This concept of "Atomic Transactions" is important to maintain the integrity of the storage system. For example, if the entire update transaction does not complete, and all of the disparate storage nodes are not appropriately "synchronized," then the storage system is left in a state of disarray, at least for the Dataspace table 100 of the grid client 64 in question. Otherwise, if transactions are interrupted for any reason (e.g., simply by powering off a client PC in the middle of a backup process) and are otherwise left in an incomplete state, the system's overall data integrity would become compromised rather quickly.
[0081] Obviously, many modifications and variations of the present invention are possible in light of the above teachings. Thus, it is to be understood that, within the scope of the appended claims, the invention may be practiced otherwise than is specifically described above.
[0082] What is claimed and desired to be secured by a Letters Patent of the United States is:
Claims
1. An information dispersal system comprising : a plurality of storage nodes connected to a communication network; a metadata management system for managing data transfers to and from the storage nodes; and a system for dispersing the information into subsets, said metadata management system configured to store said subsets in at least two or more different storage nodes.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/404,071 | 2006-04-13 | ||
US11/404,071 US7574579B2 (en) | 2005-09-30 | 2006-04-13 | Metadata management system for an information dispersed storage system |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2007120437A2 true WO2007120437A2 (en) | 2007-10-25 |
WO2007120437A3 WO2007120437A3 (en) | 2009-04-02 |
Family
ID=38610032
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2007/007285 WO2007120437A2 (en) | 2006-04-13 | 2007-03-22 | Metadata management system for an information dispersed storage system |
Country Status (2)
Country | Link |
---|---|
US (2) | US7574579B2 (en) |
WO (1) | WO2007120437A2 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2010057173A3 (en) * | 2008-11-17 | 2010-10-07 | Unisys Corporation | Storage communities of interest using cryptographic splitting |
WO2011007141A1 (en) | 2009-07-17 | 2011-01-20 | Extas Global Ltd. | Distributed storage |
US9026844B2 (en) | 2008-09-02 | 2015-05-05 | Qando Services Inc. | Distributed storage and communication |
US10324849B2 (en) | 2017-02-13 | 2019-06-18 | International Business Machines Corporation | Headless resilient backup and restore software ecosystem selecting server based on historical client latency, workload and metadata presence in cache |
Families Citing this family (707)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8577795B2 (en) | 2002-10-10 | 2013-11-05 | Convergys Information Management Group, Inc. | System and method for revenue and authorization management |
KR100890573B1 (en) * | 2003-08-05 | 2009-03-25 | 세파톤 인코포레이티드 | Emulated storage system |
US7668093B1 (en) * | 2004-08-05 | 2010-02-23 | Convergys Information Management Group, Inc. | Architecture for balancing workload |
US20100162004A1 (en) * | 2008-12-23 | 2010-06-24 | David Dodgson | Storage of cryptographically-split data blocks at geographically-separated locations |
US8694668B2 (en) * | 2005-09-30 | 2014-04-08 | Cleversafe, Inc. | Streaming media software interface to a dispersed data storage network |
US9027080B2 (en) | 2008-03-31 | 2015-05-05 | Cleversafe, Inc. | Proxy access to a dispersed storage network |
US10866754B2 (en) | 2010-04-26 | 2020-12-15 | Pure Storage, Inc. | Content archiving in a distributed storage network |
US7574570B2 (en) * | 2005-09-30 | 2009-08-11 | Cleversafe Inc | Billing system for information dispersal system |
US11221917B1 (en) | 2005-09-30 | 2022-01-11 | Pure Storage, Inc. | Integrity processing in a dispersed storage network |
US7574579B2 (en) | 2005-09-30 | 2009-08-11 | Cleversafe, Inc. | Metadata management system for an information dispersed storage system |
US10747616B2 (en) | 2015-03-31 | 2020-08-18 | Pure Storage, Inc. | Adapting rebuilding of encoded data slices in a dispersed storage network |
US8595435B2 (en) * | 2009-07-30 | 2013-11-26 | Cleversafe, Inc. | Dispersed storage write process |
US9632722B2 (en) | 2010-05-19 | 2017-04-25 | International Business Machines Corporation | Balancing storage unit utilization within a dispersed storage network |
US9774684B2 (en) | 2005-09-30 | 2017-09-26 | International Business Machines Corporation | Storing data in a dispersed storage network |
US10270858B2 (en) | 2005-09-30 | 2019-04-23 | International Business Machines Corporation | Inducing memory device idle time through rolling read prioritizations |
US11416339B1 (en) | 2005-09-30 | 2022-08-16 | Pure Storage, Inc. | Validating requests based on stored vault information |
US11327674B2 (en) * | 2012-06-05 | 2022-05-10 | Pure Storage, Inc. | Storage vault tiering and data migration in a distributed storage network |
US10938418B2 (en) | 2005-09-30 | 2021-03-02 | Pure Storage, Inc. | Online disk replacement/removal |
US8285878B2 (en) * | 2007-10-09 | 2012-10-09 | Cleversafe, Inc. | Block based access to a dispersed data storage network |
US11080138B1 (en) | 2010-04-26 | 2021-08-03 | Pure Storage, Inc. | Storing integrity information in a vast storage system |
US10257276B2 (en) | 2005-09-30 | 2019-04-09 | International Business Machines Corporation | Predictive rebalancing according to future usage expectations |
US7953937B2 (en) * | 2005-09-30 | 2011-05-31 | Cleversafe, Inc. | Systems, methods, and apparatus for subdividing data for storage in a dispersed data storage grid |
US10044807B2 (en) | 2005-09-30 | 2018-08-07 | International Business Machines Corporation | Optimistic checked writes |
US10282440B2 (en) | 2015-03-31 | 2019-05-07 | International Business Machines Corporation | Prioritizing rebuilding of encoded data slices |
US9996413B2 (en) * | 2007-10-09 | 2018-06-12 | International Business Machines Corporation | Ensuring data integrity on a dispersed storage grid |
US10356177B2 (en) | 2005-09-30 | 2019-07-16 | International Business Machines Corporation | Prioritizing ranges to rebuild based on namespace health |
US11620185B2 (en) | 2005-09-30 | 2023-04-04 | Pure Storage, Inc. | Integrity processing in a dispersed storage network |
US7904475B2 (en) * | 2007-10-09 | 2011-03-08 | Cleversafe, Inc. | Virtualized data storage vaults on a dispersed data storage network |
US11909418B1 (en) | 2005-09-30 | 2024-02-20 | Pure Storage, Inc. | Access authentication in a dispersed storage network |
US7546427B2 (en) * | 2005-09-30 | 2009-06-09 | Cleversafe, Inc. | System for rebuilding dispersed data |
US10169229B2 (en) | 2012-06-05 | 2019-01-01 | International Business Machines Corporation | Protocols for expanding existing sites in a dispersed storage network |
US8555109B2 (en) * | 2009-07-30 | 2013-10-08 | Cleversafe, Inc. | Method and apparatus for distributed storage integrity processing |
US8880799B2 (en) * | 2005-09-30 | 2014-11-04 | Cleversafe, Inc. | Rebuilding data on a dispersed storage network |
US10154034B2 (en) | 2010-04-26 | 2018-12-11 | International Business Machines Corporation | Cooperative data access request authorization in a dispersed storage network |
US8171101B2 (en) * | 2005-09-30 | 2012-05-01 | Cleversafe, Inc. | Smart access to a dispersed data storage network |
US10855769B2 (en) | 2005-09-30 | 2020-12-01 | Pure Storage, Inc. | Prioritizing memory devices to replace based on namespace health |
US10051057B2 (en) | 2005-09-30 | 2018-08-14 | International Business Machines Corporation | Prioritizing read locations based on an error history |
US8938591B2 (en) * | 2005-09-30 | 2015-01-20 | Cleversafe, Inc. | Dispersed storage processing unit and methods with data aggregation for use in a dispersed storage system |
US8352782B2 (en) * | 2005-09-30 | 2013-01-08 | Cleversafe, Inc. | Range based rebuilder for use with a dispersed data storage network |
US8209363B2 (en) | 2007-10-09 | 2012-06-26 | Cleversafe, Inc. | File system adapted for use with a dispersed data storage network |
US11340988B2 (en) | 2005-09-30 | 2022-05-24 | Pure Storage, Inc. | Generating integrity information in a vast storage system |
US10389814B2 (en) | 2005-09-30 | 2019-08-20 | Pure Storage, Inc. | Prioritizing memory devices to replace based on namespace health |
US10860424B1 (en) * | 2005-09-30 | 2020-12-08 | Pure Storage, Inc. | Background verification processing in a storage network |
US10432726B2 (en) | 2005-09-30 | 2019-10-01 | Pure Storage, Inc. | Last-resort operations to save at-risk-data |
US11841770B2 (en) | 2005-09-30 | 2023-12-12 | Pure Storage, Inc. | Storage unit connection security in a storage network and methods for use therewith |
US11272009B1 (en) | 2005-09-30 | 2022-03-08 | Pure Storage, Inc. | Managed data slice maintenance in a distributed storage system |
US11474903B1 (en) | 2005-09-30 | 2022-10-18 | Pure Storage, Inc. | Rebuilding of encoded data slices using locally decodable code segments |
US10250686B2 (en) | 2005-09-30 | 2019-04-02 | International Business Machines Corporation | Finding alternate storage locations to support failing disk migration |
US8012542B2 (en) * | 2005-12-30 | 2011-09-06 | E.I. Du Pont De Nemours And Company | Fluoropolymer coating compositions containing adhesive polymers and substrate coating process |
US10027478B2 (en) | 2007-10-09 | 2018-07-17 | International Business Machines Corporation | Differential key backup |
US8478865B2 (en) * | 2007-10-09 | 2013-07-02 | Cleversafe, Inc. | Systems, methods, and apparatus for matching a connection request with a network interface adapted for use with a dispersed data storage network |
US9697171B2 (en) | 2007-10-09 | 2017-07-04 | Internaitonal Business Machines Corporation | Multi-writer revision synchronization in a dispersed storage network |
US8185614B2 (en) * | 2007-10-09 | 2012-05-22 | Cleversafe, Inc. | Systems, methods, and apparatus for identifying accessible dispersed digital storage vaults utilizing a centralized registry |
US8549351B2 (en) * | 2007-10-09 | 2013-10-01 | Cleversafe, Inc. | Pessimistic data reading in a dispersed storage network |
US8819179B2 (en) * | 2007-10-09 | 2014-08-26 | Cleversafe, Inc. | Data revision synchronization in a dispersed storage network |
US9959076B2 (en) | 2007-10-09 | 2018-05-01 | International Business Machines Corporation | Optimized disk load distribution |
US9888076B2 (en) | 2007-10-09 | 2018-02-06 | International Business Machines Corporation | Encoded data slice caching in a distributed storage network |
US8572429B2 (en) * | 2007-10-09 | 2013-10-29 | Cleversafe, Inc. | Optimistic data writing in a dispersed storage network |
US8965956B2 (en) * | 2007-10-09 | 2015-02-24 | Cleversafe, Inc. | Integrated client for use with a dispersed data storage network |
US8533256B2 (en) * | 2007-10-09 | 2013-09-10 | Cleversafe, Inc. | Object interface to a dispersed data storage network |
US8442070B1 (en) * | 2008-02-01 | 2013-05-14 | Hobnob, Inc. | Fractional threshold encoding and aggregation |
US8856552B2 (en) * | 2008-03-31 | 2014-10-07 | Cleversafe, Inc. | Directory synchronization of a dispersed storage network |
US9501355B2 (en) | 2008-03-31 | 2016-11-22 | International Business Machines Corporation | Storing data and directory information in a distributed storage network |
US8266114B2 (en) * | 2008-09-22 | 2012-09-11 | Riverbed Technology, Inc. | Log structured content addressable deduplicating storage |
US8819011B2 (en) * | 2008-07-16 | 2014-08-26 | Cleversafe, Inc. | Command line interpreter for accessing a data object stored in a distributed storage network |
US8630987B2 (en) * | 2008-07-16 | 2014-01-14 | Cleversafe, Inc. | System and method for accessing a data object stored in a distributed storage network |
US9307020B2 (en) * | 2008-10-02 | 2016-04-05 | International Business Machines Corporation | Dispersal and retrieval of data fragments in a peer-to-peer data backup and archival network |
US20100088268A1 (en) * | 2008-10-02 | 2010-04-08 | International Business Machines Corporation | Encryption of data fragments in a peer-to-peer data backup and archival network |
US8935355B2 (en) * | 2008-10-02 | 2015-01-13 | International Business Machines Corporation | Periodic shuffling of data fragments in a peer-to-peer data backup and archival network |
US20100125730A1 (en) * | 2008-11-17 | 2010-05-20 | David Dodgson | Block-level data storage security system |
US8392682B2 (en) | 2008-12-17 | 2013-03-05 | Unisys Corporation | Storage security using cryptographic splitting |
US8386798B2 (en) * | 2008-12-23 | 2013-02-26 | Unisys Corporation | Block-level data storage using an outstanding write list |
US20100162003A1 (en) * | 2008-12-23 | 2010-06-24 | David Dodgson | Retrieval of cryptographically-split data blocks from fastest-responding storage devices |
US8135980B2 (en) | 2008-12-23 | 2012-03-13 | Unisys Corporation | Storage availability using cryptographic splitting |
US20100162032A1 (en) * | 2008-12-23 | 2010-06-24 | David Dodgson | Storage availability using cryptographic splitting |
US8752153B2 (en) | 2009-02-05 | 2014-06-10 | Wwpass Corporation | Accessing data based on authenticated user, provider and system |
JP5802137B2 (en) | 2009-02-05 | 2015-10-28 | ダブリューダブリューパス コーポレイションWwpass Corporation | Centralized authentication system and method with secure private data storage |
US8713661B2 (en) | 2009-02-05 | 2014-04-29 | Wwpass Corporation | Authentication service |
US8751829B2 (en) | 2009-02-05 | 2014-06-10 | Wwpass Corporation | Dispersed secure data storage and retrieval |
US8839391B2 (en) | 2009-02-05 | 2014-09-16 | Wwpass Corporation | Single token authentication |
US11868498B1 (en) | 2009-04-20 | 2024-01-09 | Pure Storage, Inc. | Storage integrity processing in a storage network |
US9092294B2 (en) * | 2009-04-20 | 2015-07-28 | Cleversafe, Inc. | Systems, apparatus, and methods for utilizing a reachability set to manage a network upgrade |
US8656187B2 (en) * | 2009-04-20 | 2014-02-18 | Cleversafe, Inc. | Dispersed storage secure data decoding |
US9483656B2 (en) | 2009-04-20 | 2016-11-01 | International Business Machines Corporation | Efficient and secure data storage utilizing a dispersed data storage system |
US8504847B2 (en) * | 2009-04-20 | 2013-08-06 | Cleversafe, Inc. | Securing data in a dispersed storage network using shared secret slices |
US8744071B2 (en) * | 2009-04-20 | 2014-06-03 | Cleversafe, Inc. | Dispersed data storage system data encryption and encoding |
US10447474B2 (en) * | 2009-04-20 | 2019-10-15 | Pure Storage, Inc. | Dispersed data storage system data decoding and decryption |
US8819781B2 (en) * | 2009-04-20 | 2014-08-26 | Cleversafe, Inc. | Management of network devices within a dispersed data storage network |
US10104045B2 (en) * | 2009-04-20 | 2018-10-16 | International Business Machines Corporation | Verifying data security in a dispersed storage network |
US8601259B2 (en) * | 2009-04-20 | 2013-12-03 | Cleversafe, Inc. | Securing data in a dispersed storage network using security sentinel value |
US10230692B2 (en) * | 2009-06-30 | 2019-03-12 | International Business Machines Corporation | Distributed storage processing module |
US9558059B2 (en) | 2009-07-30 | 2017-01-31 | International Business Machines Corporation | Detecting data requiring rebuilding in a dispersed storage network |
US9207870B2 (en) | 2009-07-30 | 2015-12-08 | Cleversafe, Inc. | Allocating storage units in a dispersed storage network |
US8706980B2 (en) * | 2009-07-30 | 2014-04-22 | Cleversafe, Inc. | Method and apparatus for slice partial rebuilding in a dispersed storage network |
US9009575B2 (en) | 2009-07-30 | 2015-04-14 | Cleversafe, Inc. | Rebuilding a data revision in a dispersed storage network |
US10108492B2 (en) | 2009-07-30 | 2018-10-23 | International Business Machines Corporation | Rebuilding data stored in a dispersed storage network |
US8489915B2 (en) * | 2009-07-30 | 2013-07-16 | Cleversafe, Inc. | Method and apparatus for storage integrity processing based on error types in a dispersed storage network |
US8275966B2 (en) * | 2009-07-30 | 2012-09-25 | Cleversafe, Inc. | Dispersed storage network virtual address generations |
US9208025B2 (en) | 2009-07-30 | 2015-12-08 | Cleversafe, Inc. | Virtual memory mapping in a dispersed storage network |
US8533424B2 (en) * | 2009-07-31 | 2013-09-10 | Cleversafe, Inc. | Computing system utilizing dispersed storage |
US9167277B2 (en) * | 2009-08-03 | 2015-10-20 | Cleversafe, Inc. | Dispersed storage network data manipulation |
US9047217B2 (en) * | 2009-08-27 | 2015-06-02 | Cleversafe, Inc. | Nested distributed storage unit and applications thereof |
US8782086B2 (en) | 2009-08-27 | 2014-07-15 | Cleversafe, Inc. | Updating dispersed storage network access control information |
US9411810B2 (en) * | 2009-08-27 | 2016-08-09 | International Business Machines Corporation | Method and apparatus for identifying data inconsistency in a dispersed storage network |
US8229972B2 (en) * | 2009-08-28 | 2012-07-24 | International Business Machines Corporation | Extended data storage system |
US8924387B2 (en) * | 2009-09-29 | 2014-12-30 | Cleversafe, Inc. | Social networking utilizing a dispersed storage network |
US8548913B2 (en) * | 2009-09-29 | 2013-10-01 | Cleversafe, Inc. | Method and apparatus to secure an electronic commerce transaction |
US8473677B2 (en) * | 2009-09-29 | 2013-06-25 | Cleversafe, Inc. | Distributed storage network memory access based on memory state |
US8478937B2 (en) * | 2009-09-30 | 2013-07-02 | Cleversafe, Inc. | Method and apparatus for dispersed storage memory device utilization |
US20170147219A1 (en) * | 2009-09-30 | 2017-05-25 | International Business Machines Corporation | Utilization of solid-state memory devices in a dispersed storage network |
US8402344B2 (en) * | 2009-10-05 | 2013-03-19 | Cleversafe, Inc. | Method and apparatus for controlling dispersed storage of streaming data |
US10757187B2 (en) | 2009-10-29 | 2020-08-25 | Pure Storage, Inc. | Streaming all-or-nothing encoding with random offset support |
US9015431B2 (en) * | 2009-10-29 | 2015-04-21 | Cleversafe, Inc. | Distributed storage revision rollbacks |
US8291277B2 (en) * | 2009-10-29 | 2012-10-16 | Cleversafe, Inc. | Data distribution utilizing unique write parameters in a dispersed storage system |
US9661356B2 (en) | 2009-10-29 | 2017-05-23 | International Business Machines Corporation | Distribution of unique copies of broadcast data utilizing fault-tolerant retrieval from dispersed storage |
US10389845B2 (en) | 2009-10-29 | 2019-08-20 | Pure Storage, Inc. | Determining how to service requests based on several indicators |
US9774678B2 (en) | 2009-10-29 | 2017-09-26 | International Business Machines Corporation | Temporarily storing data in a dispersed storage network |
US8589637B2 (en) * | 2009-10-30 | 2013-11-19 | Cleversafe, Inc. | Concurrent set storage in distributed storage network |
US9195408B2 (en) | 2009-10-30 | 2015-11-24 | Cleversafe, Inc. | Highly autonomous dispersed storage system retrieval method |
US9413529B2 (en) | 2009-10-30 | 2016-08-09 | International Business Machines Corporation | Distributed storage network and method for storing and retrieving encryption keys |
US9311185B2 (en) | 2009-10-30 | 2016-04-12 | Cleversafe, Inc. | Dispersed storage unit solicitation method and apparatus |
US10073737B2 (en) | 2009-10-30 | 2018-09-11 | International Business Machines Corporation | Slice location identification |
US10509709B2 (en) | 2009-10-30 | 2019-12-17 | Pure Storage, Inc. | Indirect storage of data in a dispersed storage system |
US9900150B2 (en) * | 2009-10-30 | 2018-02-20 | International Business Machines Corporation | Dispersed storage camera device and method of operation |
US9098376B2 (en) | 2009-10-30 | 2015-08-04 | Cleversafe, Inc. | Distributed storage network for modification of a data object |
US8769035B2 (en) | 2009-10-30 | 2014-07-01 | Cleversafe, Inc. | Distributed storage network for storing a data object based on storage requirements |
US9270298B2 (en) | 2009-11-24 | 2016-02-23 | International Business Machines Corporation | Selecting storage units to rebuild an encoded data slice |
US9152514B2 (en) | 2009-11-24 | 2015-10-06 | Cleversafe, Inc. | Rebuilding a data segment in a dispersed storage network |
US8918897B2 (en) * | 2009-11-24 | 2014-12-23 | Cleversafe, Inc. | Dispersed storage network data slice integrity verification |
US9501349B2 (en) | 2009-11-24 | 2016-11-22 | International Business Machines Corporation | Changing dispersed storage error encoding parameters |
US20190056998A1 (en) * | 2009-11-25 | 2019-02-21 | International Business Machines Corporation | Large scale dispersed storage network using set top boxes and centralized control |
US9489264B2 (en) | 2009-11-25 | 2016-11-08 | International Business Machines Corporation | Storing an encoded data slice as a set of sub-slices |
US20180239673A1 (en) * | 2009-11-25 | 2018-08-23 | International Business Machines Corporation | Large scale dispersed storage network using wireless devices and centralized control |
US8527807B2 (en) * | 2009-11-25 | 2013-09-03 | Cleversafe, Inc. | Localized dispersed storage memory system |
US8819452B2 (en) | 2009-11-25 | 2014-08-26 | Cleversafe, Inc. | Efficient storage of encrypted data in a dispersed storage network |
US8688907B2 (en) * | 2009-11-25 | 2014-04-01 | Cleversafe, Inc. | Large scale subscription based dispersed storage network |
US9672109B2 (en) | 2009-11-25 | 2017-06-06 | International Business Machines Corporation | Adaptive dispersed storage network (DSN) and system |
US9836352B2 (en) | 2009-11-25 | 2017-12-05 | International Business Machines Corporation | Detecting a utilization imbalance between dispersed storage network storage units |
US9996548B2 (en) * | 2009-11-25 | 2018-06-12 | International Business Machines Corporation | Dispersed storage using localized peer-to-peer capable wireless devices in a peer-to-peer or femto cell supported carrier served fashion |
US8621268B2 (en) * | 2009-11-25 | 2013-12-31 | Cleversafe, Inc. | Write threshold utilization in a dispersed storage system |
US10015141B2 (en) * | 2009-11-25 | 2018-07-03 | International Business Machines Corporation | Dispersed data storage in a VPN group of devices |
US9626248B2 (en) | 2009-11-25 | 2017-04-18 | International Business Machines Corporation | Likelihood based rebuilding of missing encoded data slices |
US8762343B2 (en) | 2009-12-29 | 2014-06-24 | Cleversafe, Inc. | Dispersed storage of software |
US10067831B2 (en) | 2009-12-29 | 2018-09-04 | International Business Machines Corporation | Slice migration in a dispersed storage network |
US10289505B2 (en) | 2009-12-29 | 2019-05-14 | International Business Machines Corporation | Dispersed multi-media content for a centralized digital video storage system |
US9507735B2 (en) | 2009-12-29 | 2016-11-29 | International Business Machines Corporation | Digital content retrieval utilizing dispersed storage |
US9672108B2 (en) * | 2009-12-29 | 2017-06-06 | International Business Machines Corporation | Dispersed storage network (DSN) and system with improved security |
US10031669B2 (en) | 2009-12-29 | 2018-07-24 | International Business Machines Corporation | Scheduling migration related traffic to be non-disruptive and performant |
US9413393B2 (en) * | 2009-12-29 | 2016-08-09 | International Business Machines Corporation | Encoding multi-media content for a centralized digital video storage system |
US8352831B2 (en) * | 2009-12-29 | 2013-01-08 | Cleversafe, Inc. | Digital content distribution utilizing dispersed storage |
US10372686B2 (en) | 2009-12-29 | 2019-08-06 | International Business Machines Corporation | Policy-based storage in a dispersed storage network |
US10148788B2 (en) | 2009-12-29 | 2018-12-04 | International Business Machines Corporation | Method for providing schedulers in a distributed storage network |
US10001923B2 (en) | 2009-12-29 | 2018-06-19 | International Business Machines Corporation | Generation collapse |
US9727266B2 (en) | 2009-12-29 | 2017-08-08 | International Business Machines Corporation | Selecting storage units in a dispersed storage network |
US8468368B2 (en) * | 2009-12-29 | 2013-06-18 | Cleversafe, Inc. | Data encryption parameter dispersal |
US10158648B2 (en) | 2009-12-29 | 2018-12-18 | International Business Machines Corporation | Policy-based access in a dispersed storage network |
US9922063B2 (en) | 2009-12-29 | 2018-03-20 | International Business Machines Corporation | Secure storage of secret data in a dispersed storage network |
US9866595B2 (en) | 2009-12-29 | 2018-01-09 | International Busines Machines Corporation | Policy based slice deletion in a dispersed storage network |
US20180335967A1 (en) | 2009-12-29 | 2018-11-22 | International Business Machines Corporation | User customizable data processing plan in a dispersed storage network |
US10133632B2 (en) | 2009-12-29 | 2018-11-20 | International Business Machines Corporation | Determining completion of migration in a dispersed storage network |
US9330241B2 (en) | 2009-12-29 | 2016-05-03 | International Business Machines Corporation | Applying digital rights management to multi-media file playback |
US9305597B2 (en) | 2009-12-29 | 2016-04-05 | Cleversafe, Inc. | Accessing stored multi-media content based on a subscription priority level |
US9798467B2 (en) | 2009-12-29 | 2017-10-24 | International Business Machines Corporation | Security checks for proxied requests |
US9369526B2 (en) | 2009-12-29 | 2016-06-14 | International Business Machines Corporation | Distributed storage time synchronization based on retrieval delay |
US8990585B2 (en) * | 2009-12-29 | 2015-03-24 | Cleversafe, Inc. | Time based dispersed storage access |
US10237281B2 (en) | 2009-12-29 | 2019-03-19 | International Business Machines Corporation | Access policy updates in a dispersed storage network |
US8522113B2 (en) * | 2010-01-28 | 2013-08-27 | Cleversafe, Inc. | Selecting storage facilities and dispersal parameters in a dispersed storage network |
US8954667B2 (en) * | 2010-01-28 | 2015-02-10 | Cleversafe, Inc. | Data migration in a dispersed storage network |
US9760440B2 (en) | 2010-01-28 | 2017-09-12 | International Business Machines Corporation | Site-based namespace allocation |
US8649521B2 (en) * | 2010-01-28 | 2014-02-11 | Cleversafe, Inc. | Obfuscation of sequenced encoded data slices |
US8918674B2 (en) * | 2010-01-28 | 2014-12-23 | Cleversafe, Inc. | Directory file system in a dispersed storage network |
US9043548B2 (en) | 2010-01-28 | 2015-05-26 | Cleversafe, Inc. | Streaming content storage |
US20190108366A1 (en) * | 2010-01-28 | 2019-04-11 | International Business Machines Corporation | Secure data transmission utilizing distributed storage |
US9201732B2 (en) | 2010-01-28 | 2015-12-01 | Cleversafe, Inc. | Selective activation of memory to retrieve data in a dispersed storage network |
US11301592B2 (en) | 2010-01-28 | 2022-04-12 | Pure Storage, Inc. | Distributed storage with data obfuscation and method for use therewith |
US9606867B2 (en) | 2014-06-05 | 2017-03-28 | International Business Machines Corporation | Maintaining data storage in accordance with an access metric |
US9135115B2 (en) * | 2010-02-27 | 2015-09-15 | Cleversafe, Inc. | Storing data in multiple formats including a dispersed storage format |
US20180365105A1 (en) | 2014-06-05 | 2018-12-20 | International Business Machines Corporation | Establishing an operation execution schedule in a dispersed storage network |
US10007575B2 (en) | 2010-02-27 | 2018-06-26 | International Business Machines Corporation | Alternative multiple memory format storage in a storage network |
US10268374B2 (en) | 2010-02-27 | 2019-04-23 | International Business Machines Corporation | Redundant array of independent discs and dispersed storage network system re-director |
US11429486B1 (en) | 2010-02-27 | 2022-08-30 | Pure Storage, Inc. | Rebuilding data via locally decodable redundancy in a vast storage network |
US9311184B2 (en) * | 2010-02-27 | 2016-04-12 | Cleversafe, Inc. | Storing raid data as encoded data slices in a dispersed storage network |
US10324791B2 (en) | 2010-11-01 | 2019-06-18 | International Business Machines Corporation | Selectable parallel processing of dispersed storage error encoding |
US8566552B2 (en) * | 2010-03-12 | 2013-10-22 | Cleversafe, Inc. | Dispersed storage network resource allocation |
US8578205B2 (en) * | 2010-03-15 | 2013-11-05 | Cleversafe, Inc. | Requesting cloud data storage |
US8495466B2 (en) * | 2010-03-16 | 2013-07-23 | Cleversafe, Inc. | Adjusting data dispersal in a dispersed storage network |
US9229824B2 (en) | 2010-03-16 | 2016-01-05 | International Business Machines Corporation | Caching rebuilt encoded data slices in a dispersed storage network |
US9170884B2 (en) | 2010-03-16 | 2015-10-27 | Cleversafe, Inc. | Utilizing cached encoded data slices in a dispersed storage network |
US10447767B2 (en) | 2010-04-26 | 2019-10-15 | Pure Storage, Inc. | Resolving a performance issue within a dispersed storage network |
US10956292B1 (en) | 2010-04-26 | 2021-03-23 | Pure Storage, Inc. | Utilizing integrity information for data retrieval in a vast storage system |
US9092386B2 (en) | 2010-04-26 | 2015-07-28 | Cleversafe, Inc. | Indicating an error within a dispersed storage network |
US8938552B2 (en) | 2010-08-02 | 2015-01-20 | Cleversafe, Inc. | Resolving a protocol issue within a dispersed storage network |
US8625635B2 (en) | 2010-04-26 | 2014-01-07 | Cleversafe, Inc. | Dispersed storage network frame protocol header |
US9047218B2 (en) | 2010-04-26 | 2015-06-02 | Cleversafe, Inc. | Dispersed storage network slice name verification |
US9606858B2 (en) | 2010-04-26 | 2017-03-28 | International Business Machines Corporation | Temporarily storing an encoded data slice |
US9898373B2 (en) | 2010-04-26 | 2018-02-20 | International Business Machines Corporation | Prioritizing rebuilding of stored data in a dispersed storage network |
US9495117B2 (en) | 2010-04-26 | 2016-11-15 | International Business Machines Corporation | Storing data in a dispersed storage network |
US8914669B2 (en) | 2010-04-26 | 2014-12-16 | Cleversafe, Inc. | Secure rebuilding of an encoded data slice in a dispersed storage network |
US10911230B2 (en) | 2010-05-19 | 2021-02-02 | Pure Storage, Inc. | Securely activating functionality of a computing device in a dispersed storage network |
US8621580B2 (en) | 2010-05-19 | 2013-12-31 | Cleversafe, Inc. | Retrieving access information in a dispersed storage network |
US10353774B2 (en) | 2015-10-30 | 2019-07-16 | International Business Machines Corporation | Utilizing storage unit latency data in a dispersed storage network |
US8595184B2 (en) | 2010-05-19 | 2013-11-26 | Microsoft Corporation | Scaleable fault-tolerant metadata service |
US10193689B2 (en) | 2010-05-19 | 2019-01-29 | International Business Machines Corporation | Storing access information in a dispersed storage network |
US8861727B2 (en) | 2010-05-19 | 2014-10-14 | Cleversafe, Inc. | Storage of sensitive data in a dispersed storage network |
US11740972B1 (en) | 2010-05-19 | 2023-08-29 | Pure Storage, Inc. | Migrating data in a vast storage network |
US8683205B2 (en) | 2010-05-19 | 2014-03-25 | Cleversafe, Inc. | Accessing data utilizing entity registration in multiple dispersed storage networks |
US8909858B2 (en) | 2010-06-09 | 2014-12-09 | Cleversafe, Inc. | Storing encoded data slices in a dispersed storage network |
US8621269B2 (en) | 2010-06-22 | 2013-12-31 | Cleversafe, Inc. | Identifying a slice name information error in a dispersed storage network |
US8555142B2 (en) | 2010-06-22 | 2013-10-08 | Cleversafe, Inc. | Verifying integrity of data stored in a dispersed storage memory |
US8842746B2 (en) * | 2010-08-02 | 2014-09-23 | Cleversafe, Inc. | Receiving encoded data slices via wireless communication |
US9077734B2 (en) | 2010-08-02 | 2015-07-07 | Cleversafe, Inc. | Authentication of devices of a dispersed storage network |
US10162524B2 (en) | 2010-08-02 | 2018-12-25 | International Business Machines Corporation | Determining whether to compress a data segment in a dispersed storage network |
US20190095101A1 (en) | 2010-08-02 | 2019-03-28 | International Business Machines Corporation | Authenticating a credential in a dispersed storage network |
US9063968B2 (en) | 2010-08-02 | 2015-06-23 | Cleversafe, Inc. | Identifying a compromised encoded data slice |
US10255135B2 (en) | 2010-08-25 | 2019-04-09 | International Business Machines Corporation | Method and apparatus for non-interactive information dispersal |
US9940195B2 (en) | 2010-08-25 | 2018-04-10 | International Business Machines Corporation | Encryption of slice partials |
US8762793B2 (en) | 2010-08-26 | 2014-06-24 | Cleversafe, Inc. | Migrating encoded data slices from a re-provisioned memory device of a dispersed storage network memory |
US9116831B2 (en) | 2010-10-06 | 2015-08-25 | Cleversafe, Inc. | Correcting an errant encoded data slice |
US9112535B2 (en) | 2010-10-06 | 2015-08-18 | Cleversafe, Inc. | Data transmission utilizing partitioning and dispersed storage error encoding |
US10298957B2 (en) | 2010-10-06 | 2019-05-21 | International Business Machines Corporation | Content-based encoding in a multiple routing path communications system |
US9843412B2 (en) | 2010-10-06 | 2017-12-12 | International Business Machines Corporation | Optimizing routing of data across a communications network |
US9571230B2 (en) | 2010-10-06 | 2017-02-14 | International Business Machines Corporation | Adjusting routing of data within a network path |
US10970168B2 (en) | 2010-10-06 | 2021-04-06 | Pure Storage, Inc. | Adjusting dispersed storage error encoding parameters based on path performance |
US10082970B2 (en) | 2010-11-01 | 2018-09-25 | International Business Machines Corporation | Storing an effective dynamic width of encoded data slices |
US10146645B2 (en) | 2010-11-01 | 2018-12-04 | International Business Machines Corporation | Multiple memory format storage in a storage network |
US8707105B2 (en) | 2010-11-01 | 2014-04-22 | Cleversafe, Inc. | Updating a set of memory devices in a dispersed storage network |
US10289318B2 (en) | 2010-11-01 | 2019-05-14 | International Business Machines Corporation | Adjusting optimistic writes in a dispersed storage network |
US10768833B2 (en) | 2010-11-01 | 2020-09-08 | Pure Storage, Inc. | Object dispersal load balancing |
US9015499B2 (en) | 2010-11-01 | 2015-04-21 | Cleversafe, Inc. | Verifying data integrity utilizing dispersed storage |
US9552305B2 (en) | 2010-11-01 | 2017-01-24 | International Business Machines Corporation | Compacting dispersed storage space |
US10805042B2 (en) | 2010-11-01 | 2020-10-13 | Pure Storage, Inc. | Creating transmission data slices for use in a dispersed storage network |
US9590838B2 (en) | 2010-11-09 | 2017-03-07 | International Business Machines Corporation | Transferring data of a dispersed storage network |
US8627065B2 (en) | 2010-11-09 | 2014-01-07 | Cleversafe, Inc. | Validating a certificate chain in a dispersed storage network |
US9336139B2 (en) * | 2010-11-29 | 2016-05-10 | Cleversafe, Inc. | Selecting a memory for storage of an encoded data slice in a dispersed storage network |
US10372350B2 (en) | 2010-11-29 | 2019-08-06 | Pure Storage, Inc. | Shared ownership of namespace ranges |
US10802763B2 (en) | 2010-11-29 | 2020-10-13 | Pure Storage, Inc. | Remote storage verification |
US10922179B2 (en) | 2010-11-29 | 2021-02-16 | Pure Storage, Inc. | Post rebuild verification |
US11307930B1 (en) | 2010-11-29 | 2022-04-19 | Pure Storage, Inc. | Optimized selection of participants in distributed data rebuild/verification |
US11789631B2 (en) | 2010-11-29 | 2023-10-17 | Pure Storage, Inc. | Utilizing metadata storage trees in a vast storage network |
US9454431B2 (en) | 2010-11-29 | 2016-09-27 | International Business Machines Corporation | Memory selection for slice storage in a dispersed storage network |
US8892845B2 (en) | 2010-12-22 | 2014-11-18 | Cleversafe, Inc. | Segmenting data for storage in a dispersed storage network |
US9319463B2 (en) | 2010-12-27 | 2016-04-19 | Cleversafe, Inc. | Reproducing data from obfuscated data retrieved from a dispersed storage network |
US8694752B2 (en) | 2011-02-01 | 2014-04-08 | Cleversafe, Inc. | Transferring data in response to detection of a memory system imbalance |
US10402268B2 (en) | 2011-02-01 | 2019-09-03 | Pure Storage, Inc. | Utilizing a dispersed storage network access token module to acquire digital content from a digital content provider |
US8910022B2 (en) | 2011-03-02 | 2014-12-09 | Cleversafe, Inc. | Retrieval of encoded data slices and encoded instruction slices by a computing device |
US20120226855A1 (en) | 2011-03-02 | 2012-09-06 | Cleversafe, Inc. | Sharing a directory of a dispersed storage network |
US10298684B2 (en) | 2011-04-01 | 2019-05-21 | International Business Machines Corporation | Adaptive replication of dispersed data to improve data access performance |
US8880978B2 (en) | 2011-04-01 | 2014-11-04 | Cleversafe, Inc. | Utilizing a local area network memory and a dispersed storage network memory to access data |
US11418580B2 (en) | 2011-04-01 | 2022-08-16 | Pure Storage, Inc. | Selective generation of secure signatures in a distributed storage network |
US8949688B2 (en) | 2011-04-01 | 2015-02-03 | Cleversafe, Inc. | Updating error recovery information in a dispersed storage network |
US10452836B2 (en) | 2011-05-09 | 2019-10-22 | Pure Storage, Inc. | Retrieving a hypertext markup language file from a dispersed storage network memory |
US9219604B2 (en) | 2011-05-09 | 2015-12-22 | Cleversafe, Inc. | Generating an encrypted message for storage |
US8954787B2 (en) | 2011-05-09 | 2015-02-10 | Cleversafe, Inc. | Establishing trust in a maintenance free storage container |
US20170192684A1 (en) | 2011-05-09 | 2017-07-06 | International Business Machines Corporation | Auditing a transaction in a dispersed storage network |
US9298550B2 (en) | 2011-05-09 | 2016-03-29 | Cleversafe, Inc. | Assigning a dispersed storage network address range in a maintenance free storage container |
US10949301B2 (en) | 2011-06-06 | 2021-03-16 | Pure Storage, Inc. | Pre-positioning pre-stored content in a content distribution system |
US8756480B2 (en) | 2011-06-06 | 2014-06-17 | Cleversafe, Inc. | Prioritized deleting of slices stored in a dispersed storage network |
US10061650B2 (en) | 2011-06-06 | 2018-08-28 | International Business Machines Corporation | Priority based rebuilding |
US10042709B2 (en) | 2011-06-06 | 2018-08-07 | International Business Machines Corporation | Rebuild prioritization during a plurality of concurrent data object write operations |
US8782439B2 (en) | 2011-06-06 | 2014-07-15 | Cleversafe, Inc. | Securing a data segment for storage |
US8924770B2 (en) | 2011-07-06 | 2014-12-30 | Cleversafe, Inc. | Rebuilding a data slice of a maintenance free storage container |
US9460148B2 (en) | 2011-07-06 | 2016-10-04 | International Business Machines Corporation | Completing distribution of multi-media content to an accessing device |
US9170868B2 (en) | 2011-07-27 | 2015-10-27 | Cleversafe, Inc. | Identifying an error cause within a dispersed storage network |
US10678619B2 (en) | 2011-07-27 | 2020-06-09 | Pure Storage, Inc. | Unified logs and device statistics |
US11016702B2 (en) | 2011-07-27 | 2021-05-25 | Pure Storage, Inc. | Hierarchical event tree |
US8782491B2 (en) | 2011-08-17 | 2014-07-15 | Cleversafe, Inc. | Detecting intentional corruption of data in a dispersed storage network |
US9971802B2 (en) | 2011-08-17 | 2018-05-15 | International Business Machines Corporation | Audit record transformation in a dispersed storage network |
US10454678B2 (en) | 2011-08-17 | 2019-10-22 | Pure Storage, Inc. | Accesor-based audit trails |
US10120756B2 (en) | 2011-08-17 | 2018-11-06 | International Business Machines Corporation | Audit object generation in a dispersed storage network |
US11907060B2 (en) | 2011-09-06 | 2024-02-20 | Pure Storage, Inc. | Coding of data streams in a vast storage network |
US8751894B2 (en) | 2011-09-06 | 2014-06-10 | Cleversafe, Inc. | Concurrent decoding of data streams |
US20190179696A1 (en) | 2011-09-06 | 2019-06-13 | International Business Machines Corporation | Demultiplexing decoded data streams in a distributed storage network |
US10235237B2 (en) | 2011-09-06 | 2019-03-19 | Intertnational Business Machines Corporation | Decoding data streams in a distributed storage network |
US8782494B2 (en) | 2011-10-04 | 2014-07-15 | Cleversafe, Inc. | Reproducing data utilizing a zero information gain function |
US8555130B2 (en) | 2011-10-04 | 2013-10-08 | Cleversafe, Inc. | Storing encoded data slices in a dispersed storage unit |
US9785491B2 (en) | 2011-10-04 | 2017-10-10 | International Business Machines Corporation | Processing a certificate signing request in a dispersed storage network |
US8683286B2 (en) * | 2011-11-01 | 2014-03-25 | Cleversafe, Inc. | Storing data in a dispersed storage network |
US8839368B2 (en) | 2011-11-01 | 2014-09-16 | Cleversafe, Inc. | Acquiring a trusted set of encoded data slices |
US10437678B2 (en) | 2011-11-01 | 2019-10-08 | Pure Storage, Inc. | Updating an encoded data slice |
US10496500B2 (en) | 2011-11-01 | 2019-12-03 | Pure Storage, Inc. | Preemptively reading extra encoded data slices |
US10365969B2 (en) | 2011-11-01 | 2019-07-30 | International Business Machines Corporation | Multiple wireless communication systems stream slices based on geography |
US11329830B1 (en) | 2011-11-01 | 2022-05-10 | Pure Storage, Inc. | Dispersed credentials |
US8627066B2 (en) | 2011-11-03 | 2014-01-07 | Cleversafe, Inc. | Processing a dispersed storage network access request utilizing certificate chain validation information |
US10318445B2 (en) | 2011-11-28 | 2019-06-11 | International Business Machines Corporation | Priority level adaptation in a dispersed storage network |
US10055283B2 (en) | 2011-11-28 | 2018-08-21 | International Business Machines Corporation | Securely distributing random keys in a dispersed storage network |
US10558592B2 (en) | 2011-11-28 | 2020-02-11 | Pure Storage, Inc. | Priority level adaptation in a dispersed storage network |
US8848906B2 (en) | 2011-11-28 | 2014-09-30 | Cleversafe, Inc. | Encrypting data for storage in a dispersed storage network |
US10387071B2 (en) | 2011-11-28 | 2019-08-20 | Pure Storage, Inc. | On-the-fly cancellation of unnecessary read requests |
US9203625B2 (en) | 2011-11-28 | 2015-12-01 | Cleversafe, Inc. | Transferring encoded data slices in a distributed storage network |
US10977194B2 (en) | 2011-11-28 | 2021-04-13 | Pure Storage, Inc. | Securely storing random keys in a dispersed storage network |
US11474958B1 (en) | 2011-11-28 | 2022-10-18 | Pure Storage, Inc. | Generating and queuing system messages with priorities in a storage network |
US8555079B2 (en) | 2011-12-06 | 2013-10-08 | Wwpass Corporation | Token management |
US8972719B2 (en) | 2011-12-06 | 2015-03-03 | Wwpass Corporation | Passcode restoration |
US8656180B2 (en) | 2011-12-06 | 2014-02-18 | Wwpass Corporation | Token activation |
US10348640B2 (en) | 2011-12-12 | 2019-07-09 | International Business Machines Corporation | Partial task execution in a dispersed storage network |
US10146621B2 (en) | 2011-12-12 | 2018-12-04 | International Business Machines Corporation | Chaining computes in a distributed computing system |
US10360106B2 (en) | 2011-12-12 | 2019-07-23 | International Business Machines Corporation | Throttled real-time writes |
US20180083930A1 (en) | 2011-12-12 | 2018-03-22 | International Business Machines Corporation | Reads for dispersed computation jobs |
US10666596B2 (en) | 2011-12-12 | 2020-05-26 | Pure Storage, Inc. | Messaging via a shared memory of a distributed computing system |
US20130238900A1 (en) | 2011-12-12 | 2013-09-12 | Cleversafe, Inc. | Dispersed storage network secure hierarchical file directory |
US10346218B2 (en) | 2011-12-12 | 2019-07-09 | International Business Machines Corporation | Partial task allocation in a dispersed storage network |
US9430286B2 (en) | 2011-12-12 | 2016-08-30 | International Business Machines Corporation | Authorizing distributed task processing in a distributed storage network |
US10104168B2 (en) | 2011-12-12 | 2018-10-16 | International Business Machines Corporation | Method for managing throughput in a distributed storage network |
US9141468B2 (en) | 2011-12-12 | 2015-09-22 | Cleversafe, Inc. | Managing memory utilization in a distributed storage and task network |
US9817701B2 (en) | 2011-12-12 | 2017-11-14 | International Business Machines Corporation | Threshold computing in a distributed computing system |
US10176045B2 (en) | 2011-12-12 | 2019-01-08 | International Business Machines Corporation | Internet based shared memory in a distributed computing system |
US9584359B2 (en) | 2011-12-12 | 2017-02-28 | International Business Machines Corporation | Distributed storage and computing of interim data |
US9298548B2 (en) | 2011-12-12 | 2016-03-29 | Cleversafe, Inc. | Distributed computing in a distributed storage and task network |
US9674155B2 (en) | 2011-12-12 | 2017-06-06 | International Business Machines Corporation | Encrypting segmented data in a distributed computing system |
US9009567B2 (en) | 2011-12-12 | 2015-04-14 | Cleversafe, Inc. | Encrypting distributed computing data |
US9146810B2 (en) | 2012-01-31 | 2015-09-29 | Cleversafe, Inc. | Identifying a potentially compromised encoded data slice |
US9891995B2 (en) * | 2012-01-31 | 2018-02-13 | International Business Machines Corporation | Cooperative decentralized rebuild scanning |
US10140177B2 (en) | 2012-01-31 | 2018-11-27 | International Business Machines Corporation | Transferring a partial task in a distributed computing system |
US10671585B2 (en) | 2012-01-31 | 2020-06-02 | Pure Storage, Inc. | Storing indexed data to a dispersed storage network |
US9465861B2 (en) * | 2012-01-31 | 2016-10-11 | International Business Machines Corporation | Retrieving indexed data from a dispersed storage network |
US9514132B2 (en) * | 2012-01-31 | 2016-12-06 | International Business Machines Corporation | Secure data migration in a dispersed storage network |
US10157051B2 (en) | 2012-03-02 | 2018-12-18 | International Business Machines Corporation | Upgrading devices in a dispersed storage network |
US10402393B2 (en) | 2012-03-02 | 2019-09-03 | Pure Storage, Inc. | Slice migration in a dispersed storage network |
US9171031B2 (en) | 2012-03-02 | 2015-10-27 | Cleversafe, Inc. | Merging index nodes of a hierarchical dispersed storage index |
US11232093B2 (en) | 2012-03-02 | 2022-01-25 | Pure Storage, Inc. | Slice migration in a dispersed storage network |
US9195684B2 (en) | 2012-03-02 | 2015-11-24 | Cleversafe, Inc. | Redundant task execution in a distributed storage and task network |
US9380032B2 (en) | 2012-04-25 | 2016-06-28 | International Business Machines Corporation | Encrypting data for storage in a dispersed storage network |
US10795766B2 (en) | 2012-04-25 | 2020-10-06 | Pure Storage, Inc. | Mapping slice groupings in a dispersed storage network |
US10621044B2 (en) | 2012-04-25 | 2020-04-14 | Pure Storage, Inc. | Mapping slice groupings in a dispersed storage network |
US10447471B2 (en) | 2012-06-05 | 2019-10-15 | Pure Storage, Inc. | Systematic secret sharing |
US9613052B2 (en) | 2012-06-05 | 2017-04-04 | International Business Machines Corporation | Establishing trust within a cloud computing system |
US9164841B2 (en) | 2012-06-05 | 2015-10-20 | Cleversafe, Inc. | Resolution of a storage error in a dispersed storage network |
US10002047B2 (en) | 2012-06-05 | 2018-06-19 | International Business Machines Corporation | Read-if-not-revision-equals protocol message |
US10073638B2 (en) | 2012-06-05 | 2018-09-11 | International Business Machines Corporation | Automatic namespace ordering determination |
US10474395B2 (en) | 2012-06-05 | 2019-11-12 | Pure Storage, Inc. | Abstracting namespace mapping in a dispersed storage network through multiple hierarchies |
US9330277B2 (en) | 2012-06-21 | 2016-05-03 | Google Technology Holdings LLC | Privacy manager for restricting correlation of meta-content having protected information based on privacy rules |
US8959574B2 (en) | 2012-06-21 | 2015-02-17 | Google Technology Holdings LLC | Content rights protection with arbitrary correlation of second content |
US10430276B2 (en) | 2012-06-25 | 2019-10-01 | Pure Storage, Inc. | Optimal orderings of processing unit priorities in a dispersed storage network |
US9141297B2 (en) | 2012-06-25 | 2015-09-22 | Cleversafe, Inc. | Verifying encoded data slice integrity in a dispersed storage network |
US10120574B2 (en) | 2012-06-25 | 2018-11-06 | International Business Machines Corporation | Reversible data modifications within DS units |
US10157011B2 (en) | 2012-06-25 | 2018-12-18 | International Business Machines Corporation | Temporary suspension of vault access |
US20180336097A1 (en) | 2012-06-25 | 2018-11-22 | International Business Machines Corporation | Namespace affinity and failover for processing units in a dispersed storage network |
US8935761B2 (en) | 2012-06-25 | 2015-01-13 | Cleversafe, Inc. | Accessing storage nodes in an on-line media storage system |
US11093327B1 (en) | 2012-06-25 | 2021-08-17 | Pure Storage, Inc. | Failure abatement approach for failed storage units common to multiple vaults |
US10114697B2 (en) | 2012-06-25 | 2018-10-30 | International Business Machines Corporation | Large object parallel writing |
US9537609B2 (en) | 2012-08-02 | 2017-01-03 | International Business Machines Corporation | Storing a stream of data in a dispersed storage network |
US10651975B2 (en) | 2012-08-02 | 2020-05-12 | Pure Storage, Inc. | Forwarding data amongst cooperative DSTN processing units of a massive data ingestion system |
US10331518B2 (en) | 2012-08-31 | 2019-06-25 | International Business Machines Corporation | Encoding data in a dispersed storage network |
US9176822B2 (en) | 2012-08-31 | 2015-11-03 | Cleversafe, Inc. | Adjusting dispersed storage error encoding parameters |
US10409679B2 (en) | 2012-08-31 | 2019-09-10 | Pure Storage, Inc. | Migrating data slices in a dispersed storage network |
US11360851B2 (en) | 2012-08-31 | 2022-06-14 | Pure Storage, Inc. | Duplicating authentication information between connections |
US10241863B2 (en) | 2012-08-31 | 2019-03-26 | International Business Machines Corporation | Slice rebuilding in a dispersed storage network |
US9875158B2 (en) | 2012-08-31 | 2018-01-23 | International Business Machines Corporation | Slice storage in a dispersed storage network |
US10409678B2 (en) | 2012-08-31 | 2019-09-10 | Pure Storage, Inc. | Self-optimizing read-ahead |
US10318549B2 (en) | 2012-09-13 | 2019-06-11 | International Business Machines Corporation | Batching modifications to nodes in a dispersed index |
US10057351B2 (en) | 2012-09-13 | 2018-08-21 | International Business Machines Corporation | Modifying information dispersal algorithm configurations in a dispersed storage network |
US10417253B2 (en) | 2012-09-13 | 2019-09-17 | Pure Storage, Inc. | Multi-level data storage in a dispersed storage network |
US10331698B2 (en) | 2012-09-13 | 2019-06-25 | International Business Machines Corporation | Rebuilding data in a dispersed storage network |
US9483539B2 (en) | 2012-09-13 | 2016-11-01 | International Business Machines Corporation | Updating local data utilizing a distributed storage network |
US10402423B2 (en) | 2012-09-13 | 2019-09-03 | Pure Storage, Inc. | Sliding windows for batching index updates |
US10606700B2 (en) | 2012-10-08 | 2020-03-31 | Pure Storage, Inc. | Enhanced dispersed storage error encoding using multiple encoding layers |
US10331519B2 (en) | 2012-10-08 | 2019-06-25 | International Business Machines Corporation | Application of secret sharing schemes at multiple levels of a dispersed storage network |
US10127111B2 (en) | 2012-10-08 | 2018-11-13 | International Business Machines Corporation | Client provided request prioritization hints |
US9648087B2 (en) | 2012-10-08 | 2017-05-09 | International Business Machines Corporation | Allocating distributed storage and task execution resources |
US9223723B2 (en) | 2012-10-30 | 2015-12-29 | Cleversafe, Inc. | Verifying data of a dispersed storage network |
US9936020B2 (en) | 2012-10-30 | 2018-04-03 | International Business Machines Corporation | Access control of data in a dispersed storage network |
US10587691B2 (en) | 2012-12-05 | 2020-03-10 | Pure Storage, Inc. | Impatient writes |
US10558621B2 (en) | 2012-12-05 | 2020-02-11 | Pure Storage, Inc. | Lock stealing writes for improved reliability |
US9521197B2 (en) | 2012-12-05 | 2016-12-13 | International Business Machines Corporation | Utilizing data object storage tracking in a dispersed storage network |
US9811533B2 (en) | 2012-12-05 | 2017-11-07 | International Business Machines Corporation | Accessing distributed computing functions in a distributed computing system |
US9448941B1 (en) * | 2012-12-31 | 2016-09-20 | Emc Corporation | System and method for cache management |
US10642992B2 (en) | 2013-01-04 | 2020-05-05 | Pure Storage, Inc. | Password augmented all-or-nothin transform |
US10423491B2 (en) | 2013-01-04 | 2019-09-24 | Pure Storage, Inc. | Preventing multiple round trips when writing to target widths |
US10402270B2 (en) | 2013-01-04 | 2019-09-03 | Pure Storage, Inc. | Deterministically determining affinity for a source name range |
US10241866B2 (en) | 2013-01-04 | 2019-03-26 | International Business Machines Corporation | Allocating rebuilding queue entries in a dispersed storage network |
US10013203B2 (en) | 2013-01-04 | 2018-07-03 | International Business Machines Corporation | Achieving storage compliance in a dispersed storage network |
US20190250823A1 (en) | 2013-01-04 | 2019-08-15 | International Business Machines Corporation | Efficient computation of only the required slices |
US10229002B2 (en) | 2013-01-04 | 2019-03-12 | International Business Machines Corporation | Process to migrate named objects to a dispersed or distributed storage network (DSN) |
US9558067B2 (en) | 2013-01-04 | 2017-01-31 | International Business Machines Corporation | Mapping storage of data in a dispersed storage network |
US10204009B2 (en) | 2013-01-04 | 2019-02-12 | International Business Machines Corporation | Prioritized rebuilds using dispersed indices |
US11416340B1 (en) | 2013-01-04 | 2022-08-16 | Pure Storage, Inc. | Storage system with multiple storage types in a vast storage network |
US9311187B2 (en) | 2013-01-04 | 2016-04-12 | Cleversafe, Inc. | Achieving storage compliance in a dispersed storage network |
US9043499B2 (en) | 2013-02-05 | 2015-05-26 | Cleversafe, Inc. | Modifying a dispersed storage network memory data access response plan |
US10310763B2 (en) | 2013-02-05 | 2019-06-04 | International Business Machines Corporation | Forming a distributed storage network memory without namespace aware distributed storage units |
US10664360B2 (en) | 2013-02-05 | 2020-05-26 | Pure Storage, Inc. | Identifying additional resources to accelerate rebuildling |
US10621021B2 (en) | 2013-02-05 | 2020-04-14 | Pure Storage, Inc. | Using dispersed data structures to point to slice or date source replicas |
US10430122B2 (en) | 2013-02-05 | 2019-10-01 | Pure Storage, Inc. | Using partial rebuilding to change information dispersal algorithm (IDA) |
US10055441B2 (en) | 2013-02-05 | 2018-08-21 | International Business Machines Corporation | Updating shared group information in a dispersed storage network |
US10268554B2 (en) | 2013-02-05 | 2019-04-23 | International Business Machines Corporation | Using dispersed computation to change dispersal characteristics |
US20140236670A1 (en) * | 2013-02-21 | 2014-08-21 | Bank Of America Corporation | Data Communication and Analytics Platform |
US9274908B2 (en) | 2013-02-26 | 2016-03-01 | International Business Machines Corporation | Resolving write conflicts in a dispersed storage network |
US10642489B2 (en) | 2013-02-26 | 2020-05-05 | Pure Storage, Inc. | Determining when to initiate an intra-distributed storage unit rebuild vs. an inter-distributed storage unit rebuild |
US11036392B2 (en) | 2013-02-26 | 2021-06-15 | Pure Storage, Inc. | Determining when to use convergent encryption |
US10075523B2 (en) | 2013-04-01 | 2018-09-11 | International Business Machines Corporation | Efficient storage of data in a dispersed storage network |
US9405926B2 (en) * | 2014-06-30 | 2016-08-02 | Paul Lewis | Systems and methods for jurisdiction independent data storage in a multi-vendor cloud environment |
US10984116B2 (en) | 2013-04-15 | 2021-04-20 | Calamu Technologies Corporation | Systems and methods for digital currency or crypto currency storage in a multi-vendor cloud environment |
US9104332B2 (en) * | 2013-04-16 | 2015-08-11 | International Business Machines Corporation | Managing metadata and data for a logical volume in a distributed and declustered system |
US9298398B2 (en) | 2013-04-16 | 2016-03-29 | International Business Machines Corporation | Fine-grained control of data placement |
US9329938B2 (en) | 2013-04-16 | 2016-05-03 | International Business Machines Corporation | Essential metadata replication |
US9423981B2 (en) | 2013-04-16 | 2016-08-23 | International Business Machines Corporation | Logical region allocation with immediate availability |
US9104597B2 (en) | 2013-04-16 | 2015-08-11 | International Business Machines Corporation | Destaging cache data using a distributed freezer |
US9619404B2 (en) | 2013-04-16 | 2017-04-11 | International Business Machines Corporation | Backup cache with immediate availability |
US9298617B2 (en) | 2013-04-16 | 2016-03-29 | International Business Machines Corporation | Parallel destaging with replicated cache pinning |
US10223213B2 (en) | 2013-05-03 | 2019-03-05 | International Business Machines Corporation | Salted zero expansion all or nothing transformation |
US9456035B2 (en) | 2013-05-03 | 2016-09-27 | International Business Machines Corporation | Storing related data in a dispersed storage network |
US9405609B2 (en) | 2013-05-22 | 2016-08-02 | International Business Machines Corporation | Storing data in accordance with a performance threshold |
US9424132B2 (en) | 2013-05-30 | 2016-08-23 | International Business Machines Corporation | Adjusting dispersed storage network traffic due to rebuilding |
US11226860B1 (en) | 2013-05-30 | 2022-01-18 | Pure Storage, Inc. | Difference based rebuild list scanning |
US9432341B2 (en) | 2013-05-30 | 2016-08-30 | International Business Machines Corporation | Securing data in a dispersed storage network |
US9652470B2 (en) | 2013-07-01 | 2017-05-16 | International Business Machines Corporation | Storing data in a dispersed storage network |
US10169369B2 (en) | 2013-07-01 | 2019-01-01 | International Business Machines Corporation | Meeting storage requirements with limited storage resources |
US9501360B2 (en) | 2013-07-01 | 2016-11-22 | International Business Machines Corporation | Rebuilding data while reading data in a dispersed storage network |
US9620148B2 (en) * | 2013-07-01 | 2017-04-11 | Toyota Motor Engineering & Manufacturing North America, Inc. | Systems, vehicles, and methods for limiting speech-based access to an audio metadata database |
US11221916B2 (en) | 2013-07-01 | 2022-01-11 | Pure Storage, Inc. | Prioritized data reconstruction in a dispersed storage network |
US10133635B2 (en) | 2013-07-01 | 2018-11-20 | International Business Machines Corporation | Low-width vault in distributed storage system |
US20180188964A1 (en) | 2013-07-31 | 2018-07-05 | International Business Machines Corporation | Managed storage unit shutdown in a distributed storage network |
US10681134B2 (en) | 2013-07-31 | 2020-06-09 | Pure Storage, Inc. | Accelerated learning in adaptive rebuilding by applying observations to other samples |
US10180880B2 (en) | 2013-07-31 | 2019-01-15 | International Business Machines Corporation | Adaptive rebuilding rates based on sampling and inference |
US9894157B2 (en) | 2013-07-31 | 2018-02-13 | International Business Machines Corporation | Distributed storage network with client subsets and methods for use therewith |
US20150039660A1 (en) | 2013-07-31 | 2015-02-05 | Cleversafe, Inc. | Co-locate objects request |
US9438675B2 (en) | 2013-08-29 | 2016-09-06 | International Business Machines Corporation | Dispersed storage with variable slice length and methods for use therewith |
US10514857B2 (en) | 2013-08-29 | 2019-12-24 | Pure Storage, Inc. | Dynamic adjusting of parameters based on resource scoring |
US10484474B2 (en) | 2013-08-29 | 2019-11-19 | Pure Storage, Inc. | Rotating offline DS units |
US9749414B2 (en) | 2013-08-29 | 2017-08-29 | International Business Machines Corporation | Storing low retention priority data in a dispersed storage network |
US10489071B2 (en) | 2013-08-29 | 2019-11-26 | Pure Storage, Inc. | Vault provisioning within dispersed or distributed storage network (DSN) |
US10601918B2 (en) | 2013-08-29 | 2020-03-24 | Pure Storage, Inc. | Rotating inactive storage units in a distributed storage network |
US9857974B2 (en) | 2013-10-03 | 2018-01-02 | International Business Machines Corporation | Session execution decision |
US10182115B2 (en) | 2013-11-01 | 2019-01-15 | International Business Machines Corporation | Changing rebuild priority for a class of data |
US9781208B2 (en) | 2013-11-01 | 2017-10-03 | International Business Machines Corporation | Obtaining dispersed storage network system registry information |
US10304096B2 (en) | 2013-11-01 | 2019-05-28 | International Business Machines Corporation | Renting a pipe to a storage system |
US9900316B2 (en) | 2013-12-04 | 2018-02-20 | International Business Machines Corporation | Accessing storage units of a dispersed storage network |
US11340993B2 (en) | 2014-01-06 | 2022-05-24 | Pure Storage, Inc. | Deferred rebuilding with alternate storage locations |
US9594639B2 (en) | 2014-01-06 | 2017-03-14 | International Business Machines Corporation | Configuring storage resources of a dispersed storage network |
US10922181B2 (en) | 2014-01-06 | 2021-02-16 | Pure Storage, Inc. | Using storage locations greater than an IDA width in a dispersed storage network |
US9552261B2 (en) | 2014-01-31 | 2017-01-24 | International Business Machines Corporation | Recovering data from microslices in a dispersed storage network |
US11204836B1 (en) | 2014-01-31 | 2021-12-21 | Pure Storage, Inc. | Using trap slices for anomaly detection in a distributed storage network |
US9778987B2 (en) | 2014-01-31 | 2017-10-03 | International Business Machines Corporation | Writing encoded data slices in a dispersed storage network |
US10318382B2 (en) | 2014-01-31 | 2019-06-11 | International Business Machines Corporation | Determining missing encoded data slices |
US10635312B2 (en) | 2014-02-26 | 2020-04-28 | Pure Storage, Inc. | Recovering data in a dispersed storage network |
US10678638B2 (en) | 2014-02-26 | 2020-06-09 | Pure Storage, Inc. | Resolving write conflicts in a dispersed storage network |
US9665429B2 (en) | 2014-02-26 | 2017-05-30 | International Business Machines Corporation | Storage of data with verification in a dispersed storage network |
US10592109B2 (en) | 2014-02-26 | 2020-03-17 | Pure Storage, Inc. | Selecting storage resources in a dispersed storage network |
US9529834B2 (en) | 2014-02-26 | 2016-12-27 | International Business Machines Corporation | Concatenating data objects for storage in a dispersed storage network |
US10769016B2 (en) | 2014-02-26 | 2020-09-08 | Pure Storage, Inc. | Storing a plurality of correlated data in a dispersed storage network |
US10140182B2 (en) | 2014-02-26 | 2018-11-27 | International Business Machines Corporation | Modifying allocation of storage resources in a dispersed storage network |
US10628245B2 (en) | 2014-04-02 | 2020-04-21 | Pure Storage, Inc. | Monitoring of storage units in a dispersed storage network |
US20190087599A1 (en) | 2014-04-02 | 2019-03-21 | International Business Machines Corporation | Compressing a slice name listing in a dispersed storage network |
US10761917B2 (en) | 2014-04-02 | 2020-09-01 | Pure Storage, Inc. | Using global namespace addressing in a dispersed storage network |
US10681138B2 (en) | 2014-04-02 | 2020-06-09 | Pure Storage, Inc. | Storing and retrieving multi-format content in a distributed storage network |
US10020826B2 (en) | 2014-04-02 | 2018-07-10 | International Business Machines Corporation | Generating molecular encoding information for data storage |
US10015152B2 (en) | 2014-04-02 | 2018-07-03 | International Business Machines Corporation | Securing data in a dispersed storage network |
US20150288680A1 (en) | 2014-04-02 | 2015-10-08 | Cleversafe, Inc. | Distributing registry information in a dispersed storage network |
US11347590B1 (en) | 2014-04-02 | 2022-05-31 | Pure Storage, Inc. | Rebuilding data in a distributed storage network |
US10296263B2 (en) * | 2014-04-30 | 2019-05-21 | International Business Machines Corporation | Dispersed bloom filter for determining presence of an object |
US10394476B2 (en) | 2014-04-30 | 2019-08-27 | Pure Storage, Inc. | Multi-level stage locality selection on a large system |
US10802732B2 (en) | 2014-04-30 | 2020-10-13 | Pure Storage, Inc. | Multi-level stage locality selection on a large system |
US9965336B2 (en) | 2014-04-30 | 2018-05-08 | International Business Machines Corporation | Delegating iterative storage unit access in a dispersed storage network |
US9735967B2 (en) | 2014-04-30 | 2017-08-15 | International Business Machines Corporation | Self-validating request message structure and operation |
MX364334B (en) * | 2014-05-13 | 2019-04-23 | Cloud Crowding Corp | Distributed secure data storage and transmission of streaming media content. |
US10509577B2 (en) | 2014-06-05 | 2019-12-17 | Pure Storage, Inc. | Reliable storage in a dispersed storage network |
US10095872B2 (en) | 2014-06-05 | 2018-10-09 | International Business Machines Corporation | Accessing data based on a dispersed storage network rebuilding issue |
US9690520B2 (en) | 2014-06-30 | 2017-06-27 | International Business Machines Corporation | Recovering an encoded data slice in a dispersed storage network |
US9841925B2 (en) | 2014-06-30 | 2017-12-12 | International Business Machines Corporation | Adjusting timing of storing data in a dispersed storage network |
US10673946B2 (en) | 2014-06-30 | 2020-06-02 | Pure Storage, Inc. | Using separate weighting scores for different types of data in a decentralized agreement protocol |
US10459797B2 (en) | 2014-06-30 | 2019-10-29 | Pure Storage, Inc. | Making trade-offs between rebuild scanning and failing memory device flexibility |
US10447612B2 (en) | 2014-06-30 | 2019-10-15 | Pure Storage, Inc. | Migrating encoded data slices in a dispersed storage network |
US11606431B2 (en) | 2014-06-30 | 2023-03-14 | Pure Storage, Inc. | Maintaining failure independence for storage of a set of encoded data slices |
US11099763B1 (en) | 2014-06-30 | 2021-08-24 | Pure Storage, Inc. | Migrating generational storage to a decentralized agreement protocol paradigm |
US10440105B2 (en) | 2014-06-30 | 2019-10-08 | Pure Storage, Inc. | Using a decentralized agreement protocol to rank storage locations for target width |
US11398988B1 (en) | 2014-06-30 | 2022-07-26 | Pure Storage, Inc. | Selection of access resources in a distributed storage network |
US10042564B2 (en) | 2014-06-30 | 2018-08-07 | International Business Machines Corporation | Accessing data while migrating storage of the data |
US9838478B2 (en) | 2014-06-30 | 2017-12-05 | International Business Machines Corporation | Identifying a task execution resource of a dispersed storage network |
US10089036B2 (en) | 2014-07-31 | 2018-10-02 | International Business Machines Corporation | Migrating data in a distributed storage network |
US10644874B2 (en) | 2014-07-31 | 2020-05-05 | Pure Storage, Inc. | Limiting brute force attacks against dispersed credentials in a distributed storage system |
US11728964B2 (en) | 2014-07-31 | 2023-08-15 | Pure Storage, Inc. | Performance aided data migration in a distributed storage network |
US10613936B2 (en) | 2014-07-31 | 2020-04-07 | Pure Storage, Inc. | Fractional slices in a distributed storage system |
US10049120B2 (en) * | 2014-09-05 | 2018-08-14 | International Business Machines Corporation | Consistency based access of data in a dispersed storage network |
US10402395B2 (en) | 2014-09-05 | 2019-09-03 | Pure Storage, Inc. | Facilitating data consistency in a dispersed storage network |
US10176191B2 (en) * | 2014-09-05 | 2019-01-08 | International Business Machines Corporation | Recovering from conflicts that emerge from eventually consistent operations |
US11442921B1 (en) | 2014-09-05 | 2022-09-13 | Pure Storage, Inc. | Data access in a dispersed storage network with consistency |
US10146622B2 (en) | 2014-09-08 | 2018-12-04 | International Business Machines Corporation | Combining deduplication with locality for efficient and fast storage |
US10268545B2 (en) | 2014-09-08 | 2019-04-23 | International Business Machines Corporation | Using reinforcement learning to select a DS processing unit |
US9591076B2 (en) | 2014-09-08 | 2017-03-07 | International Business Machines Corporation | Maintaining a desired number of storage units |
DE102014113430A1 (en) * | 2014-09-17 | 2016-03-17 | Bundesdruckerei Gmbh | Distributed data storage using authorization tokens |
US10095582B2 (en) | 2014-10-29 | 2018-10-09 | International Business Machines Corporation | Partial rebuilding techniques in a dispersed storage unit |
US9916114B2 (en) | 2014-10-29 | 2018-03-13 | International Business Machines Corporation | Deterministically sharing a plurality of processing resources |
US10282135B2 (en) | 2014-10-29 | 2019-05-07 | International Business Machines Corporation | Strong consistency write threshold |
US10481833B2 (en) | 2014-10-29 | 2019-11-19 | Pure Storage, Inc. | Transferring data encoding functions in a distributed storage network |
US20180101457A1 (en) | 2014-10-29 | 2018-04-12 | International Business Machines Corporation | Retrying failed write operations in a dispersed storage network |
US10459792B2 (en) | 2014-10-29 | 2019-10-29 | Pure Storage, Inc. | Using an eventually consistent dispersed memory to implement storage tiers |
US10223033B2 (en) | 2014-10-29 | 2019-03-05 | International Business Machines Corporation | Coordinating arrival times of data slices in a dispersed storage network |
US10120739B2 (en) | 2014-12-02 | 2018-11-06 | International Business Machines Corporation | Prioritized data rebuilding in a dispersed storage network |
US10521298B2 (en) | 2014-12-02 | 2019-12-31 | Pure Storage, Inc. | Temporarily storing dropped and rebuilt slices in a DSN memory |
US10503592B2 (en) | 2014-12-02 | 2019-12-10 | Pure Storage, Inc. | Overcoming bottlenecks in partial and traditional rebuild operations |
US10402271B2 (en) | 2014-12-02 | 2019-09-03 | Pure Storage, Inc. | Overcoming bottlenecks in zero information gain (ZIG) rebuild operations |
US9727275B2 (en) | 2014-12-02 | 2017-08-08 | International Business Machines Corporation | Coordinating storage of data in dispersed storage networks |
US10558527B2 (en) | 2014-12-02 | 2020-02-11 | Pure Storage, Inc. | Rebuilding strategy in memory managed multi-site duplication |
US10481832B2 (en) | 2014-12-02 | 2019-11-19 | Pure Storage, Inc. | Applying a probability function to avoid storage operations for already-deleted data |
US10042848B1 (en) | 2014-12-19 | 2018-08-07 | Amazon Technologies, Inc. | Sparse index-based storage, retrieval, and management of stored data |
US9959274B2 (en) * | 2014-12-19 | 2018-05-01 | Amazon Technologies, Inc. | Volume-level redundancy coding techniques for sequential transfer optimized storage devices |
US10656866B2 (en) | 2014-12-31 | 2020-05-19 | Pure Storage, Inc. | Unidirectional vault synchronization to support tiering |
US10423359B2 (en) | 2014-12-31 | 2019-09-24 | Pure Storage, Inc. | Linking common attributes among a set of synchronized vaults |
US10642687B2 (en) | 2014-12-31 | 2020-05-05 | Pure Storage, Inc. | Pessimistic reads and other smart-read enhancements with synchronized vaults |
US10452317B2 (en) | 2014-12-31 | 2019-10-22 | Pure Storage, Inc. | DAP redistribution operation within a dispersed storage network |
US9727427B2 (en) | 2014-12-31 | 2017-08-08 | International Business Machines Corporation | Synchronizing storage of data copies in a dispersed storage network |
US10489247B2 (en) | 2014-12-31 | 2019-11-26 | Pure Storage, Inc. | Generating time-ordered globally unique revision numbers |
US11604707B2 (en) | 2014-12-31 | 2023-03-14 | Pure Storage, Inc. | Handling failures when synchronizing objects during a write operation |
US10387252B2 (en) | 2014-12-31 | 2019-08-20 | Pure Storage, Inc. | Synchronously storing data in a plurality of dispersed storage networks |
US10623495B2 (en) | 2014-12-31 | 2020-04-14 | Pure Storage, Inc. | Keeping synchronized writes from getting out of synch |
US10126974B2 (en) | 2014-12-31 | 2018-11-13 | International Business Machines Corporation | Redistributing encoded data slices in a dispersed storage network |
US10621042B2 (en) | 2014-12-31 | 2020-04-14 | Pure Storage, Inc. | Vault transformation within a dispersed storage network |
US10169123B2 (en) | 2015-01-30 | 2019-01-01 | International Business Machines Corporation | Distributed data rebuilding |
US10511665B2 (en) | 2015-01-30 | 2019-12-17 | Pure Storage, Inc. | Efficient resource reclamation after deletion of slice from common file |
US10498823B2 (en) | 2015-01-30 | 2019-12-03 | Pure Storage, Inc. | Optimally apportioning rebuilding resources |
US10289342B2 (en) | 2015-01-30 | 2019-05-14 | International Business Machines Corporation | Data access optimization protocol in a dispersed storage network |
US9826038B2 (en) | 2015-01-30 | 2017-11-21 | International Business Machines Corporation | Selecting a data storage resource of a dispersed storage network |
US10423490B2 (en) | 2015-01-30 | 2019-09-24 | Pure Storage, Inc. | Read-source requests to support bundled writes in a distributed storage system |
US10620878B2 (en) | 2015-01-30 | 2020-04-14 | Pure Storage, Inc. | Write threshold plus value in dispersed storage network write operations |
US10498822B2 (en) | 2015-01-30 | 2019-12-03 | Pure Storage, Inc. | Adaptive scanning rates |
US10440116B2 (en) | 2015-01-30 | 2019-10-08 | Pure Storage, Inc. | Minimizing data movement through rotation of spare memory devices |
US10802915B2 (en) | 2015-01-30 | 2020-10-13 | Pure Storage, Inc. | Time based storage of encoded data slices |
US10530862B2 (en) | 2015-01-30 | 2020-01-07 | Pure Storage, Inc. | Determining slices to rebuild from low-level failures |
US10506045B2 (en) | 2015-01-30 | 2019-12-10 | Pure Storage, Inc. | Memory access using deterministic function and secure seed |
US9740547B2 (en) | 2015-01-30 | 2017-08-22 | International Business Machines Corporation | Storing data using a dual path storage approach |
US10740180B2 (en) | 2015-01-30 | 2020-08-11 | Pure Storage, Inc. | Storing and retrieving data using proxies |
US10594793B2 (en) | 2015-01-30 | 2020-03-17 | Pure Storage, Inc. | Read-prepare requests to multiple memories |
US10592132B2 (en) | 2015-01-30 | 2020-03-17 | Pure Storage, Inc. | Read-foreign-slices request for improved read efficiency with bundled writes |
US10409772B2 (en) | 2015-02-27 | 2019-09-10 | Pure Storage, Inc. | Accessing serially stored data in a dispersed storage network |
US10069915B2 (en) | 2015-02-27 | 2018-09-04 | International Business Machines Corporation | Storing data in a dispersed storage network |
US10275185B2 (en) | 2015-02-27 | 2019-04-30 | International Business Machines Corporation | Fail-in-place supported via decentralized or Distributed Agreement Protocol (DAP) |
US11188665B2 (en) | 2015-02-27 | 2021-11-30 | Pure Storage, Inc. | Using internal sensors to detect adverse interference and take defensive actions |
US11836369B1 (en) | 2015-02-27 | 2023-12-05 | Pure Storage, Inc. | Storing data in an expanded storage pool of a vast storage network |
US10078472B2 (en) | 2015-02-27 | 2018-09-18 | International Business Machines Corporation | Rebuilding encoded data slices in a dispersed storage network |
US10437676B2 (en) | 2015-02-27 | 2019-10-08 | Pure Storage, Inc. | Urgent reads and using data source health to determine error recovery procedures |
US10404410B2 (en) | 2015-02-27 | 2019-09-03 | Pure Storage, Inc. | Storage unit (SU) report cards |
US10387067B2 (en) | 2015-02-27 | 2019-08-20 | Pure Storage, Inc. | Optimizing data storage in a dispersed storage network |
US10423502B2 (en) | 2015-02-27 | 2019-09-24 | Pure Storage, Inc. | Stand-by distributed storage units |
US10503591B2 (en) | 2015-02-27 | 2019-12-10 | Pure Storage, Inc. | Selecting retrieval locations in a dispersed storage network |
US10579451B2 (en) | 2015-02-27 | 2020-03-03 | Pure Storage, Inc. | Pro-actively preparing a dispersed storage network memory for higher-loads |
US10437677B2 (en) | 2015-02-27 | 2019-10-08 | Pure Storage, Inc. | Optimized distributed rebuilding within a dispersed storage network |
US10440115B2 (en) | 2015-02-27 | 2019-10-08 | Pure Storage, Inc. | Write intent messaging in a dispersed storage network |
US10528425B2 (en) | 2015-02-27 | 2020-01-07 | Pure Storage, Inc. | Transitioning to an optimized data storage approach in a dispersed storage network |
US10534668B2 (en) | 2015-02-27 | 2020-01-14 | Pure Storage, Inc. | Accessing data in a dispersed storage network |
US10530861B2 (en) | 2015-02-27 | 2020-01-07 | Pure Storage, Inc. | Utilizing multiple storage pools in a dispersed storage network |
US10657000B2 (en) | 2015-02-27 | 2020-05-19 | Pure Storage, Inc. | Optimizing data storage in a dispersed storage network |
US10852957B2 (en) | 2015-03-31 | 2020-12-01 | Pure Storage, Inc. | Migration agent employing moveslice request |
US10963180B2 (en) | 2015-03-31 | 2021-03-30 | Pure Storage, Inc. | Adding incremental storage resources in a dispersed storage network |
US10387070B2 (en) | 2015-03-31 | 2019-08-20 | Pure Storage, Inc. | Migrating data in response to adding incremental storage resources in a dispersed storage network |
US10331384B2 (en) | 2015-03-31 | 2019-06-25 | International Business Machines Corporation | Storing data utilizing a maximum accessibility approach in a dispersed storage network |
US10713374B2 (en) | 2015-03-31 | 2020-07-14 | Pure Storage, Inc. | Resolving detected access anomalies in a dispersed storage network |
US10534661B2 (en) | 2015-03-31 | 2020-01-14 | Pure Storage, Inc. | Selecting a storage error abatement alternative in a dispersed storage network |
US10528282B2 (en) | 2015-03-31 | 2020-01-07 | Pure Storage, Inc. | Modifying and utilizing a file structure in a dispersed storage network |
US10079887B2 (en) | 2015-03-31 | 2018-09-18 | International Business Machines Corporation | Expanding storage capacity of a set of storage units in a distributed storage network |
US10437515B2 (en) | 2015-03-31 | 2019-10-08 | Pure Storage, Inc. | Selecting storage units in a dispersed storage network |
US11055177B2 (en) | 2015-03-31 | 2021-07-06 | Pure Storage, Inc. | Correlating operational information with an error condition in a dispersed storage network |
US10915261B2 (en) | 2015-03-31 | 2021-02-09 | Pure Storage, Inc. | Selecting a set of storage units in a distributed storage network |
US10963430B2 (en) | 2015-04-01 | 2021-03-30 | Dropbox, Inc. | Shared workspaces with selective content item synchronization |
US9922201B2 (en) | 2015-04-01 | 2018-03-20 | Dropbox, Inc. | Nested namespaces for selective content sharing |
US10078561B2 (en) | 2015-04-30 | 2018-09-18 | International Business Machines Corporation | Handling failing memory devices in a dispersed storage network |
US10168904B2 (en) | 2015-04-30 | 2019-01-01 | International Business Machines Corporation | Quasi-error notifications in a dispersed storage network |
US10067998B2 (en) | 2015-04-30 | 2018-09-04 | International Business Machines Corporation | Distributed sync list |
US10037171B2 (en) | 2015-04-30 | 2018-07-31 | International Business Machines Corporation | Accessing common data in a dispersed storage network |
US10216594B2 (en) | 2015-04-30 | 2019-02-26 | International Business Machines Corporation | Automated stalled process detection and recovery |
US10268376B2 (en) | 2015-04-30 | 2019-04-23 | International Business Machines Corporation | Automated deployment and assignment of access devices in a dispersed storage network |
US10055170B2 (en) | 2015-04-30 | 2018-08-21 | International Business Machines Corporation | Scheduling storage unit maintenance tasks in a dispersed storage network |
US10254992B2 (en) | 2015-04-30 | 2019-04-09 | International Business Machines Corporation | Rebalancing data storage in a dispersed storage network |
US10157094B2 (en) | 2015-04-30 | 2018-12-18 | International Business Machines Corporation | Validating system registry files in a dispersed storage network |
US10891058B2 (en) | 2015-05-29 | 2021-01-12 | Pure Storage, Inc. | Encoding slice verification information to support verifiable rebuilding |
US10789128B2 (en) | 2015-05-29 | 2020-09-29 | Pure Storage, Inc. | External healing mode for a dispersed storage network memory |
US10613798B2 (en) | 2015-05-29 | 2020-04-07 | Pure Storage, Inc. | Slice fanout write request |
US11115221B2 (en) | 2015-05-29 | 2021-09-07 | Pure Storage, Inc. | Verifying a rebuilt encoded data slice using slice verification information |
US10409522B2 (en) | 2015-05-29 | 2019-09-10 | Pure Storage, Inc. | Reclaiming storage capacity in a dispersed storage network |
US10523241B2 (en) | 2015-05-29 | 2019-12-31 | Pure Storage, Inc. | Object fan out write operation |
US10430107B2 (en) | 2015-05-29 | 2019-10-01 | Pure Storage, Inc. | Identifying stored data slices during a slice migration activity in a dispersed storage network |
US10324657B2 (en) | 2015-05-29 | 2019-06-18 | International Business Machines Corporation | Accounting for data whose rebuilding is deferred |
US10169125B2 (en) | 2015-05-29 | 2019-01-01 | International Business Machines Corporation | Re-encoding data in a dispersed storage network |
US10838664B2 (en) * | 2015-05-29 | 2020-11-17 | Pure Storage, Inc. | Determining a storage location according to legal requirements |
US10402122B2 (en) | 2015-05-29 | 2019-09-03 | Pure Storage, Inc. | Transferring encoded data slices in a dispersed storage network |
US10437671B2 (en) | 2015-06-30 | 2019-10-08 | Pure Storage, Inc. | Synchronizing replicated stored data |
US11669546B2 (en) | 2015-06-30 | 2023-06-06 | Pure Storage, Inc. | Synchronizing replicated data in a storage network |
US10055291B2 (en) | 2015-06-30 | 2018-08-21 | International Business Machines Corporation | Method and system for processing data access requests during data transfers |
US10229004B2 (en) | 2015-07-31 | 2019-03-12 | International Business Machines Corporation | Data transfer priority levels |
US11782789B2 (en) | 2015-07-31 | 2023-10-10 | Pure Storage, Inc. | Encoding data and associated metadata in a storage network |
US10466914B2 (en) | 2015-08-31 | 2019-11-05 | Pure Storage, Inc. | Verifying authorized access in a dispersed storage network |
US10073652B2 (en) | 2015-09-24 | 2018-09-11 | International Business Machines Corporation | Performance optimized storage vaults in a dispersed storage network |
US9571573B1 (en) * | 2015-10-29 | 2017-02-14 | Dropbox, Inc. | Peer-to-peer synchronization protocol for multi-premises hosting of digital content items |
US10691718B2 (en) | 2015-10-29 | 2020-06-23 | Dropbox, Inc. | Synchronization protocol for multi-premises hosting of digital content items |
US10169147B2 (en) | 2015-10-30 | 2019-01-01 | International Business Machines Corporation | End-to-end secure data storage in a dispersed storage network |
US10423362B2 (en) | 2015-11-30 | 2019-09-24 | International Business Machines Corporation | Utilizing multiple dispersal algorithms to encode data for storage in a dispersed storage network |
US10409514B2 (en) | 2015-11-30 | 2019-09-10 | International Business Machines Corporation | IP multicast message transmission for event notifications |
US20170192688A1 (en) | 2015-12-30 | 2017-07-06 | International Business Machines Corporation | Lazy deletion of vaults in packed slice storage (pss) and zone slice storage (zss) |
CN105677250B (en) * | 2016-01-04 | 2019-07-12 | 北京百度网讯科技有限公司 | The update method and updating device of object data in object storage system |
US10855759B2 (en) | 2016-01-26 | 2020-12-01 | Pure Storage, Inc. | Utilizing a hierarchical index in a dispersed storage network |
US9537952B1 (en) | 2016-01-29 | 2017-01-03 | Dropbox, Inc. | Apparent cloud access for hosted content items |
US10089178B2 (en) | 2016-02-29 | 2018-10-02 | International Business Machines Corporation | Developing an accurate dispersed storage network memory performance model through training |
US10931402B2 (en) | 2016-03-15 | 2021-02-23 | Cloud Storage, Inc. | Distributed storage system data management and security |
MX2018011241A (en) | 2016-03-15 | 2018-11-22 | Datomia Res Labs Ou | Distributed storage system data management and security. |
US10387248B2 (en) | 2016-03-29 | 2019-08-20 | International Business Machines Corporation | Allocating data for storage by utilizing a location-based hierarchy in a dispersed storage network |
US10831381B2 (en) | 2016-03-29 | 2020-11-10 | International Business Machines Corporation | Hierarchies of credential and access control sharing between DSN memories |
US10419538B2 (en) | 2016-04-26 | 2019-09-17 | International Business Machines Corporation | Selecting memory for data access in a dispersed storage network |
US10169082B2 (en) | 2016-04-27 | 2019-01-01 | International Business Machines Corporation | Accessing data in accordance with an execution deadline |
US10007444B2 (en) | 2016-04-29 | 2018-06-26 | International Business Machines Corporation | Batching access requests in a dispersed storage network |
US10628399B2 (en) | 2016-04-29 | 2020-04-21 | International Business Machines Corporation | Storing data in a dispersed storage network with consistency |
US10091298B2 (en) | 2016-05-27 | 2018-10-02 | International Business Machines Corporation | Enhancing performance of data storage in a dispersed storage network |
US10122795B2 (en) | 2016-05-31 | 2018-11-06 | International Business Machines Corporation | Consistency level driven data storage in a dispersed storage network |
US10353772B2 (en) | 2016-05-31 | 2019-07-16 | International Business Machines Corporation | Selecting data for storage in a dispersed storage network |
US10027755B2 (en) | 2016-06-01 | 2018-07-17 | International Business Machines Corporation | Selecting storage units in one or more dispersed storage networks |
US10394650B2 (en) | 2016-06-03 | 2019-08-27 | International Business Machines Corporation | Multiple writes using inter-site storage unit relationship |
US10719499B2 (en) | 2016-06-06 | 2020-07-21 | INTERNATIONAL BUSINESS MACHINES CORPORATIOb | Establishing distributed consensus via alternate voting strategies in a dispersed storage network |
US10652350B2 (en) | 2016-06-06 | 2020-05-12 | International Business Machines Corporation | Caching for unique combination reads in a dispersed storage network |
US10735545B2 (en) | 2016-06-06 | 2020-08-04 | International Business Machines Corporation | Routing vault access requests in a dispersed storage network |
US10334045B2 (en) | 2016-06-06 | 2019-06-25 | International Business Machines Corporation | Indicating multiple encoding schemes in a dispersed storage network |
US10007438B2 (en) | 2016-06-25 | 2018-06-26 | International Business Machines Corporation | Method and system for achieving consensus using alternate voting strategies (AVS) with incomplete information |
US10564852B2 (en) | 2016-06-25 | 2020-02-18 | International Business Machines Corporation | Method and system for reducing memory device input/output operations |
US10235085B2 (en) | 2016-06-27 | 2019-03-19 | International Business Machines Corporation | Relocating storage unit data in response to detecting hotspots in a dispersed storage network |
US11115469B2 (en) | 2016-06-28 | 2021-09-07 | International Business Machines Corporation | Efficient updates within a dispersed storage network |
US10025505B2 (en) | 2016-06-29 | 2018-07-17 | International Business Machines Corporation | Accessing data in a dispersed storage network during write operations |
US10157021B2 (en) | 2016-06-29 | 2018-12-18 | International Business Machines Corporation | Processing incomplete data access transactions |
US10387286B2 (en) | 2016-06-30 | 2019-08-20 | International Business Machines Corporation | Managing configuration updates in a dispersed storage network |
US9934092B2 (en) | 2016-07-12 | 2018-04-03 | International Business Machines Corporation | Manipulating a distributed agreement protocol to identify a desired set of storage units |
US10114696B2 (en) | 2016-07-14 | 2018-10-30 | International Business Machines Corporation | Tracking data access in a dispersed storage network |
US10534666B2 (en) | 2016-07-14 | 2020-01-14 | International Business Machines Corporation | Determining storage requirements based on licensing right in a dispersed storage network |
US10102067B2 (en) | 2016-07-14 | 2018-10-16 | International Business Machines Corporation | Performing a desired manipulation of an encoded data slice based on a metadata restriction and a storage operational condition |
US9992063B2 (en) | 2016-07-18 | 2018-06-05 | International Business Machines Corporation | Utilizing reallocation via a decentralized, or distributed, agreement protocol (DAP) for storage unit (SU) replacement |
US10360103B2 (en) | 2016-07-18 | 2019-07-23 | International Business Machines Corporation | Focused storage pool expansion to prevent a performance degradation |
US10277490B2 (en) | 2016-07-19 | 2019-04-30 | International Business Machines Corporation | Monitoring inter-site bandwidth for rebuilding |
US10769015B2 (en) | 2016-07-19 | 2020-09-08 | International Business Machines Corporation | Throttling access requests at different layers of a DSN memory |
US10554752B2 (en) | 2016-07-20 | 2020-02-04 | International Business Machines Corporation | Efficient transfer of encoded data slice sets to new or alternate storage units |
US10031809B2 (en) | 2016-07-20 | 2018-07-24 | International Business Machines Corporation | Efficient method for rebuilding a set of encoded data slices |
US10127112B2 (en) | 2016-07-20 | 2018-11-13 | International Business Machines Corporation | Assigning prioritized rebuild resources optimally |
US10459796B2 (en) | 2016-07-20 | 2019-10-29 | International Business Machines Corporation | Prioritizing rebuilding based on a longevity estimate of the rebuilt slice |
US10416930B2 (en) | 2016-07-21 | 2019-09-17 | International Business Machines Corporation | Global access permit listing |
US10379744B2 (en) | 2016-07-21 | 2019-08-13 | International Business Machines Corporation | System for collecting end-user feedback and usability metrics |
US10459790B2 (en) | 2016-07-26 | 2019-10-29 | International Business Machines Corporation | Elastic storage in a dispersed storage network |
US10395043B2 (en) | 2016-07-29 | 2019-08-27 | International Business Machines Corporation | Securely storing data in an elastically scalable dispersed storage network |
US10031805B2 (en) | 2016-08-09 | 2018-07-24 | International Business Machines Corporation | Assigning slices to storage locations based on a predicted lifespan |
US10223036B2 (en) | 2016-08-10 | 2019-03-05 | International Business Machines Corporation | Expanding a dispersed storage network (DSN) |
US10129023B2 (en) | 2016-08-11 | 2018-11-13 | International Business Machines Corporation | Enhancing security for multiple storage configurations |
US10348829B2 (en) | 2016-08-15 | 2019-07-09 | International Business Machines Corporation | Auto indexing with customizable metadata |
US10013309B2 (en) | 2016-08-17 | 2018-07-03 | International Business Machines Corporation | Missing slice reconstruction in a dispersed storage network |
US10078468B2 (en) | 2016-08-18 | 2018-09-18 | International Business Machines Corporation | Slice migration in a dispersed storage network |
US10379778B2 (en) | 2016-08-18 | 2019-08-13 | International Business Machines Corporation | Using a master encryption key to sanitize a dispersed storage network memory |
US10389683B2 (en) | 2016-08-26 | 2019-08-20 | International Business Machines Corporation | Securing storage units in a dispersed storage network |
US10379773B2 (en) | 2016-08-29 | 2019-08-13 | International Business Machines Corporation | Storage unit for use in a dispersed storage network |
US10581807B2 (en) | 2016-08-29 | 2020-03-03 | International Business Machines Corporation | Using dispersal techniques to securely store cryptographic resources and respond to attacks |
US10061524B2 (en) | 2016-09-01 | 2018-08-28 | International Business Machines Corporation | Wear-leveling of memory devices |
US10169149B2 (en) | 2016-09-06 | 2019-01-01 | International Business Machines Corporation | Standard and non-standard dispersed storage network data access |
US10387079B2 (en) | 2016-09-09 | 2019-08-20 | International Business Machines Corporation | Placement of dispersed storage data based on requestor properties |
US10225271B2 (en) | 2016-09-09 | 2019-03-05 | International Business Machines Corporation | Distributed storage network with enhanced security monitoring |
US10547615B2 (en) | 2016-09-12 | 2020-01-28 | International Business Machines Corporation | Security response protocol based on security alert encoded data slices of a distributed storage network |
US10558396B2 (en) | 2016-09-14 | 2020-02-11 | International Business Machines Corporation | Pre-caching data according to a current or predicted requester location |
US10558389B2 (en) | 2016-09-20 | 2020-02-11 | International Business Machines Corporation | Per-storage class quality of service (QoS) management within a distributed storage network (DSN) where the DSN stores data using dispersed storage error decoding/encoding |
US10067822B2 (en) | 2016-09-26 | 2018-09-04 | International Business Machines Corporation | Combined slice objects in alternate memory locations |
US10448062B2 (en) | 2016-10-26 | 2019-10-15 | International Business Machines Corporation | Pre-fetching media content to reduce peak loads |
US10394630B2 (en) | 2016-10-26 | 2019-08-27 | International Business Machines Corporation | Estimating relative data importance in a dispersed storage network |
US10585751B2 (en) | 2016-10-27 | 2020-03-10 | International Business Machines Corporation | Partial rebuild operation within a dispersed storage network including local memory and cloud-based alternative memory |
US10481977B2 (en) | 2016-10-27 | 2019-11-19 | International Business Machines Corporation | Dispersed storage of error encoded data objects having multiple resolutions |
US11169731B2 (en) | 2016-10-31 | 2021-11-09 | International Business Machines Corporation | Managing storage resources in a dispersed storage network |
US10540247B2 (en) | 2016-11-10 | 2020-01-21 | International Business Machines Corporation | Handling degraded conditions using a redirect module |
US10585607B2 (en) | 2016-11-10 | 2020-03-10 | International Business Machines Corporation | Determining an optimum selection of functions for units in a DSN memory |
US10114698B2 (en) | 2017-01-05 | 2018-10-30 | International Business Machines Corporation | Detecting and responding to data loss events in a dispersed storage network |
US10782921B2 (en) | 2017-01-25 | 2020-09-22 | International Business Machines Corporation | Non-writing device finalization of a write operation initiated by another device |
US10180787B2 (en) | 2017-02-09 | 2019-01-15 | International Business Machines Corporation | Dispersed storage write process with lock/persist |
US10241865B2 (en) | 2017-02-15 | 2019-03-26 | International Business Machines Corporation | Handling storage unit failure in a dispersed storage network |
US10579309B2 (en) | 2017-02-16 | 2020-03-03 | International Business Machines Corporation | Method for increasing throughput in a distributed storage network |
US10248495B2 (en) | 2017-02-17 | 2019-04-02 | International Business Machines Corporation | Eventual consistency intent cleanup in a dispersed storage network |
US10552341B2 (en) | 2017-02-17 | 2020-02-04 | International Business Machines Corporation | Zone storage—quickly returning to a state of consistency following an unexpected event |
US10382553B2 (en) | 2017-02-20 | 2019-08-13 | International Business Machines Corporation | Zone storage—resilient and efficient storage transactions |
US10394468B2 (en) | 2017-02-23 | 2019-08-27 | International Business Machines Corporation | Handling data slice revisions in a dispersed storage network |
US10241677B2 (en) | 2017-02-24 | 2019-03-26 | International Business Machines Corporation | Ensuring consistency between content and metadata with intents |
US9998147B1 (en) | 2017-02-27 | 2018-06-12 | International Business Machines Corporation | Method for using write intents in a distributed storage network |
US10642532B2 (en) | 2017-02-28 | 2020-05-05 | International Business Machines Corporation | Storing data sequentially in zones in a dispersed storage network |
US10372380B2 (en) | 2017-03-01 | 2019-08-06 | International Business Machines Corporation | Asserting integrity with a verifiable codec |
US10169392B2 (en) | 2017-03-08 | 2019-01-01 | International Business Machines Corporation | Persistent data structures on a dispersed storage network memory |
US11226980B2 (en) | 2017-03-13 | 2022-01-18 | International Business Machines Corporation | Replicating containers in object storage using intents |
US10235241B2 (en) | 2017-03-15 | 2019-03-19 | International Business Machines Corporation | Method for partial updating data content in a distributed storage network |
US10693640B2 (en) | 2017-03-17 | 2020-06-23 | International Business Machines Corporation | Use of key metadata during write and read operations in a dispersed storage network memory |
US10241861B2 (en) | 2017-03-23 | 2019-03-26 | International Business Machines Corporation | Method for tenant isolation in a distributed computing system |
US10133634B2 (en) | 2017-03-30 | 2018-11-20 | International Business Machines Corporation | Method for performing in-place disk format changes in a distributed storage network |
US10360391B2 (en) | 2017-04-03 | 2019-07-23 | International Business Machines Corporation | Verifiable keyed all-or-nothing transform |
US10379961B2 (en) | 2017-04-11 | 2019-08-13 | International Business Machines Corporation | Ensuring metadata and index consistency using write intents |
US10545699B2 (en) | 2017-04-11 | 2020-01-28 | International Business Machines Corporation | Dynamic retention policies and optional deletes |
CN108733311B (en) * | 2017-04-17 | 2021-09-10 | 伊姆西Ip控股有限责任公司 | Method and apparatus for managing storage system |
US10567509B2 (en) | 2017-05-15 | 2020-02-18 | International Business Machines Corporation | Rebuilding derived content |
US10491386B2 (en) | 2017-06-01 | 2019-11-26 | International Business Machines Corporation | Slice-level keyed encryption with support for efficient rekeying |
US10339003B2 (en) | 2017-06-01 | 2019-07-02 | International Business Machines Corporation | Processing data access transactions in a dispersed storage network using source revision indicators |
US10467097B2 (en) | 2017-06-02 | 2019-11-05 | International Business Machines Corporation | Indicating data health in a DSN memory |
US10372381B2 (en) | 2017-06-05 | 2019-08-06 | International Business Machines Corporation | Implicit leader election in a distributed storage network |
US10361813B2 (en) | 2017-06-16 | 2019-07-23 | International Business Machine Corporation | Using slice routers for improved storage placement determination |
US10534548B2 (en) | 2017-06-20 | 2020-01-14 | International Business Machines Corporation | Validating restricted operations on a client using trusted environments |
US10324855B2 (en) | 2017-06-23 | 2019-06-18 | International Business Machines Corporation | Associating a processing thread and memory section to a memory device |
US10594790B2 (en) | 2017-06-28 | 2020-03-17 | International Business Machines Corporation | Data compression in a dispersed storage network |
US10540111B2 (en) | 2017-06-28 | 2020-01-21 | International Business Machines Corporation | Managing data container instances in a dispersed storage network |
US10599502B2 (en) | 2017-08-07 | 2020-03-24 | International Business Machines Corporation | Fault detection and recovery in a distributed storage network |
US10509699B2 (en) | 2017-08-07 | 2019-12-17 | International Business Machines Corporation | Zone aware request scheduling and data placement |
US10671746B2 (en) | 2017-08-28 | 2020-06-02 | International Business Machines Corporation | Controlling access when processing intents in a dispersed storage network |
US10379942B2 (en) | 2017-09-27 | 2019-08-13 | International Business Machines Corporation | Efficient transfer of objects between containers on the same vault |
US10585748B2 (en) | 2017-09-29 | 2020-03-10 | International Business Machines Corporation | Scalable cloud—assigning scores to requesters and treating requests differently based on those scores |
US10409661B2 (en) | 2017-09-29 | 2019-09-10 | International Business Machines Corporation | Slice metadata for optimized dispersed storage network memory storage strategies |
US10802713B2 (en) | 2017-09-29 | 2020-10-13 | International Business Machines Corporation | Requester-associated storage entity data |
US10540120B2 (en) | 2017-11-14 | 2020-01-21 | International Business Machines Corporation | Contention avoidance on associative commutative updates |
US10423497B2 (en) | 2017-11-28 | 2019-09-24 | International Business Machines Corporation | Mechanism for representing system configuration changes as a series of objects writable to an object storage container |
US10565392B2 (en) | 2017-11-28 | 2020-02-18 | International Business Machines Corporation | Secure and verifiable update operations |
US10785194B2 (en) | 2017-12-07 | 2020-09-22 | International Business Machines Corporation | Processing intents using trusted entities in a dispersed storage network |
US10681135B2 (en) | 2017-12-08 | 2020-06-09 | International Business Machines Corporation | Generating slices from a broadcast message and a recipient identity |
US11412041B2 (en) | 2018-06-25 | 2022-08-09 | International Business Machines Corporation | Automatic intervention of global coordinator |
US10936452B2 (en) | 2018-11-14 | 2021-03-02 | International Business Machines Corporation | Dispersed storage network failover units used to improve local reliability |
US11108559B2 (en) * | 2019-01-02 | 2021-08-31 | International Business Machines Corporation | Producing proof of receipt, existence and other data provenance evidence |
EP3918484A4 (en) | 2019-01-29 | 2023-03-29 | Cloud Storage, Inc. | Encoding and storage node repairing method for minimum storage regenerating codes for distributed storage systems |
US11288396B2 (en) | 2019-06-18 | 2022-03-29 | International Business Machines Corporation | Data security through physical separation of data |
US11290531B2 (en) | 2019-12-04 | 2022-03-29 | Dropbox, Inc. | Immediate cloud content item creation from local file system interface |
US11593026B2 (en) | 2020-03-06 | 2023-02-28 | International Business Machines Corporation | Zone storage optimization using predictive protocol patterns |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020166079A1 (en) * | 2001-01-29 | 2002-11-07 | Ulrich Thomas R. | Dynamic data recovery |
US20040024963A1 (en) * | 2002-08-05 | 2004-02-05 | Nisha Talagala | Method and system for striping data to accommodate integrity metadata |
US20050114594A1 (en) * | 2003-11-24 | 2005-05-26 | Corbett Peter F. | Semi-static distribution technique |
Family Cites Families (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4092732A (en) | 1977-05-31 | 1978-05-30 | International Business Machines Corporation | System for recovering data stored in failed memory unit |
US5485474A (en) | 1988-02-25 | 1996-01-16 | The President And Fellows Of Harvard College | Scheme for information dispersal and reconstruction |
US5987622A (en) | 1993-12-10 | 1999-11-16 | Tm Patents, Lp | Parallel computer system including parallel storage subsystem including facility for correction of data in the event of failure of a storage device in parallel storage subsystem |
US5848230A (en) | 1995-05-25 | 1998-12-08 | Tandem Computers Incorporated | Continuously available computer memory systems |
US5809285A (en) | 1995-12-21 | 1998-09-15 | Compaq Computer Corporation | Computer system having a virtual drive array controller |
US6012159A (en) | 1996-01-17 | 2000-01-04 | Kencast, Inc. | Method and system for error-free data transfer |
US5890156A (en) | 1996-05-02 | 1999-03-30 | Alcatel Usa, Inc. | Distributed redundant database |
US6058454A (en) | 1997-06-09 | 2000-05-02 | International Business Machines Corporation | Method and system for automatically configuring redundant arrays of disk memory devices |
US6088330A (en) | 1997-09-09 | 2000-07-11 | Bruck; Joshua | Reliable array of distributed computing nodes |
US5991414A (en) | 1997-09-12 | 1999-11-23 | International Business Machines Corporation | Method and apparatus for the secure distributed storage and retrieval of information |
US6272658B1 (en) | 1997-10-27 | 2001-08-07 | Kencast, Inc. | Method and system for reliable broadcasting of data files and streams |
JPH11167443A (en) | 1997-12-02 | 1999-06-22 | Casio Comput Co Ltd | Interface device |
US6374336B1 (en) | 1997-12-24 | 2002-04-16 | Avid Technology, Inc. | Computer system and process for transferring multiple high bandwidth streams of data between multiple storage units and multiple applications in a scalable and reliable manner |
US6415373B1 (en) | 1997-12-24 | 2002-07-02 | Avid Technology, Inc. | Computer system and process for transferring multiple high bandwidth streams of data between multiple storage units and multiple applications in a scalable and reliable manner |
WO2000011554A1 (en) | 1998-08-19 | 2000-03-02 | Boris Nikolaevich Vilkov | A system and method for identification of transformation of memory device addresses |
US6356949B1 (en) * | 1999-01-29 | 2002-03-12 | Intermec Ip Corp. | Automatic data collection device that receives data output instruction from data consumer |
US6609223B1 (en) | 1999-04-06 | 2003-08-19 | Kencast, Inc. | Method for packet-level fec encoding, in which on a source packet-by-source packet basis, the error correction contributions of a source packet to a plurality of wildcard packets are computed, and the source packet is transmitted thereafter |
US6826711B2 (en) | 2000-02-18 | 2004-11-30 | Avamar Technologies, Inc. | System and method for data protection with multidimensional parity |
US7103915B2 (en) | 2000-11-13 | 2006-09-05 | Digital Doors, Inc. | Data security system and method |
US7146644B2 (en) | 2000-11-13 | 2006-12-05 | Digital Doors, Inc. | Data security system and method responsive to electronic attacks |
US7140044B2 (en) | 2000-11-13 | 2006-11-21 | Digital Doors, Inc. | Data security system and method for separation of user communities |
US8176563B2 (en) * | 2000-11-13 | 2012-05-08 | DigitalDoors, Inc. | Data security system and method with editor |
US6785783B2 (en) | 2000-11-30 | 2004-08-31 | International Business Machines Corporation | NUMA system with redundant main memory architecture |
WO2002065275A1 (en) | 2001-01-11 | 2002-08-22 | Yottayotta, Inc. | Storage virtualization system and methods |
US6879596B1 (en) | 2001-04-11 | 2005-04-12 | Applied Micro Circuits Corporation | System and method for systolic array sorting of information segments |
US7024609B2 (en) | 2001-04-20 | 2006-04-04 | Kencast, Inc. | System for protecting the transmission of live data streams, and upon reception, for reconstructing the live data streams and recording them into files |
GB2377049A (en) * | 2001-06-30 | 2002-12-31 | Hewlett Packard Co | Billing for utilisation of a data storage array |
US7003688B1 (en) | 2001-11-15 | 2006-02-21 | Xiotech Corporation | System and method for a reserved memory area shared by all redundant storage controllers |
US7171493B2 (en) | 2001-12-19 | 2007-01-30 | The Charles Stark Draper Laboratory | Camouflage of network traffic to resist attack |
WO2004012337A2 (en) | 2002-07-29 | 2004-02-05 | Robert Halford | Multi-dimensional data protection and mirroring method for micro level data |
US8332483B2 (en) * | 2003-12-15 | 2012-12-11 | International Business Machines Corporation | Apparatus, system, and method for autonomic control of grid system resources |
US7206899B2 (en) | 2003-12-29 | 2007-04-17 | Intel Corporation | Method, system, and program for managing data transfer and construction |
US7240236B2 (en) | 2004-03-23 | 2007-07-03 | Archivas, Inc. | Fixed content distributed data storage using permutation ring encoding |
JP4446839B2 (en) | 2004-08-30 | 2010-04-07 | 株式会社日立製作所 | Storage device and storage management device |
US7386758B2 (en) | 2005-01-13 | 2008-06-10 | Hitachi, Ltd. | Method and apparatus for reconstructing data in object-based storage arrays |
US7672930B2 (en) * | 2005-04-05 | 2010-03-02 | Wal-Mart Stores, Inc. | System and methods for facilitating a linear grid database with data organization by dimension |
US7953937B2 (en) | 2005-09-30 | 2011-05-31 | Cleversafe, Inc. | Systems, methods, and apparatus for subdividing data for storage in a dispersed data storage grid |
US7546427B2 (en) | 2005-09-30 | 2009-06-09 | Cleversafe, Inc. | System for rebuilding dispersed data |
US7574579B2 (en) | 2005-09-30 | 2009-08-11 | Cleversafe, Inc. | Metadata management system for an information dispersed storage system |
US7574570B2 (en) | 2005-09-30 | 2009-08-11 | Cleversafe Inc | Billing system for information dispersal system |
US7721157B2 (en) * | 2006-03-08 | 2010-05-18 | Omneon Video Networks | Multi-node computer system component proactive monitoring and proactive repair |
-
2006
- 2006-04-13 US US11/404,071 patent/US7574579B2/en active Active
-
2007
- 2007-03-22 WO PCT/US2007/007285 patent/WO2007120437A2/en active Application Filing
-
2009
- 2009-07-07 US US12/499,013 patent/US8019960B2/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020166079A1 (en) * | 2001-01-29 | 2002-11-07 | Ulrich Thomas R. | Dynamic data recovery |
US20040024963A1 (en) * | 2002-08-05 | 2004-02-05 | Nisha Talagala | Method and system for striping data to accommodate integrity metadata |
US20050114594A1 (en) * | 2003-11-24 | 2005-05-26 | Corbett Peter F. | Semi-static distribution technique |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9026844B2 (en) | 2008-09-02 | 2015-05-05 | Qando Services Inc. | Distributed storage and communication |
WO2010057173A3 (en) * | 2008-11-17 | 2010-10-07 | Unisys Corporation | Storage communities of interest using cryptographic splitting |
WO2011007141A1 (en) | 2009-07-17 | 2011-01-20 | Extas Global Ltd. | Distributed storage |
US10324849B2 (en) | 2017-02-13 | 2019-06-18 | International Business Machines Corporation | Headless resilient backup and restore software ecosystem selecting server based on historical client latency, workload and metadata presence in cache |
US10817427B2 (en) | 2017-02-13 | 2020-10-27 | International Business Machines Corporation | Headless resilient backup and restore software ecosystem and with a first backup server and a second backup server in a plurality of backup servers, with the first backup server selecting a the second backup server to start a job where selection is based on historical client latency, scheduled backup server workload, and whether metadata is cached in any of the plurality of backup servers |
Also Published As
Publication number | Publication date |
---|---|
US8019960B2 (en) | 2011-09-13 |
US20070079083A1 (en) | 2007-04-05 |
WO2007120437A3 (en) | 2009-04-02 |
US7574579B2 (en) | 2009-08-11 |
US20100023529A1 (en) | 2010-01-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7574579B2 (en) | Metadata management system for an information dispersed storage system | |
US7818518B2 (en) | System for rebuilding dispersed data | |
US8140777B2 (en) | Billing system for information dispersal system | |
US20220368457A1 (en) | Distributed Storage System Data Management And Security | |
US8244991B2 (en) | Systems, methods, and apparatus for subdividing data for storage in a dispersed data storage grid | |
Sookhak et al. | Remote data auditing in cloud computing environments: a survey, taxonomy, and open issues | |
CN110602147B (en) | Data encryption safe storage method, system and storage medium based on cloud platform | |
EP3688955B1 (en) | Secure storage of data through encryption and segmentation | |
US20110213940A1 (en) | Virtualized data storage vaults on a dispersed data storage network | |
JP2009512077A (en) | Method and system for data backup | |
Frank et al. | Percival: A searchable secret-split datastore | |
Bhavya et al. | DDEAS: Distributed Deduplication System with Efficient Access in Cloud Data Storage | |
Storer et al. | POTSHARDS: Secure Long-Term Archival Storage Without Encryption | |
Frank | Percival: A Reliable, Long-Term, Distributed Storage System Free of Fixed-Key Encryption | |
Chaitanya et al. | Middleware for a re-configurable distributed archival store based on secret sharing | |
Rokade et al. | SDDMCSS: Secure and Data De-duplicated Multi-Cloud Storage System | |
Pochhi et al. | A Survey on Secure and Authorized Data Deduplication | |
Shaju et al. | Hybrid Cloud Approach for Reliable Authorized Deduplication | |
Gangathade et al. | Review on Secure System With Improved Reliability Using Distributed Deduplication | |
Appliance | POTSHARDS: Secure Long-Term Storage Without Encryption |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 07753876 Country of ref document: EP Kind code of ref document: A2 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 07753876 Country of ref document: EP Kind code of ref document: A2 |