US20050216813A1 - Fixed content distributed data storage using permutation ring encoding - Google Patents
Fixed content distributed data storage using permutation ring encoding Download PDFInfo
- Publication number
- US20050216813A1 US20050216813A1 US10/806,488 US80648804A US2005216813A1 US 20050216813 A1 US20050216813 A1 US 20050216813A1 US 80648804 A US80648804 A US 80648804A US 2005216813 A1 US2005216813 A1 US 2005216813A1
- Authority
- US
- United States
- Prior art keywords
- data
- code
- blocks
- matrix
- permutation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M13/00—Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
- H03M13/03—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words
- H03M13/05—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits
- H03M13/11—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits using multiple parity bits
- H03M13/1102—Codes on graphs and decoding on graphs, e.g. low-density parity check [LDPC] codes
- H03M13/1191—Codes on graphs other than LDPC codes
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M13/00—Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
- H03M13/03—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words
- H03M13/05—Error detection or forward error correction by redundancy in data representation, i.e. code words containing more digits than the source words using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits
- H03M13/13—Linear codes
- H03M13/15—Cyclic codes, i.e. cyclic shifts of codewords produce other codewords, e.g. codes defined by a generator polynomial, Bose-Chaudhuri-Hocquenghem [BCH] codes
- H03M13/151—Cyclic codes, i.e. cyclic shifts of codewords produce other codewords, e.g. codes defined by a generator polynomial, Bose-Chaudhuri-Hocquenghem [BCH] codes using error location or error correction polynomials
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2211/00—Indexing scheme relating to details of data-processing equipment not covered by groups G06F3/00 - G06F13/00
- G06F2211/10—Indexing scheme relating to G06F11/10
- G06F2211/1002—Indexing scheme relating to G06F11/1076
- G06F2211/1028—Distributed, i.e. distributed RAID systems with parity
Definitions
- the present invention relates generally to techniques for highly available, reliable, and persistent data storage in a distributed computer network.
- RAIN Redundant Array of Independent Nodes
- a representative archive comprises storage nodes that provide the long-term data storage, and access nodes that provide the interface through which data files enter the archive.
- file protection schemes include simple file mirroring, RAID-5 schemes that spread the file contents across multiple nodes using a recovery stripe to recreate any missing stripes, or variations on RAID 5 that use multiple recovery stripes to ensure that simultaneous node failures do not lead to overall system failure.
- IDA Information Dispersal Algorithm
- Rabin IDA itself is a variant of a Reed-Solomon error correcting code, such as a linear block code used to ensure data integrity during transmission over a communications channel.
- Rabin IDA breaks apart a data file so that the pieces can be distributed to multiple sites for fault tolerance without compromising the integrity of the data.
- IDA uses matrix algebra over finite fields to disperse the information of a file F into n pieces that are transmitted or stored on n different machines (or disks) such that the contents of the original file F can be reconstructed from the contents of any m of its pieces, where m ⁇ n. Because of the way in which the data is broken up, only a subset of the original pieces are required to reassemble the original data. In IDA, an important objective is to ensure integrity of the dispersed data, and this is accomplished by ensuring that each fragment of the data is not usable; in of itself, to recover the original data.
- Tornado codes are similar to Reed-Solomon codes in that an input file is represented by K input symbols and is used to determine N output symbols, where N is fixed before the encoding process begins.
- a file is partitioned into a set of equal size fragments (called data nodes)
- a set of check nodes that are equal in size and population are then created.
- the encoding of the file involves a series of specially designed bipartite graphs. Each check node is assigned two or more nodes to be its neighbors, and the contents of the check node is set to be the bit-wise XOR of the value of its neighbors.
- the nodes are sequentially numbered, and the encoded file is distributed containing one or more nodes.
- Decoding is symmetric to the encoding process, except that the check nodes are used to restore their neighbors.
- To restore a missing node the contents of the check node is XORed with the contents of certain neighbor nodes, and the resulting value is assigned to the missing neighbor.
- Tornado codes provide certain advantages but also have limitations. Among other issues, a graph is specific to a file size, so a new graph needs to be generated for each file size used. Furthermore, the graphs needed by the Tornado codes are complicated to construct, and they require different custom settings of parameters for different sized files to obtain the best performance. These graphs are usually quite large and require a significant amount of memory for their storage.
- an encoder uses an input file of data and a key to produce an output symbol.
- An output symbol with key I is generated by determining a weight, W(I), for the output symbol to be generated, selecting W(I) of the input symbols associated with the output symbol according to a function of I, and generating the output symbol's value B(I) from a predetermined value function F(I) of the selected W(I) input symbols.
- An encoder can be called repeatedly to generate multiple output symbols.
- the output symbols are generally independent of each other, and an unbounded number (subject to the resolution of I) can be generated, if needed.
- a decoder receives some or all of the output symbols generated.
- the number of output symbols needed to decode an input file is equal to, or slightly greater than, the number of input symbols comprising the file, assuming that input symbols and output symbols represent the same number of bits of data are then created. This approach is said to provide certain advantages over Tornado or other Reed-Solomon based coding techniques.
- RAIN Redundant Array of Independent Nodes
- a still further and more specific object of the present invention is to implement a file protection scheme for fixed content in a distributed data archive using matrix computations that leverage a permutation operation that comprises a superposition of cycle permutations.
- an N+K coding technique for use to protect data that is being distributed in a redundant array of independent nodes (RAIN).
- the data itself may be of any type, and it may also include system metadata.
- the data to be distributed is encoded by a dispersal operation that uses a group of permutation ring operators.
- the dispersal operation is effected using a matrix of the form [I N — C] where IN is an n ⁇ n identity sub-matrix and C is a k ⁇ n sub-matrix of code blocks.
- the identity sub-matrix is used to preserve the original data.
- the sub-matrix C preferably comprises a set of permutation ring operators that are used to generate the code blocks.
- the operators are preferably “polynomials” that are selected from a group ring of a permutation group with base ring Z 2 , e.g., a set of permutations whose action on the data is taken to be the XOR of the actions of the individual permutations
- Each code block is preferably stored on a separate node.
- an N+K (4,2) coding scheme is implemented.
- a file to be archived comprises four (4) data blocks (A 1 , A 2 , A 3 , A 4 ).
- a dispersal matrix comprises six (6) code blocks (C 0 , C 1 , C 2 , C 3 , C 4 , C 5 ). Because the identity sub-matrix is used, however, the first four code blocks (C 0 , C 1 , C 2 , C 3 ) are just copies of the first four data blocks, and these data blocks 8 are then stored in four distinct nodes of the array.
- the sub-matrix C is then generated as follows.
- g is a permutation operator that comprises a polynomial of cyclic permutations, such as: b 0 *c 0 +b 1 *c 1 +b k c k + . . . b (m-1) *c (m-1) , where b k is a bit (0 or 1), c 0 is the identity (“do nothing to the data”), and c k is a cycle operation c repeated k times, e.g., the operation: “cycle the data k words.”
- the C 4 code block is then stored in the 5 th node, and the C 5 code block is stored in the 6 th node to complete the encoding process.
- FIG. 1 is a simplified block diagram of a fixed content storage archive in which the file protection scheme of the present invention may be implemented;
- FIG. 2 is a simplified representation of an N+K coding algorithm that underlies the theory of operation of the file protection scheme of the invention
- FIG. 3 is an illustrative de-convolution operation according to the present invention.
- the present invention preferably is implemented in a scalable disk-based archival storage management system, preferably a system architecture based on a heterogeneous redundant array of independent nodes.
- enterprises can create permanent storage for fixed content information such as documents, e-mail, satellite images, diagnostic images, check images, voice recordings, video, and the like, among others.
- High levels of reliability are achieved by replicating data on independent servers, or so-called storage nodes.
- each node is symmetric with its peers. Thus, because any given node can perform all functions, the failure of any one node has little impact on the archive's availability.
- a physical boundary of an individual archive is referred to herein as a cluster.
- a cluster is not a single device, but rather a collection of devices. Devices may be homogeneous or heterogeneous.
- a typical device is a computer or machine running an operating system such as Linux.
- Clusters of Linux-based systems hosted on commodity hardware provide an archive that can be scaled from a few storage node servers to many nodes that store thousands of terabytes of data. This architecture ensures that storage capacity can always keep pace with an organization's increasing archive requirements.
- data is replicated across the cluster so that the archive is always protected from device failure. If a disk or node fails, the cluster automatically fails over to other nodes in the cluster that maintain replicas of the same data.
- An illustrative cluster preferably comprises the following general categories of components: nodes 102 , a pair of network switches 104 , power distribution units (PDUs) 106 , and uninterruptible power supplies (UPSs) 108 .
- a node 102 typically comprises one or more commodity servers and contains a CPU (e.g., Intel x 86 , suitable random access memory (RAM), one or more hard drives (e.g., standard IDE/SATA, SCSI, or the like), and two or more network interface (NIC) cards.
- a typical node is a 2 U rack mounted unit with a 2.4 GHz chip, 512 MB RAM, and six (6) 200 GB hard drives.
- the network switches 104 typically include an internal switch 105 that enables peer-to-peer communication between nodes, and an external switch 107 that allows extra-cluster access to each node. Each switch requires enough ports to handle all potential nodes in a cluster. Ethernet or GigE switches may be used for this purpose. PDUs 106 are used to power all nodes and switches, and the UPSs 108 are used that protect all nodes and switches.
- a cluster is connectable to a network, such as the publicly-routed Internet, an enterprise intranet, or other wide area or local area network. In an illustrative embodiment, the cluster is implemented within an enterprise computing environment.
- the cluster's domain may be a new sub-domain of an existing domain.
- the sub-domain is delegated in the corporate DNS server to the name servers in the cluster itself.
- End users access the cluster using any conventional interface or access tool.
- access to the cluster may be effected over any IP-based protocol (HTTP, FTP, NFS, SMB, or the like), via an application programming interface (API), or through any other known or later-developed access method, service, program or tool.
- HTTP IP-based protocol
- FTP FTP
- NFS non-based protocol
- SMB application programming interface
- API application programming interface
- the cluster stores metadata for each file as well as its content.
- This metadata preferably is maintained in a database that is distributed evenly among all nodes in the cluster.
- each node includes a metadata manager 110 .
- metadata manager 110 When new nodes are added to the cluster, individual node responsibilities are adjusted to the new capacity; this includes redistributing metadata across all nodes so that new members assume an equal share.
- other node metadata managers compensate for the reduced capacity by assuming a greater share.
- metadata information preferably is replicated across multiple nodes, where each node is directly responsible for managing some percentage of all cluster metadata, and copies this data to a set number of other nodes.
- a file being inserted into the cluster may be authenticated in any convenient manner, e.g., by assigning a digital signature that is generated from the actual content of the file.
- the archive can periodically check the authenticity of the stored file's content by regenerating this signature from the stored content and comparing it to the original signature. The signatures must match in order to verify data authenticity.
- n is the number of original data blocks
- k the number of code blocks that can be lost with the original data still recoverable
- r representing the extra code blocks needed that contain additional redundant information.
- the simplest scheme is to just store c copies of the data. This is a (n, c ⁇ 1,(c ⁇ 1)(n ⁇ 1)) scheme, as any c ⁇ 1 lost blocks can be recovered, but c*n blocks must be stored. This type of scheme has a large redundancy cost.
- any scheme that reduces r from (c ⁇ 1)(n ⁇ 1) of the “copying” scheme involves using code blocks that somehow mix the data.
- any such mixing can be thought of in terms of a matrix product such as illustrated in FIG. 2 .
- G is a t by n matrix
- A is an n column vector (the data blocks).
- the g's can be any functions acting on the code blocks, and f a function that acts on these intermediate results to mix them, producing the C i elements.
- each g ij in the matrix is a superposition (preferably by XOR) of a given number of permutations of a code block.
- the “f” operation preferably remains XOR, although this is not a limitation.
- the operators “g” are members of a “group ring” over a group of permutations of the blocks, with a base ring Z 2-residues mod 2 .
- the g's in the matrix preferably are based on superpositions of powers of a single cyclic permutation.
- g be a “polynomial”: b 0 *c 0 +b 1 *c 1 +b k c k + . . . b (m-1) *c (m-1) , where b k is a bit (0 or 1), c 0 is the identity (“do nothing to the data”), and c k is a cycle operation c repeated k times, e.g., the operation: “cycle the data k words right.”
- the “+” operand here is just addition mod 2, and the action of g on a data block is then calculated by using the distributive law.
- the operator can be identified by just a string of bits.
- the first bit may be the identity, which corresponds to the c 0 case.
- these bit strings can be written as integers, e.g., 1+c becomes 3, with 9 (binary 1001) represents the operator “XOR the unshifted copy with a copy of the data shifted 3 over.”
- the string (a 1 +a m ) (a 2 +a 1 ) . . . (a m +a (m-1) ) resulting from the operation of (1+c) may be considered a “convolution” of A.
- the matrix (1+c) is almost invertible; indeed, given a small constant amount of additional information, one can invert the operation. Generally, how much extra information is required varies if c is changed. For instance, if a permutation d is used that just swaps even and odd words of A, the (1+d) element requires more information to invert (i.e., out of every two words, one is needed).
- the permutation operators preferably are sums of powers of a given cyclic permutation, where the length of the sum is less (preferably much less) than the number of words cycled (which may vary with the available buffer size). Such operators typically are only practically invertible when the sums contain one permutation.
- the de-convolution operation typically proceeds as described above, but using a v-word window to produce an additional decoded word. The window is then slid over one word and the process continues until complete.
- a key is stored separately for each data block whose decoding requires de-convolution.
- the small size of each key means that its storage is not burdensome.
- the keys can be replicated k times and stored with metadata, or the keys for given data blocks can be appended to other data blocks, which other data blocks preferably are chosen to insure that they are not missing in the case that a particular sub-matrix inversion requiring a specific de-convolution needs to be performed.
- the “g” operators are restricted to permutation operators, with each operator being a superposition of a small number of permutations of a code block.
- This criteria is insured by selecting a n+k by n matrix, all of whose n by n sub-matrices have non-zero determinants.
- the matrix should be chosen so that the encode and decode operations are as computationally-efficient as possible. For the N+K case (4,2), the following matrix may be used:
- this code takes four (4) data blocks (A 1 , A 2 , A 3 , A 4 ) and produces six (6) code blocks (C 0 , C 1 , C 2 , C 3 , C 4 , C 5 ). Because the identity sub-matrix is used, however, the first four code blocks (C 0 , C 1 , C 2 , C 3 ) are just copies of the first four data blocks, which is a desirable result. As indicated above, it is preferred to have as much of the data as possible freely available and to only use the checksum pieces to reconstruct any of the original data that may be unavailable. The use of the identity sub-matrix ensures this desirable property.
- the C 4 code block is then A 1 +cA 2 +A 3 +cA 4 ; the [1335] row generates the C 5 code block as: A 1 +(1+c)A 2 +(1+c)A 3 +(1+c 2 )A 4 .
- the decode operation (using de-convolution as needed) will work as long only the determinant is non-zero.
- an implementation of the invention does not require use of Gaussian elimination schematically as it is described above; rather, preferably the specific operations and the order of those operations used to invert each case are optimized so as to run most efficiently on the preferred computer hardware used.
- c can be selected to be a permutation with a short cycle length, e.g., permute every 4 words in a cycle.
- a short cycle length e.g., permute every 4 words in a cycle.
- more elements become invertible.
- the string [7 13 14] would be invertible, as would the string [1 2 4 8]. This property can be exploited to obviate de-convolution, but potential at a cost of more XOR operations.
- a useful matrix should have the property of being invertible, with as few operations as possible.
- a suitable matrix inversion method such as Gaussian elimination can be used, with the variation that not all elements have inverses.
- a diagonalized sub-matrix will not necessarily have the identity element in the column solved for; thus, de-convolution is used.
- the operations used for both coding and decoding are just bitwise XOR and, as needed, de-convolution (which is itself preferably based on XOR).
- the cycling operation itself just requires some pointer arithmetic and/or adjusting the beginning or end of the result block.
- the number of XOR operations is essentially given by the number of non-zero bits involved in operators for the operations, with each “1” bit in an operator calling for another copy of a data block.
- the present invention may be readily implemented in software, firmware, in special purpose hardware, or in any appropriate combination.
- the symbolic inversion steps are identified for each loss case given that matrix.
- the loss cases can then be readily implemented in code, and encode and decode processes can then be called as routines as needed to the archival and retrieval processes.
- an NK coder is described for use to protect data that is being distributed in an RAIN archive.
- the data itself may be of any type, and it may also include the archive metadata.
- the data to be distributed is encoded by a matrix operation that uses an identity sub-matrix to preserve the data words, and that uses permutation ring operators to generate the code words.
- the operators are preferably polynomials that are selected from a group ring of a permutation group with base ring Z 2 .
- f( ) is preferably addition mod 2 (i.e., XOR)
- g( ) is a permutation operator, such as a polynomial of cyclic permutations.
- the invention is desirable as most operators are very fast.
- decode (1+s)(A) (((a 1 +a 3 )(a 2 +a 1 )(a 3 +a 2 ).
- a de-convolution example is shown in FIG. 3 .
- a data file that is being stored in the cluster comprises a set of N data blocks that are stored in N respective nodes.
- the coding process in of itself does not require the data file to be broken down in this manner, however.
- the encoding technique described above is also useful in protecting against loss and enhancing speed of transmission on communication paths of information represented as data signals on such paths.
- the technique is useful for dispersal and reconstruction of information during communication or storage and retrieval.
- the present invention also relates to apparatus for performing the operations herein.
- This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer.
- a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
Abstract
Description
- 1. Technical Field
- The present invention relates generally to techniques for highly available, reliable, and persistent data storage in a distributed computer network.
- 2. Description of the Related Art
- A need has developed for the archival storage of “fixed content” in a highly available, reliable and persistent manner that replaces or supplements traditional tape and optical storage solutions. The term “fixed content” typically refers to any type of digital information that is expected to be retained without change for reference or other purposes. Examples of such fixed content include, among many others, e-mail, documents, diagnostic images, check images, voice recordings, film and video, and the like. The traditional Redundant Array of Independent Nodes (RAIN) storage approach has emerged as the architecture of choice for creating large online archives for the storage of such fixed content information assets. By allowing nodes to join and exit from a cluster as needed, RAIN architectures insulate a storage cluster from the failure of any one or more nodes. By replicating data on multiple nodes, RAIN-type archives can automatically compensate for node failure or removal. Typically, RAIN systems are largely delivered as hardware appliances designed from identical components within a closed system.
- A representative archive comprises storage nodes that provide the long-term data storage, and access nodes that provide the interface through which data files enter the archive. To protect files, typically one of several possible schemes are used. These well-known file protection schemes include simple file mirroring, RAID-5 schemes that spread the file contents across multiple nodes using a recovery stripe to recreate any missing stripes, or variations on RAID 5 that use multiple recovery stripes to ensure that simultaneous node failures do not lead to overall system failure. One such variation is the Information Dispersal Algorithm (IDA), original developed by Rabin and described in U.S. Pat. No. 5,485,474. Rabin IDA itself is a variant of a Reed-Solomon error correcting code, such as a linear block code used to ensure data integrity during transmission over a communications channel.
- Rabin IDA breaks apart a data file so that the pieces can be distributed to multiple sites for fault tolerance without compromising the integrity of the data. In particular, IDA uses matrix algebra over finite fields to disperse the information of a file F into n pieces that are transmitted or stored on n different machines (or disks) such that the contents of the original file F can be reconstructed from the contents of any m of its pieces, where m≦n. Because of the way in which the data is broken up, only a subset of the original pieces are required to reassemble the original data. In IDA, an important objective is to ensure integrity of the dispersed data, and this is accomplished by ensuring that each fragment of the data is not usable; in of itself, to recover the original data. This requirement is undesirable, as it is preferred to have as much of the data as possible freely available (as there may be no loss during transmission or storage), so that the checksum pieces are only used to reconstruct any of the original data that may be unavailable. Moreover, while Rabin IDA provides fault tolerance and data security, it is not computationally efficient, especially as the size of the file increases.
- To address this problem, other types of error correcting codes with smaller computational requirements were developed. Tornado codes are similar to Reed-Solomon codes in that an input file is represented by K input symbols and is used to determine N output symbols, where N is fixed before the encoding process begins. In this approach, after a file is partitioned into a set of equal size fragments (called data nodes), a set of check nodes that are equal in size and population are then created. The encoding of the file involves a series of specially designed bipartite graphs. Each check node is assigned two or more nodes to be its neighbors, and the contents of the check node is set to be the bit-wise XOR of the value of its neighbors. The nodes are sequentially numbered, and the encoded file is distributed containing one or more nodes. Decoding is symmetric to the encoding process, except that the check nodes are used to restore their neighbors. To restore a missing node, the contents of the check node is XORed with the contents of certain neighbor nodes, and the resulting value is assigned to the missing neighbor. Tornado codes provide certain advantages but also have limitations. Among other issues, a graph is specific to a file size, so a new graph needs to be generated for each file size used. Furthermore, the graphs needed by the Tornado codes are complicated to construct, and they require different custom settings of parameters for different sized files to obtain the best performance. These graphs are usually quite large and require a significant amount of memory for their storage.
- Still another approach to the problem of protecting content in distributed storage is described in U.S. Pat. No. 6,614,366, to Luby et al, which also purports to address limitations and deficiencies in Tornado coding. In this patent, an encoder uses an input file of data and a key to produce an output symbol. An output symbol with key I is generated by determining a weight, W(I), for the output symbol to be generated, selecting W(I) of the input symbols associated with the output symbol according to a function of I, and generating the output symbol's value B(I) from a predetermined value function F(I) of the selected W(I) input symbols. An encoder can be called repeatedly to generate multiple output symbols. The output symbols are generally independent of each other, and an unbounded number (subject to the resolution of I) can be generated, if needed. A decoder receives some or all of the output symbols generated. The number of output symbols needed to decode an input file is equal to, or slightly greater than, the number of input symbols comprising the file, assuming that input symbols and output symbols represent the same number of bits of data are then created. This approach is said to provide certain advantages over Tornado or other Reed-Solomon based coding techniques.
- While the approaches described above are representative of the prior art and can provide fault tolerant and secure storage, there remains a need to improve the state of the art, especially as it relates to the problem of reliable and secure storage of fixed content, especially across heterogeneous RAIN archives.
- It is a general object of the present invention to provide for highly available, reliable and persistent storage of fixed content in an archive.
- It is another object of the invention to provide an improved file protection scheme for fault tolerant and secure storage of a fixed content in a Redundant Array of Independent Nodes (RAIN) architecture, and preferably an architecture that does not require homogenous machines or devices.
- It is still another object of the invention to provide a novel file protection scheme for fixed content in a heterogeneous RAIN archive that overcomes the deficiencies of prior art approaches and that is computationally efficient in both encoding and decoding operations.
- A still further and more specific object of the present invention is to implement a file protection scheme for fixed content in a distributed data archive using matrix computations that leverage a permutation operation that comprises a superposition of cycle permutations.
- According to the present invention, an N+K coding technique is described for use to protect data that is being distributed in a redundant array of independent nodes (RAIN). The data itself may be of any type, and it may also include system metadata. According to the invention, the data to be distributed is encoded by a dispersal operation that uses a group of permutation ring operators. In a preferred embodiment, the dispersal operation is effected using a matrix of the form [IN
— C] where IN is an n×n identity sub-matrix and C is a k×n sub-matrix of code blocks. The identity sub-matrix is used to preserve the original data. The sub-matrix C preferably comprises a set of permutation ring operators that are used to generate the code blocks. The operators are preferably “polynomials” that are selected from a group ring of a permutation group with base ring Z2, e.g., a set of permutations whose action on the data is taken to be the XOR of the actions of the individual permutations The ith code block is computed as: Ci=f(gi1(A1), . . . gin(An)), where f( ) is preferably addition mod 2 (i.e., XOR), and g( ) is a permutation operator as described above. Each code block is preferably stored on a separate node. - In a more specific embodiment, an N+K (4,2) coding scheme is implemented. In this case, a file to be archived comprises four (4) data blocks (A1, A2, A3, A4). A dispersal matrix comprises six (6) code blocks (C0, C1, C2, C3, C4, C5). Because the identity sub-matrix is used, however, the first four code blocks (C0, C1, C2, C3) are just copies of the first four data blocks, and these data blocks 8 are then stored in four distinct nodes of the array. The sub-matrix C is then generated as follows. Assume that g is a permutation operator that comprises a polynomial of cyclic permutations, such as: b0*c0+b1*c1+bkck+ . . . b(m-1)*c(m-1), where bk is a bit (0 or 1), c0 is the identity (“do nothing to the data”), and ck is a cycle operation c repeated k times, e.g., the operation: “cycle the data k words.” The ith code block is then computed as: Ci=f(gi1(A1), . . . gin(An)). The C4 code block is then stored in the 5th node, and the C5 code block is stored in the 6th node to complete the encoding process.
- The foregoing has outlined some of the more pertinent features of the invention. These features should be construed to be merely illustrative. Many other beneficial results can be attained by applying the disclosed invention in a different manner or by modifying the invention as will be described.
- For a more complete understanding of the present invention and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:
-
FIG. 1 is a simplified block diagram of a fixed content storage archive in which the file protection scheme of the present invention may be implemented; -
FIG. 2 is a simplified representation of an N+K coding algorithm that underlies the theory of operation of the file protection scheme of the invention; -
FIG. 3 is an illustrative de-convolution operation according to the present invention. - The present invention preferably is implemented in a scalable disk-based archival storage management system, preferably a system architecture based on a heterogeneous redundant array of independent nodes. Using the present invention, enterprises can create permanent storage for fixed content information such as documents, e-mail, satellite images, diagnostic images, check images, voice recordings, video, and the like, among others. High levels of reliability are achieved by replicating data on independent servers, or so-called storage nodes. Preferably, each node is symmetric with its peers. Thus, because any given node can perform all functions, the failure of any one node has little impact on the archive's availability.
- In a representative embodiment, the invention is implemented in an archive that is designed to capture, preserve, manage, and retrieve digital assets. In an illustrated embodiment of
FIG. 1 , a physical boundary of an individual archive is referred to herein as a cluster. Typically, a cluster is not a single device, but rather a collection of devices. Devices may be homogeneous or heterogeneous. A typical device is a computer or machine running an operating system such as Linux. Clusters of Linux-based systems hosted on commodity hardware provide an archive that can be scaled from a few storage node servers to many nodes that store thousands of terabytes of data. This architecture ensures that storage capacity can always keep pace with an organization's increasing archive requirements. Preferably, data is replicated across the cluster so that the archive is always protected from device failure. If a disk or node fails, the cluster automatically fails over to other nodes in the cluster that maintain replicas of the same data. - An illustrative cluster preferably comprises the following general categories of components:
nodes 102, a pair of network switches 104, power distribution units (PDUs) 106, and uninterruptible power supplies (UPSs) 108. Anode 102 typically comprises one or more commodity servers and contains a CPU (e.g., Intel x86, suitable random access memory (RAM), one or more hard drives (e.g., standard IDE/SATA, SCSI, or the like), and two or more network interface (NIC) cards. A typical node is a 2 U rack mounted unit with a 2.4 GHz chip, 512 MB RAM, and six (6) 200 GB hard drives. The network switches 104 typically include aninternal switch 105 that enables peer-to-peer communication between nodes, and anexternal switch 107 that allows extra-cluster access to each node. Each switch requires enough ports to handle all potential nodes in a cluster. Ethernet or GigE switches may be used for this purpose.PDUs 106 are used to power all nodes and switches, and theUPSs 108 are used that protect all nodes and switches. Although not meant to be limiting, typically a cluster is connectable to a network, such as the publicly-routed Internet, an enterprise intranet, or other wide area or local area network. In an illustrative embodiment, the cluster is implemented within an enterprise computing environment. It may be reached, for example, by navigating through a site's corporate domain name system (DNS) name server. Thus, for example, the cluster's domain may be a new sub-domain of an existing domain. In a representative implementation, the sub-domain is delegated in the corporate DNS server to the name servers in the cluster itself. End users access the cluster using any conventional interface or access tool. Thus, for example, access to the cluster may be effected over any IP-based protocol (HTTP, FTP, NFS, SMB, or the like), via an application programming interface (API), or through any other known or later-developed access method, service, program or tool. - The cluster stores metadata for each file as well as its content. This metadata preferably is maintained in a database that is distributed evenly among all nodes in the cluster. To this end, each node includes a
metadata manager 110. When new nodes are added to the cluster, individual node responsibilities are adjusted to the new capacity; this includes redistributing metadata across all nodes so that new members assume an equal share. Conversely, when a node fails or is removed from the cluster, other node metadata managers compensate for the reduced capacity by assuming a greater share. To prevent data loss, metadata information preferably is replicated across multiple nodes, where each node is directly responsible for managing some percentage of all cluster metadata, and copies this data to a set number of other nodes. - Protection of data in the archive requires a data protection scheme. Although simple techniques such as RAID-1 (mirroring) and RAID-S (parity) may be implemented, the present invention implements a new N+K protection scheme, as will be described below. To prevent data corruption and/or sabotage, a file being inserted into the cluster may be authenticated in any convenient manner, e.g., by assigning a digital signature that is generated from the actual content of the file. The archive can periodically check the authenticity of the stored file's content by regenerating this signature from the stored content and comparing it to the original signature. The signatures must match in order to verify data authenticity.
- The novel data protection scheme of the invention is now described. By way of brief background, it is well known that the goal of block erasure codes is to encode data into blocks so that, even if a certain number of blocks are lost, the original data is recoverable. Block erasure schemes are typically characterized by three (3) parameters (n,k,r), where: n is the number of original data blocks, n+k+r=t is the total number of code blocks, k is the number of code blocks that can be lost with the original data still recoverable, and r representing the extra code blocks needed that contain additional redundant information. For example, the simplest scheme is to just store c copies of the data. This is a (n, c−1,(c−1)(n−1)) scheme, as any c−1 lost blocks can be recovered, but c*n blocks must be stored. This type of scheme has a large redundancy cost.
- For a desired n and k, a useful scheme has the following properties: minimize r (ideally r=0); efficient encode and decode operations, and implementation of the scheme in a systematic manner, in particular, wherein data blocks stored in clear, making decoding trivial if nothing is lost. The prototype “r=0” scheme for k=1 is a block parity scheme, where the blocks are taken to be bit vectors that are XORed together (i.e., the ith bit of the result is the XOR or the ith bit of each of the vectors). The result is stored as an additional code block C{n+1}. Then, if any one block is lost, the information can be recovered by simply XORing the remaining blocks with C{n+1}. This operation cancels out the contributions of the remaining block, leaving the original block.
- Any scheme that reduces r from (c−1)(n−1) of the “copying” scheme involves using code blocks that somehow mix the data. In general, any such mixing can be thought of in terms of a matrix product such as illustrated in
FIG. 2 . In this example, G is a t by n matrix, and A is an n column vector (the data blocks). This matrix product produces t(=n+k+r) code blocks. The ith code block is computed as: Ci=f(gi1(A1), . . . gin(An)). Here, the g's can be any functions acting on the code blocks, and f a function that acts on these intermediate results to mix them, producing the Ci elements. If this were a normal matrix computation over integers, the action of gij is a multiplication and the action off is an addition. There is no reason, however, that these operations must be normal multiplications and additions. If one considers that the individual blocks are usually themselves long strings of information that may be mixed by the operators gij, each of the gij can itself be considered as a matrix, operating at a finer resolution. Similarly, one may move from any (non-block) erasure code to a block erasure code just by grouping the fine scale operations into blocks. Of course, designing the code to take advantage of the block nature of the problem may produce computational savings. Decoding proceeds by inversion in the usual manner. - The present invention provides an improved data protection scheme wherein the “g” operation is instantiated by a permutation operator. In an illustrated embodiment, each gij in the matrix is a superposition (preferably by XOR) of a given number of permutations of a code block. The “f” operation preferably remains XOR, although this is not a limitation. Thus, formally, the operators “g” are members of a “group ring” over a group of permutations of the blocks, with a base ring Z2-residues mod 2. In an illustrative embodiment, the g's in the matrix preferably are based on superpositions of powers of a single cyclic permutation. In a simple example, let c be the permutation “cycle right.” Thus, c acts on a block A=ai . . . am as follows: c(a1 . . . am)→am a1 . . . a(m-1).
- More generally, let g be a “polynomial”: b0*c0+b1*c1+bkck+ . . . b(m-1)*c(m-1), where bk is a bit (0 or 1), c0 is the identity (“do nothing to the data”), and ck is a cycle operation c repeated k times, e.g., the operation: “cycle the data k words right.” The “+” operand here is just
addition mod 2, and the action of g on a data block is then calculated by using the distributive law. For instance, if g=c0+c1=1+c, then the action of g on A is as follows:
where the last string is a string of words, each being the XOR of two words of a1. Because the coefficients b0 . . . bm are just 0 or 1, the operator can be identified by just a string of bits. For ease of illustration, the first bit may be the identity, which corresponds to the c0 case. To further compress this representation, these bit strings can be written as integers, e.g., 1+c becomes 3, with 9 (binary 1001) represents the operator “XOR the unshifted copy with a copy of the data shifted 3 over.” - Because the operators are members of a group ring, the operations of addition and multiplication are meaningful. In particular, cj*ck=c(j+k), as multiplication is the group multiplication (intuitively the operator “cycle j right of (cycle k right)” is equivalent to “cycle j+k right”). The addition operation is simply
addition mod 2. For “polynomials” algorithms for multiplication and addition are as expected. In the bit-notation for operators, addition is XOR, and multiplication is the normal bitwise multiplication algorithm, except that the additions are all done without carry (simple XORs). Thus: 5*6=101*110=101*10+101*100=11110=110*1+110*100=6*5. - The string (a1+am) (a2+a1) . . . (am+a(m-1)) resulting from the operation of (1+c) may be considered a “convolution” of A. A cannot be recovered from this string. If, however, there is also one “key” word of A available, a de-convolution operation can be performed to enable the whole string to be recovered. For example, assume A*=(1+c)A, and the word a1 is available. The ith word of A* is a*i=(ai+a(i-1)). By XORing a*i, the first word A*, with a1, the value am can be recovered: a*1+a1=a1+am+a1=am. The recovered value of am can then be used to recover a(m-1) in turn: a*m+am=am+a (m-1)+am=a(m-1). This “unzipping” process is then continued to recover the rest of the unencoded words of A in descending sequence. Formally, the elements of the group ring are not all invertible. This is seen above, as there is no operator that produces A from (1+c)A. The matrix (1+c), however, is almost invertible; indeed, given a small constant amount of additional information, one can invert the operation. Generally, how much extra information is required varies if c is changed. For instance, if a permutation d is used that just swaps even and odd words of A, the (1+d) element requires more information to invert (i.e., out of every two words, one is needed).
- In an illustrated embodiment, the permutation operators preferably are sums of powers of a given cyclic permutation, where the length of the sum is less (preferably much less) than the number of words cycled (which may vary with the available buffer size). Such operators typically are only practically invertible when the sums contain one permutation. To invert a sum of two or more powers of permutations cp
— 1+cp— m, a key of v words is needed, where v=pm−p1−1 is one less than the difference between the largest and smallest power in the sum. For example, as illustrated above, 3=1+c requires a one work key to unzip, as does 6=c+c2, as both largest and smallest powers are one larger. The value 9=1+c3 requires a 3 word key to de-convolve. - For any operator, the de-convolution operation typically proceeds as described above, but using a v-word window to produce an additional decoded word. The window is then slid over one word and the process continues until complete.
- A key is stored separately for each data block whose decoding requires de-convolution. The small size of each key means that its storage is not burdensome. In a representative embodiment, the keys can be replicated k times and stored with metadata, or the keys for given data blocks can be appended to other data blocks, which other data blocks preferably are chosen to insure that they are not missing in the case that a particular sub-matrix inversion requiring a specific de-convolution needs to be performed.
- With the above as background, a preferred coding technique is now described. As noted above, according to the invention the “g” operators are restricted to permutation operators, with each operator being a superposition of a small number of permutations of a code block. Of course, it is desirable to choose a matrix of operators such that appropriate sub-matrices are invertible when blocks are lost, except for possibly normally uninvertible individual operators, which may be inverted by using the de-convolution procedure described above. This criteria is insured by selecting a n+k by n matrix, all of whose n by n sub-matrices have non-zero determinants. Further, preferably the matrix should be chosen so that the encode and decode operations are as computationally-efficient as possible. For the N+K case (4,2), the following matrix may be used:
-
- 1000
- 0100
- 0010
- 0001
- 1214
- 1335
- As can be seen, this code takes four (4) data blocks (A1, A2, A3, A4) and produces six (6) code blocks (C0, C1, C2, C3, C4, C5). Because the identity sub-matrix is used, however, the first four code blocks (C0, C1, C2, C3) are just copies of the first four data blocks, which is a desirable result. As indicated above, it is preferred to have as much of the data as possible freely available and to only use the checksum pieces to reconstruct any of the original data that may be unavailable. The use of the identity sub-matrix ensures this desirable property. Given the [1214] row, the C4 code block is then A1+cA2+A3+cA4; the [1335] row generates the C5 code block as:
A1+(1+c)A2+(1+c)A3+(1+c2)A4. - Further, during encoding “keys” consisting of the first few words of selected blocks are stored appropriately.
- To decode, it would be desirable to be able to invert the operation encoded by all 2×2 sub-matrices formed by choosing two columns from the two code rows. For these matrices to be invertible without de-convolution, however, the determinant must be an invertible element of the group ring. If
rows 2 and 4 are used, the following sub-matrix is obtained: -
- 24
- 35
- This sub-matrix has a determinant: 2*5+3*4=10+12=6 (given that addition, including in the multiplication operation, is XOR). The result is not invertible, as over the operator 6 has two permutations. Indeed, to isolate one row using Gaussian elimination, one must first form two new rows, multiplying (in the ring) each by the least common multiple of the first element in each row, divided by that element. In this example, this gives: 3(2 4)=(6 12) for the first row, and 2(3 5)=(6 10) for the second row; adding these two rows together (with ring addition) produces: (0 6). Finally, de-convolution (as described above) can be used to recover that element and thus the original data.
- Generalizing, the decode operation (using de-convolution as needed) will work as long only the determinant is non-zero. Preferably, an implementation of the invention does not require use of Gaussian elimination schematically as it is described above; rather, preferably the specific operations and the order of those operations used to invert each case are optimized so as to run most efficiently on the preferred computer hardware used.
- As a variant, c can be selected to be a permutation with a short cycle length, e.g., permute every 4 words in a cycle. As a result, more elements become invertible. In this example, the string [7 13 14] would be invertible, as would the string [1 2 4 8]. This property can be exploited to obviate de-convolution, but potential at a cost of more XOR operations.
- In general, a useful matrix should have the property of being invertible, with as few operations as possible. A suitable matrix inversion method such as Gaussian elimination can be used, with the variation that not all elements have inverses. In such case, as noted above, a diagonalized sub-matrix will not necessarily have the identity element in the column solved for; thus, de-convolution is used. Preferably, the operations used for both coding and decoding are just bitwise XOR and, as needed, de-convolution (which is itself preferably based on XOR). The cycling operation itself just requires some pointer arithmetic and/or adjusting the beginning or end of the result block. The number of XOR operations is essentially given by the number of non-zero bits involved in operators for the operations, with each “1” bit in an operator calling for another copy of a data block.
- For the N+K case (6,3), one of the following matrices may be used:
1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 2 3 4 5 6 1 4 5 16 17 20 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 2 2 2 1 1 3 2 6 3 3 2 2 1 3 5 6 3 1 - Of course, the above are merely illustrative. Because it is desirable to have relatively small coefficients, and because encode and decode costs are relatively easy to estimate, the above-identified matrices were determined by trial and error. More sophisticated algorithms may be used to determine larger cases.
- The present invention may be readily implemented in software, firmware, in special purpose hardware, or in any appropriate combination. Thus, once a suitable matrix is identified, the symbolic inversion steps are identified for each loss case given that matrix. The loss cases can then be readily implemented in code, and encode and decode processes can then be called as routines as needed to the archival and retrieval processes.
- Thus, according to the present invention, an NK coder is described for use to protect data that is being distributed in an RAIN archive. The data itself may be of any type, and it may also include the archive metadata. According to the invention, the data to be distributed is encoded by a matrix operation that uses an identity sub-matrix to preserve the data words, and that uses permutation ring operators to generate the code words. The operators are preferably polynomials that are selected from a group ring of a permutation group with base ring Z2. The ith code block is computed as: Ci=f(gi1(A1), . . . gin(An)), where f( ) is preferably addition mod 2 (i.e., XOR), and g( ) is a permutation operator, such as a polynomial of cyclic permutations. Illustrative operators include, for example, 1=s0 (“do nothing”), Sn (“shift right n words”), 1+sn (XOR, unshifted image with shifted n), and so forth. With these operators, (1+s) (a1a2a3)=(a1+a3)(a2+a1)(a3+a2). The invention is desirable as most operators are very fast. Where matrices are not invertible, the de-convolve operation can be used, i.e., given a first word al, decode (1+s)(A)=(((a1+a3)(a2+a1)(a3+a2). A de-convolution example is shown in
FIG. 3 . - Typically, a data file that is being stored in the cluster comprises a set of N data blocks that are stored in N respective nodes. The coding process in of itself does not require the data file to be broken down in this manner, however.
- One of ordinary skill in the art will appreciate that the encoding technique described above is also useful in protecting against loss and enhancing speed of transmission on communication paths of information represented as data signals on such paths. Thus, more generally, the technique is useful for dispersal and reconstruction of information during communication or storage and retrieval.
- While the above describes a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary, as alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, or the like. References in the specification to a given embodiment indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic.
- While the present invention has been described in the context of a method or process, the present invention also relates to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
Claims (16)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/806,488 US7240236B2 (en) | 2004-03-23 | 2004-03-23 | Fixed content distributed data storage using permutation ring encoding |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/806,488 US7240236B2 (en) | 2004-03-23 | 2004-03-23 | Fixed content distributed data storage using permutation ring encoding |
Publications (2)
Publication Number | Publication Date |
---|---|
US20050216813A1 true US20050216813A1 (en) | 2005-09-29 |
US7240236B2 US7240236B2 (en) | 2007-07-03 |
Family
ID=34991602
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/806,488 Expired - Fee Related US7240236B2 (en) | 2004-03-23 | 2004-03-23 | Fixed content distributed data storage using permutation ring encoding |
Country Status (1)
Country | Link |
---|---|
US (1) | US7240236B2 (en) |
Cited By (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2006117769A2 (en) * | 2005-05-04 | 2006-11-09 | National University Of Ireland, Galway | Method and apparatus for generating error-correcting and error-detecting codes using zero-divisors and units in group rings |
US20070233828A1 (en) * | 2006-03-31 | 2007-10-04 | Jeremy Gilbert | Methods and systems for providing data storage and retrieval |
US20070294310A1 (en) * | 2006-06-06 | 2007-12-20 | Hitachi, Ltd. | Method and apparatus for storing and recovering fixed content |
US20080046764A1 (en) * | 2006-08-04 | 2008-02-21 | Lsi Logic Corporation | Data Shredding RAID Mode |
US20080068899A1 (en) * | 2006-09-15 | 2008-03-20 | Fujitsu Limited | Storage management process, storage management apparatus, and computer-readable medium storing storage management program |
US20100199123A1 (en) * | 2009-02-03 | 2010-08-05 | Bittorrent, Inc. | Distributed Storage of Recoverable Data |
US7814078B1 (en) * | 2005-06-20 | 2010-10-12 | Hewlett-Packard Development Company, L.P. | Identification of files with similar content |
US20100266131A1 (en) * | 2009-04-20 | 2010-10-21 | Bart Cilfone | Natural action heuristics for management of network devices |
CN101064840B (en) * | 2006-04-26 | 2010-11-03 | 日电(中国)有限公司 | Snake-shaped matrix coder/decoder and visual sign coding and presenting system |
EP2288991A2 (en) * | 2008-05-05 | 2011-03-02 | Amplidata | Method of storing a data set in a distributed storage system, distributed storage system and computer program product for use with said method |
WO2012106329A1 (en) * | 2011-02-01 | 2012-08-09 | Cleversafe, Inc. | Utilizing a dispersed storage network access token module to store data in a dispersed storage network memory |
US8291170B1 (en) | 2010-08-19 | 2012-10-16 | Symantec Corporation | System and method for event driven backup data storage |
US8311964B1 (en) | 2009-11-12 | 2012-11-13 | Symantec Corporation | Progressive sampling for deduplication indexing |
US8352692B1 (en) * | 2007-03-30 | 2013-01-08 | Symantec Corporation | Utilizing peer-to-peer services with single instance storage techniques |
US8370315B1 (en) | 2010-05-28 | 2013-02-05 | Symantec Corporation | System and method for high performance deduplication indexing |
US8392384B1 (en) | 2010-12-10 | 2013-03-05 | Symantec Corporation | Method and system of deduplication-based fingerprint index caching |
US8392376B2 (en) | 2010-09-03 | 2013-03-05 | Symantec Corporation | System and method for scalable reference management in a deduplication based storage system |
US8396841B1 (en) | 2010-11-30 | 2013-03-12 | Symantec Corporation | Method and system of multi-level and multi-mode cloud-based deduplication |
CN103023968A (en) * | 2012-11-15 | 2013-04-03 | 中科院成都信息技术有限公司 | Network distributed storage and reading method for file |
US8458515B1 (en) | 2009-11-16 | 2013-06-04 | Symantec Corporation | Raid5 recovery in a high availability object based file system |
US8473463B1 (en) | 2010-03-02 | 2013-06-25 | Symantec Corporation | Method of avoiding duplicate backups in a computing system |
US8495323B1 (en) | 2010-12-07 | 2013-07-23 | Symantec Corporation | Method and system of providing exclusive and secure access to virtual storage objects in a virtual machine cluster |
US8677214B2 (en) * | 2011-10-04 | 2014-03-18 | Cleversafe, Inc. | Encoding data utilizing a zero information gain function |
US8756197B1 (en) | 2010-08-13 | 2014-06-17 | Symantec Corporation | Generating data set views for backup restoration |
US8983952B1 (en) | 2010-07-29 | 2015-03-17 | Symantec Corporation | System and method for partitioning backup data streams in a deduplication based storage system |
US9069786B2 (en) | 2011-10-14 | 2015-06-30 | Pure Storage, Inc. | Method for maintaining multiple fingerprint tables in a deduplicating storage system |
US9454444B1 (en) | 2009-03-19 | 2016-09-27 | Veritas Technologies Llc | Using location tracking of cluster nodes to avoid single points of failure |
US9575680B1 (en) | 2014-08-22 | 2017-02-21 | Veritas Technologies Llc | Deduplication rehydration |
US20170272209A1 (en) * | 2016-03-15 | 2017-09-21 | Cloud Crowding Corp. | Distributed Storage System Data Management And Security |
US10275397B2 (en) | 2013-02-22 | 2019-04-30 | Veritas Technologies Llc | Deduplication storage system with efficient reference updating and space reclamation |
US20190173948A1 (en) * | 2017-03-06 | 2019-06-06 | At&T Intellectual Property I, L.P. | Reliable data storage for decentralized computer systems |
US10423495B1 (en) | 2014-09-08 | 2019-09-24 | Veritas Technologies Llc | Deduplication grouping |
US10931402B2 (en) | 2016-03-15 | 2021-02-23 | Cloud Storage, Inc. | Distributed storage system data management and security |
US11182247B2 (en) | 2019-01-29 | 2021-11-23 | Cloud Storage, Inc. | Encoding and storage node repairing method for minimum storage regenerating codes for distributed storage systems |
US11349500B2 (en) * | 2020-01-15 | 2022-05-31 | EMC IP Holding Company LLC | Data recovery in a geographically diverse storage system employing erasure coding technology and data convolution technology |
Families Citing this family (686)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7068729B2 (en) | 2001-12-21 | 2006-06-27 | Digital Fountain, Inc. | Multi-stage code generator and decoder for communication systems |
US6307487B1 (en) | 1998-09-23 | 2001-10-23 | Digital Fountain, Inc. | Information additive code generator and decoder for communication systems |
US9240810B2 (en) | 2002-06-11 | 2016-01-19 | Digital Fountain, Inc. | Systems and processes for decoding chain reaction codes through inactivation |
EP2348640B1 (en) | 2002-10-05 | 2020-07-15 | QUALCOMM Incorporated | Systematic encoding of chain reaction codes |
US7139960B2 (en) | 2003-10-06 | 2006-11-21 | Digital Fountain, Inc. | Error-correcting multi-stage code generator and decoder for communication systems having single transmitters or multiple transmitters |
KR101041762B1 (en) * | 2003-12-01 | 2011-06-17 | 디지털 파운튼, 인크. | Protection of data from erasures using subsymbol based codes |
WO2005112250A2 (en) | 2004-05-07 | 2005-11-24 | Digital Fountain, Inc. | File download and streaming system |
TWI329866B (en) * | 2005-01-07 | 2010-09-01 | Promise Technology Inc | Data storage system |
US7933405B2 (en) * | 2005-04-08 | 2011-04-26 | Icera Inc. | Data access and permute unit |
US11080138B1 (en) | 2010-04-26 | 2021-08-03 | Pure Storage, Inc. | Storing integrity information in a vast storage system |
US10860424B1 (en) | 2005-09-30 | 2020-12-08 | Pure Storage, Inc. | Background verification processing in a storage network |
US11340988B2 (en) | 2005-09-30 | 2022-05-24 | Pure Storage, Inc. | Generating integrity information in a vast storage system |
US9774684B2 (en) | 2005-09-30 | 2017-09-26 | International Business Machines Corporation | Storing data in a dispersed storage network |
US11416339B1 (en) | 2005-09-30 | 2022-08-16 | Pure Storage, Inc. | Validating requests based on stored vault information |
US10855769B2 (en) | 2005-09-30 | 2020-12-01 | Pure Storage, Inc. | Prioritizing memory devices to replace based on namespace health |
US8171101B2 (en) * | 2005-09-30 | 2012-05-01 | Cleversafe, Inc. | Smart access to a dispersed data storage network |
US10866754B2 (en) | 2010-04-26 | 2020-12-15 | Pure Storage, Inc. | Content archiving in a distributed storage network |
US10051057B2 (en) | 2005-09-30 | 2018-08-14 | International Business Machines Corporation | Prioritizing read locations based on an error history |
US8209363B2 (en) | 2007-10-09 | 2012-06-26 | Cleversafe, Inc. | File system adapted for use with a dispersed data storage network |
US10747616B2 (en) | 2015-03-31 | 2020-08-18 | Pure Storage, Inc. | Adapting rebuilding of encoded data slices in a dispersed storage network |
US11620185B2 (en) | 2005-09-30 | 2023-04-04 | Pure Storage, Inc. | Integrity processing in a dispersed storage network |
US7904475B2 (en) | 2007-10-09 | 2011-03-08 | Cleversafe, Inc. | Virtualized data storage vaults on a dispersed data storage network |
US10432726B2 (en) | 2005-09-30 | 2019-10-01 | Pure Storage, Inc. | Last-resort operations to save at-risk-data |
US7546427B2 (en) * | 2005-09-30 | 2009-06-09 | Cleversafe, Inc. | System for rebuilding dispersed data |
US9063881B2 (en) | 2010-04-26 | 2015-06-23 | Cleversafe, Inc. | Slice retrieval in accordance with an access sequence in a dispersed storage network |
US8285878B2 (en) * | 2007-10-09 | 2012-10-09 | Cleversafe, Inc. | Block based access to a dispersed data storage network |
US10044807B2 (en) | 2005-09-30 | 2018-08-07 | International Business Machines Corporation | Optimistic checked writes |
US10169229B2 (en) | 2012-06-05 | 2019-01-01 | International Business Machines Corporation | Protocols for expanding existing sites in a dispersed storage network |
US11272009B1 (en) | 2005-09-30 | 2022-03-08 | Pure Storage, Inc. | Managed data slice maintenance in a distributed storage system |
US8694668B2 (en) * | 2005-09-30 | 2014-04-08 | Cleversafe, Inc. | Streaming media software interface to a dispersed data storage network |
US10257276B2 (en) | 2005-09-30 | 2019-04-09 | International Business Machines Corporation | Predictive rebalancing according to future usage expectations |
US7574570B2 (en) * | 2005-09-30 | 2009-08-11 | Cleversafe Inc | Billing system for information dispersal system |
US11474903B1 (en) | 2005-09-30 | 2022-10-18 | Pure Storage, Inc. | Rebuilding of encoded data slices using locally decodable code segments |
US10282440B2 (en) | 2015-03-31 | 2019-05-07 | International Business Machines Corporation | Prioritizing rebuilding of encoded data slices |
US8555109B2 (en) * | 2009-07-30 | 2013-10-08 | Cleversafe, Inc. | Method and apparatus for distributed storage integrity processing |
US10250686B2 (en) | 2005-09-30 | 2019-04-02 | International Business Machines Corporation | Finding alternate storage locations to support failing disk migration |
US7574579B2 (en) * | 2005-09-30 | 2009-08-11 | Cleversafe, Inc. | Metadata management system for an information dispersed storage system |
US10270858B2 (en) | 2005-09-30 | 2019-04-23 | International Business Machines Corporation | Inducing memory device idle time through rolling read prioritizations |
US9996413B2 (en) * | 2007-10-09 | 2018-06-12 | International Business Machines Corporation | Ensuring data integrity on a dispersed storage grid |
US11909418B1 (en) | 2005-09-30 | 2024-02-20 | Pure Storage, Inc. | Access authentication in a dispersed storage network |
US8352782B2 (en) * | 2005-09-30 | 2013-01-08 | Cleversafe, Inc. | Range based rebuilder for use with a dispersed data storage network |
US11327674B2 (en) | 2012-06-05 | 2022-05-10 | Pure Storage, Inc. | Storage vault tiering and data migration in a distributed storage network |
US8357048B2 (en) | 2009-09-29 | 2013-01-22 | Cleversafe, Inc. | Interactive gaming utilizing a dispersed storage network |
US11841770B2 (en) | 2005-09-30 | 2023-12-12 | Pure Storage, Inc. | Storage unit connection security in a storage network and methods for use therewith |
US7953937B2 (en) * | 2005-09-30 | 2011-05-31 | Cleversafe, Inc. | Systems, methods, and apparatus for subdividing data for storage in a dispersed data storage grid |
US10154034B2 (en) | 2010-04-26 | 2018-12-11 | International Business Machines Corporation | Cooperative data access request authorization in a dispersed storage network |
US9027080B2 (en) | 2008-03-31 | 2015-05-05 | Cleversafe, Inc. | Proxy access to a dispersed storage network |
US8880799B2 (en) * | 2005-09-30 | 2014-11-04 | Cleversafe, Inc. | Rebuilding data on a dispersed storage network |
US10356177B2 (en) | 2005-09-30 | 2019-07-16 | International Business Machines Corporation | Prioritizing ranges to rebuild based on namespace health |
US9632722B2 (en) | 2010-05-19 | 2017-04-25 | International Business Machines Corporation | Balancing storage unit utilization within a dispersed storage network |
US8595435B2 (en) * | 2009-07-30 | 2013-11-26 | Cleversafe, Inc. | Dispersed storage write process |
US10938418B2 (en) | 2005-09-30 | 2021-03-02 | Pure Storage, Inc. | Online disk replacement/removal |
US11221917B1 (en) | 2005-09-30 | 2022-01-11 | Pure Storage, Inc. | Integrity processing in a dispersed storage network |
US10389814B2 (en) | 2005-09-30 | 2019-08-20 | Pure Storage, Inc. | Prioritizing memory devices to replace based on namespace health |
US8012542B2 (en) * | 2005-12-30 | 2011-09-06 | E.I. Du Pont De Nemours And Company | Fluoropolymer coating compositions containing adhesive polymers and substrate coating process |
US7480848B2 (en) * | 2006-02-10 | 2009-01-20 | The Directv Group, Inc. | Methods and apparatus to select tornado error correction parameters |
US9136983B2 (en) | 2006-02-13 | 2015-09-15 | Digital Fountain, Inc. | Streaming and buffering using variable FEC overhead and protection periods |
US9270414B2 (en) | 2006-02-21 | 2016-02-23 | Digital Fountain, Inc. | Multiple-field based code generator and decoder for communications systems |
JP5167243B2 (en) * | 2006-04-04 | 2013-03-21 | パーマビット テクノロジー コーポレイション | Storage Allocation and Erasure Coding Techniques for Scalable and Fault Tolerant Storage Systems |
WO2007134196A2 (en) | 2006-05-10 | 2007-11-22 | Digital Fountain, Inc. | Code generator and decoder using hybrid codes |
US9386064B2 (en) | 2006-06-09 | 2016-07-05 | Qualcomm Incorporated | Enhanced block-request streaming using URL templates and construction rules |
US9380096B2 (en) | 2006-06-09 | 2016-06-28 | Qualcomm Incorporated | Enhanced block-request streaming system for handling low-latency streaming |
US9178535B2 (en) | 2006-06-09 | 2015-11-03 | Digital Fountain, Inc. | Dynamic stream interleaving and sub-stream based delivery |
US9209934B2 (en) | 2006-06-09 | 2015-12-08 | Qualcomm Incorporated | Enhanced block-request streaming using cooperative parallel HTTP and forward error correction |
US9419749B2 (en) | 2009-08-19 | 2016-08-16 | Qualcomm Incorporated | Methods and apparatus employing FEC codes with permanent inactivation of symbols for encoding and decoding processes |
US9432433B2 (en) | 2006-06-09 | 2016-08-30 | Qualcomm Incorporated | Enhanced block-request streaming system using signaling or block creation |
CN101802797B (en) | 2007-09-12 | 2013-07-17 | 数字方敦股份有限公司 | Generating and communicating source identification information to enable reliable communications |
US8819179B2 (en) | 2007-10-09 | 2014-08-26 | Cleversafe, Inc. | Data revision synchronization in a dispersed storage network |
US9888076B2 (en) | 2007-10-09 | 2018-02-06 | International Business Machines Corporation | Encoded data slice caching in a distributed storage network |
US8549351B2 (en) * | 2007-10-09 | 2013-10-01 | Cleversafe, Inc. | Pessimistic data reading in a dispersed storage network |
US8965956B2 (en) | 2007-10-09 | 2015-02-24 | Cleversafe, Inc. | Integrated client for use with a dispersed data storage network |
US10027478B2 (en) | 2007-10-09 | 2018-07-17 | International Business Machines Corporation | Differential key backup |
US9697171B2 (en) | 2007-10-09 | 2017-07-04 | Internaitonal Business Machines Corporation | Multi-writer revision synchronization in a dispersed storage network |
US8572429B2 (en) * | 2007-10-09 | 2013-10-29 | Cleversafe, Inc. | Optimistic data writing in a dispersed storage network |
US8533256B2 (en) * | 2007-10-09 | 2013-09-10 | Cleversafe, Inc. | Object interface to a dispersed data storage network |
US9959076B2 (en) | 2007-10-09 | 2018-05-01 | International Business Machines Corporation | Optimized disk load distribution |
US8478865B2 (en) * | 2007-10-09 | 2013-07-02 | Cleversafe, Inc. | Systems, methods, and apparatus for matching a connection request with a network interface adapted for use with a dispersed data storage network |
US8185614B2 (en) * | 2007-10-09 | 2012-05-22 | Cleversafe, Inc. | Systems, methods, and apparatus for identifying accessible dispersed digital storage vaults utilizing a centralized registry |
US9501355B2 (en) | 2008-03-31 | 2016-11-22 | International Business Machines Corporation | Storing data and directory information in a distributed storage network |
US20090323580A1 (en) * | 2008-06-27 | 2009-12-31 | Feng Xue | Frame structure and sequencing for enabling network coding for wireless relaying |
US8819011B2 (en) | 2008-07-16 | 2014-08-26 | Cleversafe, Inc. | Command line interpreter for accessing a data object stored in a distributed storage network |
US8630987B2 (en) * | 2008-07-16 | 2014-01-14 | Cleversafe, Inc. | System and method for accessing a data object stored in a distributed storage network |
US9281847B2 (en) | 2009-02-27 | 2016-03-08 | Qualcomm Incorporated | Mobile reception of digital video broadcasting—terrestrial services |
US8601259B2 (en) * | 2009-04-20 | 2013-12-03 | Cleversafe, Inc. | Securing data in a dispersed storage network using security sentinel value |
US9483656B2 (en) | 2009-04-20 | 2016-11-01 | International Business Machines Corporation | Efficient and secure data storage utilizing a dispersed data storage system |
US8744071B2 (en) * | 2009-04-20 | 2014-06-03 | Cleversafe, Inc. | Dispersed data storage system data encryption and encoding |
US11868498B1 (en) | 2009-04-20 | 2024-01-09 | Pure Storage, Inc. | Storage integrity processing in a storage network |
US10104045B2 (en) | 2009-04-20 | 2018-10-16 | International Business Machines Corporation | Verifying data security in a dispersed storage network |
US10447474B2 (en) | 2009-04-20 | 2019-10-15 | Pure Storage, Inc. | Dispersed data storage system data decoding and decryption |
US9092294B2 (en) * | 2009-04-20 | 2015-07-28 | Cleversafe, Inc. | Systems, apparatus, and methods for utilizing a reachability set to manage a network upgrade |
US8504847B2 (en) * | 2009-04-20 | 2013-08-06 | Cleversafe, Inc. | Securing data in a dispersed storage network using shared secret slices |
US8656187B2 (en) * | 2009-04-20 | 2014-02-18 | Cleversafe, Inc. | Dispersed storage secure data decoding |
US10230692B2 (en) * | 2009-06-30 | 2019-03-12 | International Business Machines Corporation | Distributed storage processing module |
US8706980B2 (en) * | 2009-07-30 | 2014-04-22 | Cleversafe, Inc. | Method and apparatus for slice partial rebuilding in a dispersed storage network |
US9009575B2 (en) | 2009-07-30 | 2015-04-14 | Cleversafe, Inc. | Rebuilding a data revision in a dispersed storage network |
US9208025B2 (en) | 2009-07-30 | 2015-12-08 | Cleversafe, Inc. | Virtual memory mapping in a dispersed storage network |
US9558059B2 (en) | 2009-07-30 | 2017-01-31 | International Business Machines Corporation | Detecting data requiring rebuilding in a dispersed storage network |
US8489915B2 (en) * | 2009-07-30 | 2013-07-16 | Cleversafe, Inc. | Method and apparatus for storage integrity processing based on error types in a dispersed storage network |
US9207870B2 (en) | 2009-07-30 | 2015-12-08 | Cleversafe, Inc. | Allocating storage units in a dispersed storage network |
US10108492B2 (en) | 2009-07-30 | 2018-10-23 | International Business Machines Corporation | Rebuilding data stored in a dispersed storage network |
US8275744B2 (en) * | 2009-07-30 | 2012-09-25 | Cleversafe, Inc. | Dispersed storage network virtual address fields |
US8527838B2 (en) | 2009-07-31 | 2013-09-03 | Cleversafe, Inc. | Memory controller utilizing an error coding dispersal function |
US9167277B2 (en) * | 2009-08-03 | 2015-10-20 | Cleversafe, Inc. | Dispersed storage network data manipulation |
US9288010B2 (en) | 2009-08-19 | 2016-03-15 | Qualcomm Incorporated | Universal file delivery methods for providing unequal error protection and bundled file delivery services |
US8949695B2 (en) * | 2009-08-27 | 2015-02-03 | Cleversafe, Inc. | Method and apparatus for nested dispersed storage |
US9690513B2 (en) * | 2009-08-27 | 2017-06-27 | International Business Machines Corporation | Dispersed storage processing unit and methods with operating system diversity for use in a dispersed storage system |
US8560855B2 (en) * | 2009-08-27 | 2013-10-15 | Cleversafe, Inc. | Verification of dispersed storage network access control information |
US9411810B2 (en) | 2009-08-27 | 2016-08-09 | International Business Machines Corporation | Method and apparatus for identifying data inconsistency in a dispersed storage network |
US9917874B2 (en) | 2009-09-22 | 2018-03-13 | Qualcomm Incorporated | Enhanced block-request streaming using block partitioning or request controls for improved client-side handling |
US8918534B2 (en) | 2009-09-29 | 2014-12-23 | Cleversafe, Inc. | Writing data slices to ready and non-ready distributed storage units in a distributed storage network |
US8548913B2 (en) * | 2009-09-29 | 2013-10-01 | Cleversafe, Inc. | Method and apparatus to secure an electronic commerce transaction |
US9448730B2 (en) * | 2009-09-30 | 2016-09-20 | International Business Machines Corporation | Method and apparatus for dispersed storage data transfer |
US8438456B2 (en) * | 2009-10-05 | 2013-05-07 | Cleversafe, Inc. | Method and apparatus for dispersed storage of streaming data |
US9015431B2 (en) * | 2009-10-29 | 2015-04-21 | Cleversafe, Inc. | Distributed storage revision rollbacks |
US10757187B2 (en) | 2009-10-29 | 2020-08-25 | Pure Storage, Inc. | Streaming all-or-nothing encoding with random offset support |
US9774678B2 (en) | 2009-10-29 | 2017-09-26 | International Business Machines Corporation | Temporarily storing data in a dispersed storage network |
US8522074B2 (en) * | 2009-10-29 | 2013-08-27 | Cleversafe, Inc. | Intentionally introduced storage deviations in a dispersed storage network |
US9661356B2 (en) | 2009-10-29 | 2017-05-23 | International Business Machines Corporation | Distribution of unique copies of broadcast data utilizing fault-tolerant retrieval from dispersed storage |
US10389845B2 (en) | 2009-10-29 | 2019-08-20 | Pure Storage, Inc. | Determining how to service requests based on several indicators |
US20110107182A1 (en) * | 2009-10-30 | 2011-05-05 | Cleversafe, Inc. | Dispersed storage unit solicitation method and apparatus |
US8589637B2 (en) * | 2009-10-30 | 2013-11-19 | Cleversafe, Inc. | Concurrent set storage in distributed storage network |
US9098376B2 (en) | 2009-10-30 | 2015-08-04 | Cleversafe, Inc. | Distributed storage network for modification of a data object |
US9195408B2 (en) | 2009-10-30 | 2015-11-24 | Cleversafe, Inc. | Highly autonomous dispersed storage system retrieval method |
US9413529B2 (en) | 2009-10-30 | 2016-08-09 | International Business Machines Corporation | Distributed storage network and method for storing and retrieving encryption keys |
US10073737B2 (en) | 2009-10-30 | 2018-09-11 | International Business Machines Corporation | Slice location identification |
US9311185B2 (en) | 2009-10-30 | 2016-04-12 | Cleversafe, Inc. | Dispersed storage unit solicitation method and apparatus |
US8479078B2 (en) * | 2009-10-30 | 2013-07-02 | Cleversafe, Inc. | Distributed storage network for modification of a data object |
US9900150B2 (en) * | 2009-10-30 | 2018-02-20 | International Business Machines Corporation | Dispersed storage camera device and method of operation |
US9270298B2 (en) | 2009-11-24 | 2016-02-23 | International Business Machines Corporation | Selecting storage units to rebuild an encoded data slice |
US9501349B2 (en) * | 2009-11-24 | 2016-11-22 | International Business Machines Corporation | Changing dispersed storage error encoding parameters |
US9152514B2 (en) | 2009-11-24 | 2015-10-06 | Cleversafe, Inc. | Rebuilding a data segment in a dispersed storage network |
US8918897B2 (en) | 2009-11-24 | 2014-12-23 | Cleversafe, Inc. | Dispersed storage network data slice integrity verification |
US8458233B2 (en) | 2009-11-25 | 2013-06-04 | Cleversafe, Inc. | Data de-duplication in a dispersed storage network utilizing data characterization |
US9672109B2 (en) | 2009-11-25 | 2017-06-06 | International Business Machines Corporation | Adaptive dispersed storage network (DSN) and system |
US9996548B2 (en) | 2009-11-25 | 2018-06-12 | International Business Machines Corporation | Dispersed storage using localized peer-to-peer capable wireless devices in a peer-to-peer or femto cell supported carrier served fashion |
US9626248B2 (en) | 2009-11-25 | 2017-04-18 | International Business Machines Corporation | Likelihood based rebuilding of missing encoded data slices |
US8688907B2 (en) * | 2009-11-25 | 2014-04-01 | Cleversafe, Inc. | Large scale subscription based dispersed storage network |
US8527807B2 (en) * | 2009-11-25 | 2013-09-03 | Cleversafe, Inc. | Localized dispersed storage memory system |
US9836352B2 (en) | 2009-11-25 | 2017-12-05 | International Business Machines Corporation | Detecting a utilization imbalance between dispersed storage network storage units |
US9489264B2 (en) | 2009-11-25 | 2016-11-08 | International Business Machines Corporation | Storing an encoded data slice as a set of sub-slices |
US8621268B2 (en) * | 2009-11-25 | 2013-12-31 | Cleversafe, Inc. | Write threshold utilization in a dispersed storage system |
US10015141B2 (en) | 2009-11-25 | 2018-07-03 | International Business Machines Corporation | Dispersed data storage in a VPN group of devices |
US8990585B2 (en) | 2009-12-29 | 2015-03-24 | Cleversafe, Inc. | Time based dispersed storage access |
US9330241B2 (en) | 2009-12-29 | 2016-05-03 | International Business Machines Corporation | Applying digital rights management to multi-media file playback |
US9672108B2 (en) | 2009-12-29 | 2017-06-06 | International Business Machines Corporation | Dispersed storage network (DSN) and system with improved security |
US10237281B2 (en) | 2009-12-29 | 2019-03-19 | International Business Machines Corporation | Access policy updates in a dispersed storage network |
US10133632B2 (en) | 2009-12-29 | 2018-11-20 | International Business Machines Corporation | Determining completion of migration in a dispersed storage network |
US10289505B2 (en) | 2009-12-29 | 2019-05-14 | International Business Machines Corporation | Dispersed multi-media content for a centralized digital video storage system |
US9866595B2 (en) | 2009-12-29 | 2018-01-09 | International Busines Machines Corporation | Policy based slice deletion in a dispersed storage network |
US9305597B2 (en) | 2009-12-29 | 2016-04-05 | Cleversafe, Inc. | Accessing stored multi-media content based on a subscription priority level |
US10158648B2 (en) | 2009-12-29 | 2018-12-18 | International Business Machines Corporation | Policy-based access in a dispersed storage network |
US10031669B2 (en) | 2009-12-29 | 2018-07-24 | International Business Machines Corporation | Scheduling migration related traffic to be non-disruptive and performant |
US9727266B2 (en) | 2009-12-29 | 2017-08-08 | International Business Machines Corporation | Selecting storage units in a dispersed storage network |
US9507735B2 (en) | 2009-12-29 | 2016-11-29 | International Business Machines Corporation | Digital content retrieval utilizing dispersed storage |
US9369526B2 (en) | 2009-12-29 | 2016-06-14 | International Business Machines Corporation | Distributed storage time synchronization based on retrieval delay |
US8468368B2 (en) * | 2009-12-29 | 2013-06-18 | Cleversafe, Inc. | Data encryption parameter dispersal |
US10148788B2 (en) | 2009-12-29 | 2018-12-04 | International Business Machines Corporation | Method for providing schedulers in a distributed storage network |
US8762343B2 (en) * | 2009-12-29 | 2014-06-24 | Cleversafe, Inc. | Dispersed storage of software |
US10067831B2 (en) | 2009-12-29 | 2018-09-04 | International Business Machines Corporation | Slice migration in a dispersed storage network |
US9152489B2 (en) | 2009-12-29 | 2015-10-06 | Cleversafe, Inc. | Revision synchronization of a dispersed storage network |
US20180335967A1 (en) | 2009-12-29 | 2018-11-22 | International Business Machines Corporation | User customizable data processing plan in a dispersed storage network |
US9413393B2 (en) | 2009-12-29 | 2016-08-09 | International Business Machines Corporation | Encoding multi-media content for a centralized digital video storage system |
US9798467B2 (en) | 2009-12-29 | 2017-10-24 | International Business Machines Corporation | Security checks for proxied requests |
US10372686B2 (en) | 2009-12-29 | 2019-08-06 | International Business Machines Corporation | Policy-based storage in a dispersed storage network |
US9922063B2 (en) | 2009-12-29 | 2018-03-20 | International Business Machines Corporation | Secure storage of secret data in a dispersed storage network |
US9462316B2 (en) * | 2009-12-29 | 2016-10-04 | International Business Machines Corporation | Digital content retrieval utilizing dispersed storage |
US10001923B2 (en) | 2009-12-29 | 2018-06-19 | International Business Machines Corporation | Generation collapse |
US8918674B2 (en) | 2010-01-28 | 2014-12-23 | Cleversafe, Inc. | Directory file system in a dispersed storage network |
US9760440B2 (en) | 2010-01-28 | 2017-09-12 | International Business Machines Corporation | Site-based namespace allocation |
US11301592B2 (en) | 2010-01-28 | 2022-04-12 | Pure Storage, Inc. | Distributed storage with data obfuscation and method for use therewith |
US9201732B2 (en) | 2010-01-28 | 2015-12-01 | Cleversafe, Inc. | Selective activation of memory to retrieve data in a dispersed storage network |
US9043548B2 (en) | 2010-01-28 | 2015-05-26 | Cleversafe, Inc. | Streaming content storage |
US8954667B2 (en) * | 2010-01-28 | 2015-02-10 | Cleversafe, Inc. | Data migration in a dispersed storage network |
US8959366B2 (en) * | 2010-01-28 | 2015-02-17 | Cleversafe, Inc. | De-sequencing encoded data slices |
US20110184997A1 (en) * | 2010-01-28 | 2011-07-28 | Cleversafe, Inc. | Selecting storage facilities in a plurality of dispersed storage networks |
US8850410B2 (en) * | 2010-01-29 | 2014-09-30 | International Business Machines Corporation | System using a unique marker with each software code-block |
US10268374B2 (en) | 2010-02-27 | 2019-04-23 | International Business Machines Corporation | Redundant array of independent discs and dispersed storage network system re-director |
US10216647B2 (en) | 2010-02-27 | 2019-02-26 | International Business Machines Corporation | Compacting dispersed storage space |
US10007575B2 (en) | 2010-02-27 | 2018-06-26 | International Business Machines Corporation | Alternative multiple memory format storage in a storage network |
US10324791B2 (en) | 2010-11-01 | 2019-06-18 | International Business Machines Corporation | Selectable parallel processing of dispersed storage error encoding |
US9135115B2 (en) | 2010-02-27 | 2015-09-15 | Cleversafe, Inc. | Storing data in multiple formats including a dispersed storage format |
US9606867B2 (en) | 2014-06-05 | 2017-03-28 | International Business Machines Corporation | Maintaining data storage in accordance with an access metric |
US11429486B1 (en) | 2010-02-27 | 2022-08-30 | Pure Storage, Inc. | Rebuilding data via locally decodable redundancy in a vast storage network |
US20180365105A1 (en) | 2014-06-05 | 2018-12-20 | International Business Machines Corporation | Establishing an operation execution schedule in a dispersed storage network |
US8850113B2 (en) * | 2010-02-27 | 2014-09-30 | Cleversafe, Inc. | Data migration between a raid memory and a dispersed storage network memory |
US8281182B2 (en) | 2010-03-12 | 2012-10-02 | Cleversafe, Inc. | Dispersed storage unit selection |
US8707091B2 (en) * | 2010-03-15 | 2014-04-22 | Cleversafe, Inc. | Failsafe directory file system in a dispersed storage network |
US9229824B2 (en) | 2010-03-16 | 2016-01-05 | International Business Machines Corporation | Caching rebuilt encoded data slices in a dispersed storage network |
US8495466B2 (en) * | 2010-03-16 | 2013-07-23 | Cleversafe, Inc. | Adjusting data dispersal in a dispersed storage network |
US9170884B2 (en) | 2010-03-16 | 2015-10-27 | Cleversafe, Inc. | Utilizing cached encoded data slices in a dispersed storage network |
US9495117B2 (en) | 2010-04-26 | 2016-11-15 | International Business Machines Corporation | Storing data in a dispersed storage network |
US8938552B2 (en) | 2010-08-02 | 2015-01-20 | Cleversafe, Inc. | Resolving a protocol issue within a dispersed storage network |
US10447767B2 (en) | 2010-04-26 | 2019-10-15 | Pure Storage, Inc. | Resolving a performance issue within a dispersed storage network |
US10956292B1 (en) | 2010-04-26 | 2021-03-23 | Pure Storage, Inc. | Utilizing integrity information for data retrieval in a vast storage system |
US8914669B2 (en) | 2010-04-26 | 2014-12-16 | Cleversafe, Inc. | Secure rebuilding of an encoded data slice in a dispersed storage network |
US9606858B2 (en) | 2010-04-26 | 2017-03-28 | International Business Machines Corporation | Temporarily storing an encoded data slice |
US9898373B2 (en) | 2010-04-26 | 2018-02-20 | International Business Machines Corporation | Prioritizing rebuilding of stored data in a dispersed storage network |
US9092386B2 (en) | 2010-04-26 | 2015-07-28 | Cleversafe, Inc. | Indicating an error within a dispersed storage network |
US8625635B2 (en) | 2010-04-26 | 2014-01-07 | Cleversafe, Inc. | Dispersed storage network frame protocol header |
US9225961B2 (en) | 2010-05-13 | 2015-12-29 | Qualcomm Incorporated | Frame packing for asymmetric stereo video |
US8521697B2 (en) | 2010-05-19 | 2013-08-27 | Cleversafe, Inc. | Rebuilding data in multiple dispersed storage networks |
US10353774B2 (en) | 2015-10-30 | 2019-07-16 | International Business Machines Corporation | Utilizing storage unit latency data in a dispersed storage network |
US8874868B2 (en) | 2010-05-19 | 2014-10-28 | Cleversafe, Inc. | Memory utilization balancing in a dispersed storage network |
US8621580B2 (en) | 2010-05-19 | 2013-12-31 | Cleversafe, Inc. | Retrieving access information in a dispersed storage network |
US10193689B2 (en) | 2010-05-19 | 2019-01-29 | International Business Machines Corporation | Storing access information in a dispersed storage network |
US10911230B2 (en) | 2010-05-19 | 2021-02-02 | Pure Storage, Inc. | Securely activating functionality of a computing device in a dispersed storage network |
US11740972B1 (en) | 2010-05-19 | 2023-08-29 | Pure Storage, Inc. | Migrating data in a vast storage network |
US8909858B2 (en) | 2010-06-09 | 2014-12-09 | Cleversafe, Inc. | Storing encoded data slices in a dispersed storage network |
US9231768B2 (en) | 2010-06-22 | 2016-01-05 | International Business Machines Corporation | Utilizing a deterministic all or nothing transformation in a dispersed storage network |
US8612831B2 (en) | 2010-06-22 | 2013-12-17 | Cleversafe, Inc. | Accessing data stored in a dispersed storage memory |
US9596447B2 (en) | 2010-07-21 | 2017-03-14 | Qualcomm Incorporated | Providing frame packing type information for video coding |
US8842746B2 (en) | 2010-08-02 | 2014-09-23 | Cleversafe, Inc. | Receiving encoded data slices via wireless communication |
US20190095101A1 (en) | 2010-08-02 | 2019-03-28 | International Business Machines Corporation | Authenticating a credential in a dispersed storage network |
US9063968B2 (en) | 2010-08-02 | 2015-06-23 | Cleversafe, Inc. | Identifying a compromised encoded data slice |
US9077734B2 (en) | 2010-08-02 | 2015-07-07 | Cleversafe, Inc. | Authentication of devices of a dispersed storage network |
US10162524B2 (en) | 2010-08-02 | 2018-12-25 | International Business Machines Corporation | Determining whether to compress a data segment in a dispersed storage network |
US9456015B2 (en) | 2010-08-10 | 2016-09-27 | Qualcomm Incorporated | Representation groups for network streaming of coded multimedia data |
US9940195B2 (en) | 2010-08-25 | 2018-04-10 | International Business Machines Corporation | Encryption of slice partials |
US10255135B2 (en) | 2010-08-25 | 2019-04-09 | International Business Machines Corporation | Method and apparatus for non-interactive information dispersal |
US10157002B2 (en) | 2010-08-26 | 2018-12-18 | International Business Machines Corporation | Migrating an encoded data slice based on an end-of-life memory level of a memory device |
US10970168B2 (en) | 2010-10-06 | 2021-04-06 | Pure Storage, Inc. | Adjusting dispersed storage error encoding parameters based on path performance |
US10298957B2 (en) | 2010-10-06 | 2019-05-21 | International Business Machines Corporation | Content-based encoding in a multiple routing path communications system |
US9037937B2 (en) | 2010-10-06 | 2015-05-19 | Cleversafe, Inc. | Relaying data transmitted as encoded data slices |
US8656138B2 (en) | 2010-10-06 | 2014-02-18 | Cleversafe, Inc. | Efficiently accessing an encoded data slice utilizing a memory bin |
US9571230B2 (en) | 2010-10-06 | 2017-02-14 | International Business Machines Corporation | Adjusting routing of data within a network path |
US9843412B2 (en) | 2010-10-06 | 2017-12-12 | International Business Machines Corporation | Optimizing routing of data across a communications network |
US8707105B2 (en) | 2010-11-01 | 2014-04-22 | Cleversafe, Inc. | Updating a set of memory devices in a dispersed storage network |
US10768833B2 (en) | 2010-11-01 | 2020-09-08 | Pure Storage, Inc. | Object dispersal load balancing |
US10289318B2 (en) | 2010-11-01 | 2019-05-14 | International Business Machines Corporation | Adjusting optimistic writes in a dispersed storage network |
US9015499B2 (en) | 2010-11-01 | 2015-04-21 | Cleversafe, Inc. | Verifying data integrity utilizing dispersed storage |
US10082970B2 (en) | 2010-11-01 | 2018-09-25 | International Business Machines Corporation | Storing an effective dynamic width of encoded data slices |
US10805042B2 (en) | 2010-11-01 | 2020-10-13 | Pure Storage, Inc. | Creating transmission data slices for use in a dispersed storage network |
US10146645B2 (en) | 2010-11-01 | 2018-12-04 | International Business Machines Corporation | Multiple memory format storage in a storage network |
US8627065B2 (en) | 2010-11-09 | 2014-01-07 | Cleversafe, Inc. | Validating a certificate chain in a dispersed storage network |
US9590838B2 (en) | 2010-11-09 | 2017-03-07 | International Business Machines Corporation | Transferring data of a dispersed storage network |
US11061597B2 (en) | 2010-11-09 | 2021-07-13 | Pure Storage, Inc. | Supporting live migrations and re-balancing with a virtual storage unit |
US11307930B1 (en) | 2010-11-29 | 2022-04-19 | Pure Storage, Inc. | Optimized selection of participants in distributed data rebuild/verification |
US10922179B2 (en) | 2010-11-29 | 2021-02-16 | Pure Storage, Inc. | Post rebuild verification |
US9483398B2 (en) | 2010-11-29 | 2016-11-01 | International Business Machines Corporation | Partitioning data for storage in a dispersed storage network |
US10802763B2 (en) | 2010-11-29 | 2020-10-13 | Pure Storage, Inc. | Remote storage verification |
US9454431B2 (en) | 2010-11-29 | 2016-09-27 | International Business Machines Corporation | Memory selection for slice storage in a dispersed storage network |
US10372350B2 (en) | 2010-11-29 | 2019-08-06 | Pure Storage, Inc. | Shared ownership of namespace ranges |
US11789631B2 (en) | 2010-11-29 | 2023-10-17 | Pure Storage, Inc. | Utilizing metadata storage trees in a vast storage network |
US9170882B2 (en) | 2010-12-22 | 2015-10-27 | Cleversafe, Inc. | Retrieving data segments from a dispersed storage network |
US8897443B2 (en) | 2010-12-27 | 2014-11-25 | Cleversafe, Inc. | Watermarking slices stored in a dispersed storage network |
US8694752B2 (en) | 2011-02-01 | 2014-04-08 | Cleversafe, Inc. | Transferring data in response to detection of a memory system imbalance |
US8958375B2 (en) | 2011-02-11 | 2015-02-17 | Qualcomm Incorporated | Framing for an improved radio link protocol including FEC |
US9270299B2 (en) | 2011-02-11 | 2016-02-23 | Qualcomm Incorporated | Encoding and decoding using elastic codes with flexible source block mapping |
US20120226855A1 (en) | 2011-03-02 | 2012-09-06 | Cleversafe, Inc. | Sharing a directory of a dispersed storage network |
US8868695B2 (en) | 2011-03-02 | 2014-10-21 | Cleversafe, Inc. | Configuring a generic computing device utilizing specific computing device operation information |
US10298684B2 (en) | 2011-04-01 | 2019-05-21 | International Business Machines Corporation | Adaptive replication of dispersed data to improve data access performance |
US8880978B2 (en) | 2011-04-01 | 2014-11-04 | Cleversafe, Inc. | Utilizing a local area network memory and a dispersed storage network memory to access data |
US8627091B2 (en) | 2011-04-01 | 2014-01-07 | Cleversafe, Inc. | Generating a secure signature utilizing a plurality of key shares |
US11418580B2 (en) | 2011-04-01 | 2022-08-16 | Pure Storage, Inc. | Selective generation of secure signatures in a distributed storage network |
US9141458B2 (en) | 2011-05-09 | 2015-09-22 | Cleversafe, Inc. | Adjusting a data storage address mapping in a maintenance free storage container |
US8707393B2 (en) | 2011-05-09 | 2014-04-22 | Cleversafe, Inc. | Providing dispersed storage network location information of a hypertext markup language file |
US9219604B2 (en) | 2011-05-09 | 2015-12-22 | Cleversafe, Inc. | Generating an encrypted message for storage |
US9298550B2 (en) | 2011-05-09 | 2016-03-29 | Cleversafe, Inc. | Assigning a dispersed storage network address range in a maintenance free storage container |
US20170192684A1 (en) | 2011-05-09 | 2017-07-06 | International Business Machines Corporation | Auditing a transaction in a dispersed storage network |
US10042709B2 (en) | 2011-06-06 | 2018-08-07 | International Business Machines Corporation | Rebuild prioritization during a plurality of concurrent data object write operations |
US8656253B2 (en) | 2011-06-06 | 2014-02-18 | Cleversafe, Inc. | Storing portions of data in a dispersed storage network |
US10949301B2 (en) | 2011-06-06 | 2021-03-16 | Pure Storage, Inc. | Pre-positioning pre-stored content in a content distribution system |
US8756480B2 (en) | 2011-06-06 | 2014-06-17 | Cleversafe, Inc. | Prioritized deleting of slices stored in a dispersed storage network |
US10061650B2 (en) | 2011-06-06 | 2018-08-28 | International Business Machines Corporation | Priority based rebuilding |
US9244770B2 (en) | 2011-07-06 | 2016-01-26 | International Business Machines Corporation | Responding to a maintenance free storage container security threat |
US8762770B2 (en) | 2011-07-06 | 2014-06-24 | Cleversafe, Inc. | Distribution of a customized preview of multi-media content |
US8914667B2 (en) | 2011-07-27 | 2014-12-16 | Cleversafe, Inc. | Identifying a slice error in a dispersed storage network |
US11016702B2 (en) | 2011-07-27 | 2021-05-25 | Pure Storage, Inc. | Hierarchical event tree |
US10678619B2 (en) | 2011-07-27 | 2020-06-09 | Pure Storage, Inc. | Unified logs and device statistics |
US9971802B2 (en) | 2011-08-17 | 2018-05-15 | International Business Machines Corporation | Audit record transformation in a dispersed storage network |
US10454678B2 (en) | 2011-08-17 | 2019-10-22 | Pure Storage, Inc. | Accesor-based audit trails |
US10120756B2 (en) | 2011-08-17 | 2018-11-06 | International Business Machines Corporation | Audit object generation in a dispersed storage network |
US9092385B2 (en) | 2011-08-17 | 2015-07-28 | Cleversafe, Inc. | Facilitating access of a dispersed storage network |
US9253233B2 (en) | 2011-08-31 | 2016-02-02 | Qualcomm Incorporated | Switch signaling methods providing improved switching between representations for adaptive HTTP streaming |
US8751894B2 (en) | 2011-09-06 | 2014-06-10 | Cleversafe, Inc. | Concurrent decoding of data streams |
US20190179696A1 (en) | 2011-09-06 | 2019-06-13 | International Business Machines Corporation | Demultiplexing decoded data streams in a distributed storage network |
US11907060B2 (en) | 2011-09-06 | 2024-02-20 | Pure Storage, Inc. | Coding of data streams in a vast storage network |
US10235237B2 (en) | 2011-09-06 | 2019-03-19 | Intertnational Business Machines Corporation | Decoding data streams in a distributed storage network |
US8930320B2 (en) * | 2011-09-30 | 2015-01-06 | Accenture Global Services Limited | Distributed computing backup and recovery system |
US9785491B2 (en) | 2011-10-04 | 2017-10-10 | International Business Machines Corporation | Processing a certificate signing request in a dispersed storage network |
US8555130B2 (en) | 2011-10-04 | 2013-10-08 | Cleversafe, Inc. | Storing encoded data slices in a dispersed storage unit |
US9843844B2 (en) | 2011-10-05 | 2017-12-12 | Qualcomm Incorporated | Network streaming of media data |
US9798616B2 (en) | 2011-11-01 | 2017-10-24 | International Business Machines Corporation | Wireless sending a set of encoded data slices |
US10365969B2 (en) | 2011-11-01 | 2019-07-30 | International Business Machines Corporation | Multiple wireless communication systems stream slices based on geography |
US10437678B2 (en) | 2011-11-01 | 2019-10-08 | Pure Storage, Inc. | Updating an encoded data slice |
US11329830B1 (en) | 2011-11-01 | 2022-05-10 | Pure Storage, Inc. | Dispersed credentials |
US10496500B2 (en) | 2011-11-01 | 2019-12-03 | Pure Storage, Inc. | Preemptively reading extra encoded data slices |
US8607122B2 (en) | 2011-11-01 | 2013-12-10 | Cleversafe, Inc. | Accessing a large data object in a dispersed storage network |
US8627066B2 (en) | 2011-11-03 | 2014-01-07 | Cleversafe, Inc. | Processing a dispersed storage network access request utilizing certificate chain validation information |
US11474958B1 (en) | 2011-11-28 | 2022-10-18 | Pure Storage, Inc. | Generating and queuing system messages with priorities in a storage network |
US10558592B2 (en) | 2011-11-28 | 2020-02-11 | Pure Storage, Inc. | Priority level adaptation in a dispersed storage network |
US10318445B2 (en) | 2011-11-28 | 2019-06-11 | International Business Machines Corporation | Priority level adaptation in a dispersed storage network |
US10469578B2 (en) | 2011-11-28 | 2019-11-05 | Pure Storage, Inc. | Prioritization of messages of a dispersed storage network |
US8848906B2 (en) | 2011-11-28 | 2014-09-30 | Cleversafe, Inc. | Encrypting data for storage in a dispersed storage network |
US10387071B2 (en) | 2011-11-28 | 2019-08-20 | Pure Storage, Inc. | On-the-fly cancellation of unnecessary read requests |
US10977194B2 (en) | 2011-11-28 | 2021-04-13 | Pure Storage, Inc. | Securely storing random keys in a dispersed storage network |
US10055283B2 (en) | 2011-11-28 | 2018-08-21 | International Business Machines Corporation | Securely distributing random keys in a dispersed storage network |
US10104168B2 (en) | 2011-12-12 | 2018-10-16 | International Business Machines Corporation | Method for managing throughput in a distributed storage network |
US9304857B2 (en) | 2011-12-12 | 2016-04-05 | Cleversafe, Inc. | Retrieving data from a distributed storage network |
US10146621B2 (en) | 2011-12-12 | 2018-12-04 | International Business Machines Corporation | Chaining computes in a distributed computing system |
US10176045B2 (en) | 2011-12-12 | 2019-01-08 | International Business Machines Corporation | Internet based shared memory in a distributed computing system |
US10666596B2 (en) | 2011-12-12 | 2020-05-26 | Pure Storage, Inc. | Messaging via a shared memory of a distributed computing system |
US10348640B2 (en) | 2011-12-12 | 2019-07-09 | International Business Machines Corporation | Partial task execution in a dispersed storage network |
US9584359B2 (en) | 2011-12-12 | 2017-02-28 | International Business Machines Corporation | Distributed storage and computing of interim data |
US9009567B2 (en) | 2011-12-12 | 2015-04-14 | Cleversafe, Inc. | Encrypting distributed computing data |
US9674155B2 (en) | 2011-12-12 | 2017-06-06 | International Business Machines Corporation | Encrypting segmented data in a distributed computing system |
US9141468B2 (en) | 2011-12-12 | 2015-09-22 | Cleversafe, Inc. | Managing memory utilization in a distributed storage and task network |
US20180083930A1 (en) | 2011-12-12 | 2018-03-22 | International Business Machines Corporation | Reads for dispersed computation jobs |
US10346218B2 (en) | 2011-12-12 | 2019-07-09 | International Business Machines Corporation | Partial task allocation in a dispersed storage network |
US9430286B2 (en) | 2011-12-12 | 2016-08-30 | International Business Machines Corporation | Authorizing distributed task processing in a distributed storage network |
US9817701B2 (en) | 2011-12-12 | 2017-11-14 | International Business Machines Corporation | Threshold computing in a distributed computing system |
US20130238900A1 (en) | 2011-12-12 | 2013-09-12 | Cleversafe, Inc. | Dispersed storage network secure hierarchical file directory |
US10360106B2 (en) | 2011-12-12 | 2019-07-23 | International Business Machines Corporation | Throttled real-time writes |
US9891995B2 (en) | 2012-01-31 | 2018-02-13 | International Business Machines Corporation | Cooperative decentralized rebuild scanning |
US9146810B2 (en) | 2012-01-31 | 2015-09-29 | Cleversafe, Inc. | Identifying a potentially compromised encoded data slice |
US9465861B2 (en) | 2012-01-31 | 2016-10-11 | International Business Machines Corporation | Retrieving indexed data from a dispersed storage network |
US10140177B2 (en) | 2012-01-31 | 2018-11-27 | International Business Machines Corporation | Transferring a partial task in a distributed computing system |
US9507786B2 (en) | 2012-01-31 | 2016-11-29 | International Business Machines Corporation | Retrieving data utilizing a distributed index |
US10671585B2 (en) | 2012-01-31 | 2020-06-02 | Pure Storage, Inc. | Storing indexed data to a dispersed storage network |
US10157051B2 (en) | 2012-03-02 | 2018-12-18 | International Business Machines Corporation | Upgrading devices in a dispersed storage network |
US10402393B2 (en) | 2012-03-02 | 2019-09-03 | Pure Storage, Inc. | Slice migration in a dispersed storage network |
US11232093B2 (en) | 2012-03-02 | 2022-01-25 | Pure Storage, Inc. | Slice migration in a dispersed storage network |
US10013444B2 (en) | 2012-03-02 | 2018-07-03 | International Business Machines Corporation | Modifying an index node of a hierarchical dispersed storage index |
US9588994B2 (en) | 2012-03-02 | 2017-03-07 | International Business Machines Corporation | Transferring task execution in a distributed storage and task network |
US9294226B2 (en) | 2012-03-26 | 2016-03-22 | Qualcomm Incorporated | Universal object delivery and template-based file delivery |
US10795766B2 (en) | 2012-04-25 | 2020-10-06 | Pure Storage, Inc. | Mapping slice groupings in a dispersed storage network |
US10621044B2 (en) | 2012-04-25 | 2020-04-14 | Pure Storage, Inc. | Mapping slice groupings in a dispersed storage network |
US9380032B2 (en) | 2012-04-25 | 2016-06-28 | International Business Machines Corporation | Encrypting data for storage in a dispersed storage network |
US9613052B2 (en) | 2012-06-05 | 2017-04-04 | International Business Machines Corporation | Establishing trust within a cloud computing system |
US10002047B2 (en) | 2012-06-05 | 2018-06-19 | International Business Machines Corporation | Read-if-not-revision-equals protocol message |
US10073638B2 (en) | 2012-06-05 | 2018-09-11 | International Business Machines Corporation | Automatic namespace ordering determination |
US9164841B2 (en) | 2012-06-05 | 2015-10-20 | Cleversafe, Inc. | Resolution of a storage error in a dispersed storage network |
US10474395B2 (en) | 2012-06-05 | 2019-11-12 | Pure Storage, Inc. | Abstracting namespace mapping in a dispersed storage network through multiple hierarchies |
US10447471B2 (en) | 2012-06-05 | 2019-10-15 | Pure Storage, Inc. | Systematic secret sharing |
US10120574B2 (en) | 2012-06-25 | 2018-11-06 | International Business Machines Corporation | Reversible data modifications within DS units |
US11093327B1 (en) | 2012-06-25 | 2021-08-17 | Pure Storage, Inc. | Failure abatement approach for failed storage units common to multiple vaults |
US10114697B2 (en) | 2012-06-25 | 2018-10-30 | International Business Machines Corporation | Large object parallel writing |
US10157011B2 (en) | 2012-06-25 | 2018-12-18 | International Business Machines Corporation | Temporary suspension of vault access |
US8935761B2 (en) | 2012-06-25 | 2015-01-13 | Cleversafe, Inc. | Accessing storage nodes in an on-line media storage system |
US10430276B2 (en) | 2012-06-25 | 2019-10-01 | Pure Storage, Inc. | Optimal orderings of processing unit priorities in a dispersed storage network |
US20180336097A1 (en) | 2012-06-25 | 2018-11-22 | International Business Machines Corporation | Namespace affinity and failover for processing units in a dispersed storage network |
US9141297B2 (en) | 2012-06-25 | 2015-09-22 | Cleversafe, Inc. | Verifying encoded data slice integrity in a dispersed storage network |
US10651975B2 (en) | 2012-08-02 | 2020-05-12 | Pure Storage, Inc. | Forwarding data amongst cooperative DSTN processing units of a massive data ingestion system |
US9258177B2 (en) | 2012-08-02 | 2016-02-09 | International Business Machines Corporation | Storing a data stream in a set of storage devices |
US11360851B2 (en) | 2012-08-31 | 2022-06-14 | Pure Storage, Inc. | Duplicating authentication information between connections |
US10241863B2 (en) | 2012-08-31 | 2019-03-26 | International Business Machines Corporation | Slice rebuilding in a dispersed storage network |
US10409678B2 (en) | 2012-08-31 | 2019-09-10 | Pure Storage, Inc. | Self-optimizing read-ahead |
US9875158B2 (en) | 2012-08-31 | 2018-01-23 | International Business Machines Corporation | Slice storage in a dispersed storage network |
US9176822B2 (en) | 2012-08-31 | 2015-11-03 | Cleversafe, Inc. | Adjusting dispersed storage error encoding parameters |
US10331518B2 (en) | 2012-08-31 | 2019-06-25 | International Business Machines Corporation | Encoding data in a dispersed storage network |
US10409679B2 (en) | 2012-08-31 | 2019-09-10 | Pure Storage, Inc. | Migrating data slices in a dispersed storage network |
US10318549B2 (en) | 2012-09-13 | 2019-06-11 | International Business Machines Corporation | Batching modifications to nodes in a dispersed index |
US10057351B2 (en) | 2012-09-13 | 2018-08-21 | International Business Machines Corporation | Modifying information dispersal algorithm configurations in a dispersed storage network |
US10402423B2 (en) | 2012-09-13 | 2019-09-03 | Pure Storage, Inc. | Sliding windows for batching index updates |
US10417253B2 (en) | 2012-09-13 | 2019-09-17 | Pure Storage, Inc. | Multi-level data storage in a dispersed storage network |
US10331698B2 (en) | 2012-09-13 | 2019-06-25 | International Business Machines Corporation | Rebuilding data in a dispersed storage network |
US9424326B2 (en) | 2012-09-13 | 2016-08-23 | International Business Machines Corporation | Writing data avoiding write conflicts in a dispersed storage network |
US10606700B2 (en) | 2012-10-08 | 2020-03-31 | Pure Storage, Inc. | Enhanced dispersed storage error encoding using multiple encoding layers |
US10127111B2 (en) | 2012-10-08 | 2018-11-13 | International Business Machines Corporation | Client provided request prioritization hints |
US9648087B2 (en) | 2012-10-08 | 2017-05-09 | International Business Machines Corporation | Allocating distributed storage and task execution resources |
US10331519B2 (en) | 2012-10-08 | 2019-06-25 | International Business Machines Corporation | Application of secret sharing schemes at multiple levels of a dispersed storage network |
US9794337B2 (en) | 2012-10-30 | 2017-10-17 | International Business Machines Corporation | Balancing storage node utilization of a dispersed storage network |
US9311179B2 (en) | 2012-10-30 | 2016-04-12 | Cleversafe, Inc. | Threshold decoding of data based on trust levels |
US9521197B2 (en) | 2012-12-05 | 2016-12-13 | International Business Machines Corporation | Utilizing data object storage tracking in a dispersed storage network |
US9811533B2 (en) | 2012-12-05 | 2017-11-07 | International Business Machines Corporation | Accessing distributed computing functions in a distributed computing system |
US10558621B2 (en) | 2012-12-05 | 2020-02-11 | Pure Storage, Inc. | Lock stealing writes for improved reliability |
US10587691B2 (en) | 2012-12-05 | 2020-03-10 | Pure Storage, Inc. | Impatient writes |
US10241866B2 (en) | 2013-01-04 | 2019-03-26 | International Business Machines Corporation | Allocating rebuilding queue entries in a dispersed storage network |
US10642992B2 (en) | 2013-01-04 | 2020-05-05 | Pure Storage, Inc. | Password augmented all-or-nothin transform |
US9311187B2 (en) | 2013-01-04 | 2016-04-12 | Cleversafe, Inc. | Achieving storage compliance in a dispersed storage network |
US10013203B2 (en) | 2013-01-04 | 2018-07-03 | International Business Machines Corporation | Achieving storage compliance in a dispersed storage network |
US10423491B2 (en) | 2013-01-04 | 2019-09-24 | Pure Storage, Inc. | Preventing multiple round trips when writing to target widths |
US9558067B2 (en) | 2013-01-04 | 2017-01-31 | International Business Machines Corporation | Mapping storage of data in a dispersed storage network |
US10402270B2 (en) | 2013-01-04 | 2019-09-03 | Pure Storage, Inc. | Deterministically determining affinity for a source name range |
US20190250823A1 (en) | 2013-01-04 | 2019-08-15 | International Business Machines Corporation | Efficient computation of only the required slices |
US10204009B2 (en) | 2013-01-04 | 2019-02-12 | International Business Machines Corporation | Prioritized rebuilds using dispersed indices |
US10229002B2 (en) | 2013-01-04 | 2019-03-12 | International Business Machines Corporation | Process to migrate named objects to a dispersed or distributed storage network (DSN) |
US11416340B1 (en) | 2013-01-04 | 2022-08-16 | Pure Storage, Inc. | Storage system with multiple storage types in a vast storage network |
US10055441B2 (en) | 2013-02-05 | 2018-08-21 | International Business Machines Corporation | Updating shared group information in a dispersed storage network |
US10268554B2 (en) | 2013-02-05 | 2019-04-23 | International Business Machines Corporation | Using dispersed computation to change dispersal characteristics |
US10310763B2 (en) | 2013-02-05 | 2019-06-04 | International Business Machines Corporation | Forming a distributed storage network memory without namespace aware distributed storage units |
US10621021B2 (en) | 2013-02-05 | 2020-04-14 | Pure Storage, Inc. | Using dispersed data structures to point to slice or date source replicas |
US10430122B2 (en) | 2013-02-05 | 2019-10-01 | Pure Storage, Inc. | Using partial rebuilding to change information dispersal algorithm (IDA) |
US10664360B2 (en) | 2013-02-05 | 2020-05-26 | Pure Storage, Inc. | Identifying additional resources to accelerate rebuildling |
US9043499B2 (en) | 2013-02-05 | 2015-05-26 | Cleversafe, Inc. | Modifying a dispersed storage network memory data access response plan |
US9274908B2 (en) | 2013-02-26 | 2016-03-01 | International Business Machines Corporation | Resolving write conflicts in a dispersed storage network |
US11036392B2 (en) | 2013-02-26 | 2021-06-15 | Pure Storage, Inc. | Determining when to use convergent encryption |
US10642489B2 (en) | 2013-02-26 | 2020-05-05 | Pure Storage, Inc. | Determining when to initiate an intra-distributed storage unit rebuild vs. an inter-distributed storage unit rebuild |
US10075523B2 (en) | 2013-04-01 | 2018-09-11 | International Business Machines Corporation | Efficient storage of data in a dispersed storage network |
US10223213B2 (en) | 2013-05-03 | 2019-03-05 | International Business Machines Corporation | Salted zero expansion all or nothing transformation |
US9456035B2 (en) | 2013-05-03 | 2016-09-27 | International Business Machines Corporation | Storing related data in a dispersed storage network |
US9405609B2 (en) | 2013-05-22 | 2016-08-02 | International Business Machines Corporation | Storing data in accordance with a performance threshold |
US11226860B1 (en) | 2013-05-30 | 2022-01-18 | Pure Storage, Inc. | Difference based rebuild list scanning |
US9424132B2 (en) | 2013-05-30 | 2016-08-23 | International Business Machines Corporation | Adjusting dispersed storage network traffic due to rebuilding |
US9432341B2 (en) | 2013-05-30 | 2016-08-30 | International Business Machines Corporation | Securing data in a dispersed storage network |
US9652470B2 (en) | 2013-07-01 | 2017-05-16 | International Business Machines Corporation | Storing data in a dispersed storage network |
US9501360B2 (en) | 2013-07-01 | 2016-11-22 | International Business Machines Corporation | Rebuilding data while reading data in a dispersed storage network |
US10169369B2 (en) | 2013-07-01 | 2019-01-01 | International Business Machines Corporation | Meeting storage requirements with limited storage resources |
US10133635B2 (en) | 2013-07-01 | 2018-11-20 | International Business Machines Corporation | Low-width vault in distributed storage system |
US11221916B2 (en) | 2013-07-01 | 2022-01-11 | Pure Storage, Inc. | Prioritized data reconstruction in a dispersed storage network |
US20150039660A1 (en) | 2013-07-31 | 2015-02-05 | Cleversafe, Inc. | Co-locate objects request |
US10681134B2 (en) | 2013-07-31 | 2020-06-09 | Pure Storage, Inc. | Accelerated learning in adaptive rebuilding by applying observations to other samples |
US10180880B2 (en) | 2013-07-31 | 2019-01-15 | International Business Machines Corporation | Adaptive rebuilding rates based on sampling and inference |
US9565252B2 (en) | 2013-07-31 | 2017-02-07 | International Business Machines Corporation | Distributed storage network with replication control and methods for use therewith |
US20180188964A1 (en) | 2013-07-31 | 2018-07-05 | International Business Machines Corporation | Managed storage unit shutdown in a distributed storage network |
US9438675B2 (en) | 2013-08-29 | 2016-09-06 | International Business Machines Corporation | Dispersed storage with variable slice length and methods for use therewith |
US9661074B2 (en) | 2013-08-29 | 2017-05-23 | International Business Machines Corporations | Updating de-duplication tracking data for a dispersed storage network |
US10489071B2 (en) | 2013-08-29 | 2019-11-26 | Pure Storage, Inc. | Vault provisioning within dispersed or distributed storage network (DSN) |
US10484474B2 (en) | 2013-08-29 | 2019-11-19 | Pure Storage, Inc. | Rotating offline DS units |
US10601918B2 (en) | 2013-08-29 | 2020-03-24 | Pure Storage, Inc. | Rotating inactive storage units in a distributed storage network |
US10514857B2 (en) | 2013-08-29 | 2019-12-24 | Pure Storage, Inc. | Dynamic adjusting of parameters based on resource scoring |
US9857974B2 (en) | 2013-10-03 | 2018-01-02 | International Business Machines Corporation | Session execution decision |
US20180039421A1 (en) * | 2013-10-03 | 2018-02-08 | International Business Machines Corporation | Method for collecting scheduler-relevant information for a task moving through the system |
US10304096B2 (en) | 2013-11-01 | 2019-05-28 | International Business Machines Corporation | Renting a pipe to a storage system |
US9781208B2 (en) | 2013-11-01 | 2017-10-03 | International Business Machines Corporation | Obtaining dispersed storage network system registry information |
US10182115B2 (en) | 2013-11-01 | 2019-01-15 | International Business Machines Corporation | Changing rebuild priority for a class of data |
US9900316B2 (en) | 2013-12-04 | 2018-02-20 | International Business Machines Corporation | Accessing storage units of a dispersed storage network |
US11340993B2 (en) | 2014-01-06 | 2022-05-24 | Pure Storage, Inc. | Deferred rebuilding with alternate storage locations |
US10922181B2 (en) | 2014-01-06 | 2021-02-16 | Pure Storage, Inc. | Using storage locations greater than an IDA width in a dispersed storage network |
US9594639B2 (en) | 2014-01-06 | 2017-03-14 | International Business Machines Corporation | Configuring storage resources of a dispersed storage network |
US10318382B2 (en) | 2014-01-31 | 2019-06-11 | International Business Machines Corporation | Determining missing encoded data slices |
US9778987B2 (en) | 2014-01-31 | 2017-10-03 | International Business Machines Corporation | Writing encoded data slices in a dispersed storage network |
US11204836B1 (en) | 2014-01-31 | 2021-12-21 | Pure Storage, Inc. | Using trap slices for anomaly detection in a distributed storage network |
US9552261B2 (en) | 2014-01-31 | 2017-01-24 | International Business Machines Corporation | Recovering data from microslices in a dispersed storage network |
US10592109B2 (en) | 2014-02-26 | 2020-03-17 | Pure Storage, Inc. | Selecting storage resources in a dispersed storage network |
US10140182B2 (en) | 2014-02-26 | 2018-11-27 | International Business Machines Corporation | Modifying allocation of storage resources in a dispersed storage network |
US10635312B2 (en) | 2014-02-26 | 2020-04-28 | Pure Storage, Inc. | Recovering data in a dispersed storage network |
US9665429B2 (en) | 2014-02-26 | 2017-05-30 | International Business Machines Corporation | Storage of data with verification in a dispersed storage network |
US10769016B2 (en) | 2014-02-26 | 2020-09-08 | Pure Storage, Inc. | Storing a plurality of correlated data in a dispersed storage network |
US9529834B2 (en) | 2014-02-26 | 2016-12-27 | International Business Machines Corporation | Concatenating data objects for storage in a dispersed storage network |
US10678638B2 (en) | 2014-02-26 | 2020-06-09 | Pure Storage, Inc. | Resolving write conflicts in a dispersed storage network |
US20190087599A1 (en) | 2014-04-02 | 2019-03-21 | International Business Machines Corporation | Compressing a slice name listing in a dispersed storage network |
US11347590B1 (en) | 2014-04-02 | 2022-05-31 | Pure Storage, Inc. | Rebuilding data in a distributed storage network |
US10015152B2 (en) | 2014-04-02 | 2018-07-03 | International Business Machines Corporation | Securing data in a dispersed storage network |
US10628245B2 (en) | 2014-04-02 | 2020-04-21 | Pure Storage, Inc. | Monitoring of storage units in a dispersed storage network |
US10761917B2 (en) | 2014-04-02 | 2020-09-01 | Pure Storage, Inc. | Using global namespace addressing in a dispersed storage network |
US10020826B2 (en) | 2014-04-02 | 2018-07-10 | International Business Machines Corporation | Generating molecular encoding information for data storage |
US10681138B2 (en) | 2014-04-02 | 2020-06-09 | Pure Storage, Inc. | Storing and retrieving multi-format content in a distributed storage network |
US20150288680A1 (en) | 2014-04-02 | 2015-10-08 | Cleversafe, Inc. | Distributing registry information in a dispersed storage network |
US10296263B2 (en) | 2014-04-30 | 2019-05-21 | International Business Machines Corporation | Dispersed bloom filter for determining presence of an object |
US9735967B2 (en) | 2014-04-30 | 2017-08-15 | International Business Machines Corporation | Self-validating request message structure and operation |
US10802732B2 (en) | 2014-04-30 | 2020-10-13 | Pure Storage, Inc. | Multi-level stage locality selection on a large system |
US9965336B2 (en) | 2014-04-30 | 2018-05-08 | International Business Machines Corporation | Delegating iterative storage unit access in a dispersed storage network |
US10394476B2 (en) | 2014-04-30 | 2019-08-27 | Pure Storage, Inc. | Multi-level stage locality selection on a large system |
US10152601B2 (en) | 2014-06-05 | 2018-12-11 | International Business Machines Corporation | Reliably recovering stored data in a dispersed storage network |
US10509577B2 (en) | 2014-06-05 | 2019-12-17 | Pure Storage, Inc. | Reliable storage in a dispersed storage network |
US10440105B2 (en) | 2014-06-30 | 2019-10-08 | Pure Storage, Inc. | Using a decentralized agreement protocol to rank storage locations for target width |
US10459797B2 (en) | 2014-06-30 | 2019-10-29 | Pure Storage, Inc. | Making trade-offs between rebuild scanning and failing memory device flexibility |
US10673946B2 (en) | 2014-06-30 | 2020-06-02 | Pure Storage, Inc. | Using separate weighting scores for different types of data in a decentralized agreement protocol |
US9841925B2 (en) | 2014-06-30 | 2017-12-12 | International Business Machines Corporation | Adjusting timing of storing data in a dispersed storage network |
US11398988B1 (en) | 2014-06-30 | 2022-07-26 | Pure Storage, Inc. | Selection of access resources in a distributed storage network |
US9838478B2 (en) | 2014-06-30 | 2017-12-05 | International Business Machines Corporation | Identifying a task execution resource of a dispersed storage network |
US11606431B2 (en) | 2014-06-30 | 2023-03-14 | Pure Storage, Inc. | Maintaining failure independence for storage of a set of encoded data slices |
US10042564B2 (en) | 2014-06-30 | 2018-08-07 | International Business Machines Corporation | Accessing data while migrating storage of the data |
US9923838B2 (en) | 2014-06-30 | 2018-03-20 | International Business Machines Corporation | Accessing a dispersed storage network |
US11099763B1 (en) | 2014-06-30 | 2021-08-24 | Pure Storage, Inc. | Migrating generational storage to a decentralized agreement protocol paradigm |
US9690520B2 (en) | 2014-06-30 | 2017-06-27 | International Business Machines Corporation | Recovering an encoded data slice in a dispersed storage network |
US10644874B2 (en) | 2014-07-31 | 2020-05-05 | Pure Storage, Inc. | Limiting brute force attacks against dispersed credentials in a distributed storage system |
US10089036B2 (en) | 2014-07-31 | 2018-10-02 | International Business Machines Corporation | Migrating data in a distributed storage network |
US11728964B2 (en) | 2014-07-31 | 2023-08-15 | Pure Storage, Inc. | Performance aided data migration in a distributed storage network |
US10613936B2 (en) | 2014-07-31 | 2020-04-07 | Pure Storage, Inc. | Fractional slices in a distributed storage system |
US10049120B2 (en) | 2014-09-05 | 2018-08-14 | International Business Machines Corporation | Consistency based access of data in a dispersed storage network |
US10176191B2 (en) | 2014-09-05 | 2019-01-08 | International Business Machines Corporation | Recovering from conflicts that emerge from eventually consistent operations |
US10402395B2 (en) | 2014-09-05 | 2019-09-03 | Pure Storage, Inc. | Facilitating data consistency in a dispersed storage network |
US11442921B1 (en) | 2014-09-05 | 2022-09-13 | Pure Storage, Inc. | Data access in a dispersed storage network with consistency |
US10146622B2 (en) | 2014-09-08 | 2018-12-04 | International Business Machines Corporation | Combining deduplication with locality for efficient and fast storage |
US10268545B2 (en) | 2014-09-08 | 2019-04-23 | International Business Machines Corporation | Using reinforcement learning to select a DS processing unit |
US9591076B2 (en) | 2014-09-08 | 2017-03-07 | International Business Machines Corporation | Maintaining a desired number of storage units |
US10095582B2 (en) * | 2014-10-29 | 2018-10-09 | International Business Machines Corporation | Partial rebuilding techniques in a dispersed storage unit |
US10459792B2 (en) | 2014-10-29 | 2019-10-29 | Pure Storage, Inc. | Using an eventually consistent dispersed memory to implement storage tiers |
US10282135B2 (en) | 2014-10-29 | 2019-05-07 | International Business Machines Corporation | Strong consistency write threshold |
US20180101457A1 (en) | 2014-10-29 | 2018-04-12 | International Business Machines Corporation | Retrying failed write operations in a dispersed storage network |
US10481833B2 (en) | 2014-10-29 | 2019-11-19 | Pure Storage, Inc. | Transferring data encoding functions in a distributed storage network |
US9916114B2 (en) | 2014-10-29 | 2018-03-13 | International Business Machines Corporation | Deterministically sharing a plurality of processing resources |
US10223033B2 (en) | 2014-10-29 | 2019-03-05 | International Business Machines Corporation | Coordinating arrival times of data slices in a dispersed storage network |
US10481832B2 (en) | 2014-12-02 | 2019-11-19 | Pure Storage, Inc. | Applying a probability function to avoid storage operations for already-deleted data |
US10521298B2 (en) | 2014-12-02 | 2019-12-31 | Pure Storage, Inc. | Temporarily storing dropped and rebuilt slices in a DSN memory |
US10503592B2 (en) | 2014-12-02 | 2019-12-10 | Pure Storage, Inc. | Overcoming bottlenecks in partial and traditional rebuild operations |
US10120739B2 (en) | 2014-12-02 | 2018-11-06 | International Business Machines Corporation | Prioritized data rebuilding in a dispersed storage network |
US10558527B2 (en) | 2014-12-02 | 2020-02-11 | Pure Storage, Inc. | Rebuilding strategy in memory managed multi-site duplication |
US9727275B2 (en) | 2014-12-02 | 2017-08-08 | International Business Machines Corporation | Coordinating storage of data in dispersed storage networks |
US10402271B2 (en) | 2014-12-02 | 2019-09-03 | Pure Storage, Inc. | Overcoming bottlenecks in zero information gain (ZIG) rebuild operations |
US10126974B2 (en) | 2014-12-31 | 2018-11-13 | International Business Machines Corporation | Redistributing encoded data slices in a dispersed storage network |
US10452317B2 (en) | 2014-12-31 | 2019-10-22 | Pure Storage, Inc. | DAP redistribution operation within a dispersed storage network |
US10621042B2 (en) | 2014-12-31 | 2020-04-14 | Pure Storage, Inc. | Vault transformation within a dispersed storage network |
US10387252B2 (en) | 2014-12-31 | 2019-08-20 | Pure Storage, Inc. | Synchronously storing data in a plurality of dispersed storage networks |
US10642687B2 (en) | 2014-12-31 | 2020-05-05 | Pure Storage, Inc. | Pessimistic reads and other smart-read enhancements with synchronized vaults |
US10656866B2 (en) | 2014-12-31 | 2020-05-19 | Pure Storage, Inc. | Unidirectional vault synchronization to support tiering |
US10489247B2 (en) | 2014-12-31 | 2019-11-26 | Pure Storage, Inc. | Generating time-ordered globally unique revision numbers |
US10423359B2 (en) | 2014-12-31 | 2019-09-24 | Pure Storage, Inc. | Linking common attributes among a set of synchronized vaults |
US10623495B2 (en) | 2014-12-31 | 2020-04-14 | Pure Storage, Inc. | Keeping synchronized writes from getting out of synch |
US11604707B2 (en) | 2014-12-31 | 2023-03-14 | Pure Storage, Inc. | Handling failures when synchronizing objects during a write operation |
US9727427B2 (en) | 2014-12-31 | 2017-08-08 | International Business Machines Corporation | Synchronizing storage of data copies in a dispersed storage network |
US10498823B2 (en) | 2015-01-30 | 2019-12-03 | Pure Storage, Inc. | Optimally apportioning rebuilding resources |
US10506045B2 (en) | 2015-01-30 | 2019-12-10 | Pure Storage, Inc. | Memory access using deterministic function and secure seed |
US10169123B2 (en) | 2015-01-30 | 2019-01-01 | International Business Machines Corporation | Distributed data rebuilding |
US10740180B2 (en) | 2015-01-30 | 2020-08-11 | Pure Storage, Inc. | Storing and retrieving data using proxies |
US10423490B2 (en) | 2015-01-30 | 2019-09-24 | Pure Storage, Inc. | Read-source requests to support bundled writes in a distributed storage system |
US10498822B2 (en) | 2015-01-30 | 2019-12-03 | Pure Storage, Inc. | Adaptive scanning rates |
US10511665B2 (en) | 2015-01-30 | 2019-12-17 | Pure Storage, Inc. | Efficient resource reclamation after deletion of slice from common file |
US9740547B2 (en) | 2015-01-30 | 2017-08-22 | International Business Machines Corporation | Storing data using a dual path storage approach |
US10802915B2 (en) | 2015-01-30 | 2020-10-13 | Pure Storage, Inc. | Time based storage of encoded data slices |
US10530862B2 (en) | 2015-01-30 | 2020-01-07 | Pure Storage, Inc. | Determining slices to rebuild from low-level failures |
US10289342B2 (en) | 2015-01-30 | 2019-05-14 | International Business Machines Corporation | Data access optimization protocol in a dispersed storage network |
US10592132B2 (en) | 2015-01-30 | 2020-03-17 | Pure Storage, Inc. | Read-foreign-slices request for improved read efficiency with bundled writes |
US10620878B2 (en) | 2015-01-30 | 2020-04-14 | Pure Storage, Inc. | Write threshold plus value in dispersed storage network write operations |
US10440116B2 (en) | 2015-01-30 | 2019-10-08 | Pure Storage, Inc. | Minimizing data movement through rotation of spare memory devices |
US9826038B2 (en) | 2015-01-30 | 2017-11-21 | International Business Machines Corporation | Selecting a data storage resource of a dispersed storage network |
US10594793B2 (en) | 2015-01-30 | 2020-03-17 | Pure Storage, Inc. | Read-prepare requests to multiple memories |
US10409772B2 (en) | 2015-02-27 | 2019-09-10 | Pure Storage, Inc. | Accessing serially stored data in a dispersed storage network |
US10078472B2 (en) | 2015-02-27 | 2018-09-18 | International Business Machines Corporation | Rebuilding encoded data slices in a dispersed storage network |
US10579451B2 (en) | 2015-02-27 | 2020-03-03 | Pure Storage, Inc. | Pro-actively preparing a dispersed storage network memory for higher-loads |
US10423502B2 (en) | 2015-02-27 | 2019-09-24 | Pure Storage, Inc. | Stand-by distributed storage units |
US10387067B2 (en) | 2015-02-27 | 2019-08-20 | Pure Storage, Inc. | Optimizing data storage in a dispersed storage network |
US10437677B2 (en) | 2015-02-27 | 2019-10-08 | Pure Storage, Inc. | Optimized distributed rebuilding within a dispersed storage network |
US10440115B2 (en) | 2015-02-27 | 2019-10-08 | Pure Storage, Inc. | Write intent messaging in a dispersed storage network |
US10437676B2 (en) | 2015-02-27 | 2019-10-08 | Pure Storage, Inc. | Urgent reads and using data source health to determine error recovery procedures |
US10275185B2 (en) | 2015-02-27 | 2019-04-30 | International Business Machines Corporation | Fail-in-place supported via decentralized or Distributed Agreement Protocol (DAP) |
US10534668B2 (en) | 2015-02-27 | 2020-01-14 | Pure Storage, Inc. | Accessing data in a dispersed storage network |
US10657000B2 (en) | 2015-02-27 | 2020-05-19 | Pure Storage, Inc. | Optimizing data storage in a dispersed storage network |
US11188665B2 (en) | 2015-02-27 | 2021-11-30 | Pure Storage, Inc. | Using internal sensors to detect adverse interference and take defensive actions |
US10069915B2 (en) | 2015-02-27 | 2018-09-04 | International Business Machines Corporation | Storing data in a dispersed storage network |
US10528425B2 (en) | 2015-02-27 | 2020-01-07 | Pure Storage, Inc. | Transitioning to an optimized data storage approach in a dispersed storage network |
US10404410B2 (en) | 2015-02-27 | 2019-09-03 | Pure Storage, Inc. | Storage unit (SU) report cards |
US11836369B1 (en) | 2015-02-27 | 2023-12-05 | Pure Storage, Inc. | Storing data in an expanded storage pool of a vast storage network |
US10503591B2 (en) | 2015-02-27 | 2019-12-10 | Pure Storage, Inc. | Selecting retrieval locations in a dispersed storage network |
US10530861B2 (en) | 2015-02-27 | 2020-01-07 | Pure Storage, Inc. | Utilizing multiple storage pools in a dispersed storage network |
US10331384B2 (en) | 2015-03-31 | 2019-06-25 | International Business Machines Corporation | Storing data utilizing a maximum accessibility approach in a dispersed storage network |
US10534661B2 (en) | 2015-03-31 | 2020-01-14 | Pure Storage, Inc. | Selecting a storage error abatement alternative in a dispersed storage network |
US10713374B2 (en) | 2015-03-31 | 2020-07-14 | Pure Storage, Inc. | Resolving detected access anomalies in a dispersed storage network |
US10437515B2 (en) | 2015-03-31 | 2019-10-08 | Pure Storage, Inc. | Selecting storage units in a dispersed storage network |
US10915261B2 (en) | 2015-03-31 | 2021-02-09 | Pure Storage, Inc. | Selecting a set of storage units in a distributed storage network |
US10387070B2 (en) | 2015-03-31 | 2019-08-20 | Pure Storage, Inc. | Migrating data in response to adding incremental storage resources in a dispersed storage network |
US10528282B2 (en) | 2015-03-31 | 2020-01-07 | Pure Storage, Inc. | Modifying and utilizing a file structure in a dispersed storage network |
US10852957B2 (en) | 2015-03-31 | 2020-12-01 | Pure Storage, Inc. | Migration agent employing moveslice request |
US11055177B2 (en) | 2015-03-31 | 2021-07-06 | Pure Storage, Inc. | Correlating operational information with an error condition in a dispersed storage network |
US10963180B2 (en) | 2015-03-31 | 2021-03-30 | Pure Storage, Inc. | Adding incremental storage resources in a dispersed storage network |
US10079887B2 (en) | 2015-03-31 | 2018-09-18 | International Business Machines Corporation | Expanding storage capacity of a set of storage units in a distributed storage network |
US10078561B2 (en) | 2015-04-30 | 2018-09-18 | International Business Machines Corporation | Handling failing memory devices in a dispersed storage network |
US10157094B2 (en) | 2015-04-30 | 2018-12-18 | International Business Machines Corporation | Validating system registry files in a dispersed storage network |
US10168904B2 (en) | 2015-04-30 | 2019-01-01 | International Business Machines Corporation | Quasi-error notifications in a dispersed storage network |
US10037171B2 (en) | 2015-04-30 | 2018-07-31 | International Business Machines Corporation | Accessing common data in a dispersed storage network |
US10216594B2 (en) | 2015-04-30 | 2019-02-26 | International Business Machines Corporation | Automated stalled process detection and recovery |
US10055170B2 (en) | 2015-04-30 | 2018-08-21 | International Business Machines Corporation | Scheduling storage unit maintenance tasks in a dispersed storage network |
US10268376B2 (en) | 2015-04-30 | 2019-04-23 | International Business Machines Corporation | Automated deployment and assignment of access devices in a dispersed storage network |
US10067998B2 (en) | 2015-04-30 | 2018-09-04 | International Business Machines Corporation | Distributed sync list |
US10254992B2 (en) | 2015-04-30 | 2019-04-09 | International Business Machines Corporation | Rebalancing data storage in a dispersed storage network |
US10402122B2 (en) | 2015-05-29 | 2019-09-03 | Pure Storage, Inc. | Transferring encoded data slices in a dispersed storage network |
US10838664B2 (en) | 2015-05-29 | 2020-11-17 | Pure Storage, Inc. | Determining a storage location according to legal requirements |
US10789128B2 (en) | 2015-05-29 | 2020-09-29 | Pure Storage, Inc. | External healing mode for a dispersed storage network memory |
US10613798B2 (en) | 2015-05-29 | 2020-04-07 | Pure Storage, Inc. | Slice fanout write request |
US11115221B2 (en) | 2015-05-29 | 2021-09-07 | Pure Storage, Inc. | Verifying a rebuilt encoded data slice using slice verification information |
US10324657B2 (en) | 2015-05-29 | 2019-06-18 | International Business Machines Corporation | Accounting for data whose rebuilding is deferred |
US10430107B2 (en) | 2015-05-29 | 2019-10-01 | Pure Storage, Inc. | Identifying stored data slices during a slice migration activity in a dispersed storage network |
US10409522B2 (en) | 2015-05-29 | 2019-09-10 | Pure Storage, Inc. | Reclaiming storage capacity in a dispersed storage network |
US10523241B2 (en) | 2015-05-29 | 2019-12-31 | Pure Storage, Inc. | Object fan out write operation |
US10891058B2 (en) | 2015-05-29 | 2021-01-12 | Pure Storage, Inc. | Encoding slice verification information to support verifiable rebuilding |
US10169125B2 (en) | 2015-05-29 | 2019-01-01 | International Business Machines Corporation | Re-encoding data in a dispersed storage network |
US10437671B2 (en) | 2015-06-30 | 2019-10-08 | Pure Storage, Inc. | Synchronizing replicated stored data |
US10055291B2 (en) | 2015-06-30 | 2018-08-21 | International Business Machines Corporation | Method and system for processing data access requests during data transfers |
US11669546B2 (en) | 2015-06-30 | 2023-06-06 | Pure Storage, Inc. | Synchronizing replicated data in a storage network |
US10467096B2 (en) | 2015-07-31 | 2019-11-05 | Pure Storage, Inc. | Securely storing data in a dispersed storage network |
US11782789B2 (en) | 2015-07-31 | 2023-10-10 | Pure Storage, Inc. | Encoding data and associated metadata in a storage network |
CN106469100B (en) | 2015-08-17 | 2019-04-05 | 华为技术有限公司 | A kind of method that data are restored, the corresponding apparatus and system of the method for storage |
US10466914B2 (en) | 2015-08-31 | 2019-11-05 | Pure Storage, Inc. | Verifying authorized access in a dispersed storage network |
US10073652B2 (en) | 2015-09-24 | 2018-09-11 | International Business Machines Corporation | Performance optimized storage vaults in a dispersed storage network |
US10169147B2 (en) | 2015-10-30 | 2019-01-01 | International Business Machines Corporation | End-to-end secure data storage in a dispersed storage network |
US10409514B2 (en) | 2015-11-30 | 2019-09-10 | International Business Machines Corporation | IP multicast message transmission for event notifications |
US10346246B2 (en) | 2015-11-30 | 2019-07-09 | International Business Machines Corporation | Recovering data copies in a dispersed storage network |
US20170192688A1 (en) | 2015-12-30 | 2017-07-06 | International Business Machines Corporation | Lazy deletion of vaults in packed slice storage (pss) and zone slice storage (zss) |
US10855759B2 (en) | 2016-01-26 | 2020-12-01 | Pure Storage, Inc. | Utilizing a hierarchical index in a dispersed storage network |
CN105718334B (en) * | 2016-01-27 | 2019-02-22 | 四川驭奔科技有限公司 | Multiple file extraction methods based on feature |
CN105718335B (en) * | 2016-01-27 | 2019-02-22 | 四川驭奔科技有限公司 | Single file extracting method based on feature |
US10089178B2 (en) | 2016-02-29 | 2018-10-02 | International Business Machines Corporation | Developing an accurate dispersed storage network memory performance model through training |
US10387248B2 (en) | 2016-03-29 | 2019-08-20 | International Business Machines Corporation | Allocating data for storage by utilizing a location-based hierarchy in a dispersed storage network |
US10831381B2 (en) | 2016-03-29 | 2020-11-10 | International Business Machines Corporation | Hierarchies of credential and access control sharing between DSN memories |
US10419538B2 (en) | 2016-04-26 | 2019-09-17 | International Business Machines Corporation | Selecting memory for data access in a dispersed storage network |
US10169082B2 (en) | 2016-04-27 | 2019-01-01 | International Business Machines Corporation | Accessing data in accordance with an execution deadline |
US10007444B2 (en) | 2016-04-29 | 2018-06-26 | International Business Machines Corporation | Batching access requests in a dispersed storage network |
US10628399B2 (en) | 2016-04-29 | 2020-04-21 | International Business Machines Corporation | Storing data in a dispersed storage network with consistency |
US10091298B2 (en) | 2016-05-27 | 2018-10-02 | International Business Machines Corporation | Enhancing performance of data storage in a dispersed storage network |
US10353772B2 (en) | 2016-05-31 | 2019-07-16 | International Business Machines Corporation | Selecting data for storage in a dispersed storage network |
US10122795B2 (en) | 2016-05-31 | 2018-11-06 | International Business Machines Corporation | Consistency level driven data storage in a dispersed storage network |
US10027755B2 (en) | 2016-06-01 | 2018-07-17 | International Business Machines Corporation | Selecting storage units in one or more dispersed storage networks |
US10394650B2 (en) | 2016-06-03 | 2019-08-27 | International Business Machines Corporation | Multiple writes using inter-site storage unit relationship |
US10735545B2 (en) | 2016-06-06 | 2020-08-04 | International Business Machines Corporation | Routing vault access requests in a dispersed storage network |
US10334045B2 (en) | 2016-06-06 | 2019-06-25 | International Business Machines Corporation | Indicating multiple encoding schemes in a dispersed storage network |
US10652350B2 (en) | 2016-06-06 | 2020-05-12 | International Business Machines Corporation | Caching for unique combination reads in a dispersed storage network |
US10719499B2 (en) | 2016-06-06 | 2020-07-21 | INTERNATIONAL BUSINESS MACHINES CORPORATIOb | Establishing distributed consensus via alternate voting strategies in a dispersed storage network |
US10007438B2 (en) | 2016-06-25 | 2018-06-26 | International Business Machines Corporation | Method and system for achieving consensus using alternate voting strategies (AVS) with incomplete information |
US10564852B2 (en) | 2016-06-25 | 2020-02-18 | International Business Machines Corporation | Method and system for reducing memory device input/output operations |
US10235085B2 (en) | 2016-06-27 | 2019-03-19 | International Business Machines Corporation | Relocating storage unit data in response to detecting hotspots in a dispersed storage network |
US11115469B2 (en) | 2016-06-28 | 2021-09-07 | International Business Machines Corporation | Efficient updates within a dispersed storage network |
US10025505B2 (en) | 2016-06-29 | 2018-07-17 | International Business Machines Corporation | Accessing data in a dispersed storage network during write operations |
US10157021B2 (en) | 2016-06-29 | 2018-12-18 | International Business Machines Corporation | Processing incomplete data access transactions |
US10387286B2 (en) | 2016-06-30 | 2019-08-20 | International Business Machines Corporation | Managing configuration updates in a dispersed storage network |
US9934092B2 (en) | 2016-07-12 | 2018-04-03 | International Business Machines Corporation | Manipulating a distributed agreement protocol to identify a desired set of storage units |
US10114696B2 (en) | 2016-07-14 | 2018-10-30 | International Business Machines Corporation | Tracking data access in a dispersed storage network |
US10102067B2 (en) | 2016-07-14 | 2018-10-16 | International Business Machines Corporation | Performing a desired manipulation of an encoded data slice based on a metadata restriction and a storage operational condition |
US10534666B2 (en) | 2016-07-14 | 2020-01-14 | International Business Machines Corporation | Determining storage requirements based on licensing right in a dispersed storage network |
US10360103B2 (en) | 2016-07-18 | 2019-07-23 | International Business Machines Corporation | Focused storage pool expansion to prevent a performance degradation |
US9992063B2 (en) | 2016-07-18 | 2018-06-05 | International Business Machines Corporation | Utilizing reallocation via a decentralized, or distributed, agreement protocol (DAP) for storage unit (SU) replacement |
US10277490B2 (en) | 2016-07-19 | 2019-04-30 | International Business Machines Corporation | Monitoring inter-site bandwidth for rebuilding |
US10769015B2 (en) | 2016-07-19 | 2020-09-08 | International Business Machines Corporation | Throttling access requests at different layers of a DSN memory |
US10554752B2 (en) | 2016-07-20 | 2020-02-04 | International Business Machines Corporation | Efficient transfer of encoded data slice sets to new or alternate storage units |
US10031809B2 (en) | 2016-07-20 | 2018-07-24 | International Business Machines Corporation | Efficient method for rebuilding a set of encoded data slices |
US10127112B2 (en) | 2016-07-20 | 2018-11-13 | International Business Machines Corporation | Assigning prioritized rebuild resources optimally |
US10459796B2 (en) | 2016-07-20 | 2019-10-29 | International Business Machines Corporation | Prioritizing rebuilding based on a longevity estimate of the rebuilt slice |
US10379744B2 (en) | 2016-07-21 | 2019-08-13 | International Business Machines Corporation | System for collecting end-user feedback and usability metrics |
US10416930B2 (en) | 2016-07-21 | 2019-09-17 | International Business Machines Corporation | Global access permit listing |
US10459790B2 (en) | 2016-07-26 | 2019-10-29 | International Business Machines Corporation | Elastic storage in a dispersed storage network |
US10395043B2 (en) | 2016-07-29 | 2019-08-27 | International Business Machines Corporation | Securely storing data in an elastically scalable dispersed storage network |
US10031805B2 (en) | 2016-08-09 | 2018-07-24 | International Business Machines Corporation | Assigning slices to storage locations based on a predicted lifespan |
US10223036B2 (en) | 2016-08-10 | 2019-03-05 | International Business Machines Corporation | Expanding a dispersed storage network (DSN) |
US10129023B2 (en) | 2016-08-11 | 2018-11-13 | International Business Machines Corporation | Enhancing security for multiple storage configurations |
US10348829B2 (en) | 2016-08-15 | 2019-07-09 | International Business Machines Corporation | Auto indexing with customizable metadata |
US10191809B2 (en) | 2016-08-17 | 2019-01-29 | International Business Machines Corporation | Converting a data chunk into a ring algebraic structure for fast erasure coding |
US10013309B2 (en) | 2016-08-17 | 2018-07-03 | International Business Machines Corporation | Missing slice reconstruction in a dispersed storage network |
US10078468B2 (en) | 2016-08-18 | 2018-09-18 | International Business Machines Corporation | Slice migration in a dispersed storage network |
US10379778B2 (en) | 2016-08-18 | 2019-08-13 | International Business Machines Corporation | Using a master encryption key to sanitize a dispersed storage network memory |
US10389683B2 (en) | 2016-08-26 | 2019-08-20 | International Business Machines Corporation | Securing storage units in a dispersed storage network |
US10379773B2 (en) | 2016-08-29 | 2019-08-13 | International Business Machines Corporation | Storage unit for use in a dispersed storage network |
US10581807B2 (en) | 2016-08-29 | 2020-03-03 | International Business Machines Corporation | Using dispersal techniques to securely store cryptographic resources and respond to attacks |
US10061524B2 (en) | 2016-09-01 | 2018-08-28 | International Business Machines Corporation | Wear-leveling of memory devices |
US10169149B2 (en) | 2016-09-06 | 2019-01-01 | International Business Machines Corporation | Standard and non-standard dispersed storage network data access |
US10387079B2 (en) | 2016-09-09 | 2019-08-20 | International Business Machines Corporation | Placement of dispersed storage data based on requestor properties |
US10225271B2 (en) | 2016-09-09 | 2019-03-05 | International Business Machines Corporation | Distributed storage network with enhanced security monitoring |
US10547615B2 (en) | 2016-09-12 | 2020-01-28 | International Business Machines Corporation | Security response protocol based on security alert encoded data slices of a distributed storage network |
US10558396B2 (en) | 2016-09-14 | 2020-02-11 | International Business Machines Corporation | Pre-caching data according to a current or predicted requester location |
US10558389B2 (en) | 2016-09-20 | 2020-02-11 | International Business Machines Corporation | Per-storage class quality of service (QoS) management within a distributed storage network (DSN) where the DSN stores data using dispersed storage error decoding/encoding |
US10067822B2 (en) | 2016-09-26 | 2018-09-04 | International Business Machines Corporation | Combined slice objects in alternate memory locations |
US10394630B2 (en) | 2016-10-26 | 2019-08-27 | International Business Machines Corporation | Estimating relative data importance in a dispersed storage network |
US10448062B2 (en) | 2016-10-26 | 2019-10-15 | International Business Machines Corporation | Pre-fetching media content to reduce peak loads |
US10585751B2 (en) | 2016-10-27 | 2020-03-10 | International Business Machines Corporation | Partial rebuild operation within a dispersed storage network including local memory and cloud-based alternative memory |
US10481977B2 (en) | 2016-10-27 | 2019-11-19 | International Business Machines Corporation | Dispersed storage of error encoded data objects having multiple resolutions |
US11169731B2 (en) | 2016-10-31 | 2021-11-09 | International Business Machines Corporation | Managing storage resources in a dispersed storage network |
US10540247B2 (en) | 2016-11-10 | 2020-01-21 | International Business Machines Corporation | Handling degraded conditions using a redirect module |
US10585607B2 (en) | 2016-11-10 | 2020-03-10 | International Business Machines Corporation | Determining an optimum selection of functions for units in a DSN memory |
US10114698B2 (en) | 2017-01-05 | 2018-10-30 | International Business Machines Corporation | Detecting and responding to data loss events in a dispersed storage network |
US10782921B2 (en) | 2017-01-25 | 2020-09-22 | International Business Machines Corporation | Non-writing device finalization of a write operation initiated by another device |
US10180787B2 (en) | 2017-02-09 | 2019-01-15 | International Business Machines Corporation | Dispersed storage write process with lock/persist |
US10241865B2 (en) | 2017-02-15 | 2019-03-26 | International Business Machines Corporation | Handling storage unit failure in a dispersed storage network |
US10579309B2 (en) | 2017-02-16 | 2020-03-03 | International Business Machines Corporation | Method for increasing throughput in a distributed storage network |
US10248495B2 (en) | 2017-02-17 | 2019-04-02 | International Business Machines Corporation | Eventual consistency intent cleanup in a dispersed storage network |
US10552341B2 (en) | 2017-02-17 | 2020-02-04 | International Business Machines Corporation | Zone storage—quickly returning to a state of consistency following an unexpected event |
US10382553B2 (en) | 2017-02-20 | 2019-08-13 | International Business Machines Corporation | Zone storage—resilient and efficient storage transactions |
US10394468B2 (en) | 2017-02-23 | 2019-08-27 | International Business Machines Corporation | Handling data slice revisions in a dispersed storage network |
US10241677B2 (en) | 2017-02-24 | 2019-03-26 | International Business Machines Corporation | Ensuring consistency between content and metadata with intents |
US9998147B1 (en) | 2017-02-27 | 2018-06-12 | International Business Machines Corporation | Method for using write intents in a distributed storage network |
US10642532B2 (en) | 2017-02-28 | 2020-05-05 | International Business Machines Corporation | Storing data sequentially in zones in a dispersed storage network |
US10372380B2 (en) | 2017-03-01 | 2019-08-06 | International Business Machines Corporation | Asserting integrity with a verifiable codec |
US10169392B2 (en) | 2017-03-08 | 2019-01-01 | International Business Machines Corporation | Persistent data structures on a dispersed storage network memory |
US11226980B2 (en) | 2017-03-13 | 2022-01-18 | International Business Machines Corporation | Replicating containers in object storage using intents |
US10235241B2 (en) | 2017-03-15 | 2019-03-19 | International Business Machines Corporation | Method for partial updating data content in a distributed storage network |
US10693640B2 (en) | 2017-03-17 | 2020-06-23 | International Business Machines Corporation | Use of key metadata during write and read operations in a dispersed storage network memory |
US10241861B2 (en) | 2017-03-23 | 2019-03-26 | International Business Machines Corporation | Method for tenant isolation in a distributed computing system |
US10133634B2 (en) | 2017-03-30 | 2018-11-20 | International Business Machines Corporation | Method for performing in-place disk format changes in a distributed storage network |
US10360391B2 (en) | 2017-04-03 | 2019-07-23 | International Business Machines Corporation | Verifiable keyed all-or-nothing transform |
US10545699B2 (en) | 2017-04-11 | 2020-01-28 | International Business Machines Corporation | Dynamic retention policies and optional deletes |
US10379961B2 (en) | 2017-04-11 | 2019-08-13 | International Business Machines Corporation | Ensuring metadata and index consistency using write intents |
US10567509B2 (en) | 2017-05-15 | 2020-02-18 | International Business Machines Corporation | Rebuilding derived content |
US10339003B2 (en) | 2017-06-01 | 2019-07-02 | International Business Machines Corporation | Processing data access transactions in a dispersed storage network using source revision indicators |
US10491386B2 (en) | 2017-06-01 | 2019-11-26 | International Business Machines Corporation | Slice-level keyed encryption with support for efficient rekeying |
US10467097B2 (en) | 2017-06-02 | 2019-11-05 | International Business Machines Corporation | Indicating data health in a DSN memory |
US10372381B2 (en) | 2017-06-05 | 2019-08-06 | International Business Machines Corporation | Implicit leader election in a distributed storage network |
US10361813B2 (en) | 2017-06-16 | 2019-07-23 | International Business Machine Corporation | Using slice routers for improved storage placement determination |
US10534548B2 (en) | 2017-06-20 | 2020-01-14 | International Business Machines Corporation | Validating restricted operations on a client using trusted environments |
US10324855B2 (en) | 2017-06-23 | 2019-06-18 | International Business Machines Corporation | Associating a processing thread and memory section to a memory device |
US10540111B2 (en) | 2017-06-28 | 2020-01-21 | International Business Machines Corporation | Managing data container instances in a dispersed storage network |
US10594790B2 (en) | 2017-06-28 | 2020-03-17 | International Business Machines Corporation | Data compression in a dispersed storage network |
US10509699B2 (en) | 2017-08-07 | 2019-12-17 | International Business Machines Corporation | Zone aware request scheduling and data placement |
US10599502B2 (en) | 2017-08-07 | 2020-03-24 | International Business Machines Corporation | Fault detection and recovery in a distributed storage network |
US10671746B2 (en) | 2017-08-28 | 2020-06-02 | International Business Machines Corporation | Controlling access when processing intents in a dispersed storage network |
US10379942B2 (en) | 2017-09-27 | 2019-08-13 | International Business Machines Corporation | Efficient transfer of objects between containers on the same vault |
US10802713B2 (en) | 2017-09-29 | 2020-10-13 | International Business Machines Corporation | Requester-associated storage entity data |
US10409661B2 (en) | 2017-09-29 | 2019-09-10 | International Business Machines Corporation | Slice metadata for optimized dispersed storage network memory storage strategies |
US10585748B2 (en) | 2017-09-29 | 2020-03-10 | International Business Machines Corporation | Scalable cloud—assigning scores to requesters and treating requests differently based on those scores |
US10540120B2 (en) | 2017-11-14 | 2020-01-21 | International Business Machines Corporation | Contention avoidance on associative commutative updates |
US10423497B2 (en) | 2017-11-28 | 2019-09-24 | International Business Machines Corporation | Mechanism for representing system configuration changes as a series of objects writable to an object storage container |
US10565392B2 (en) | 2017-11-28 | 2020-02-18 | International Business Machines Corporation | Secure and verifiable update operations |
US10785194B2 (en) | 2017-12-07 | 2020-09-22 | International Business Machines Corporation | Processing intents using trusted entities in a dispersed storage network |
US10681135B2 (en) | 2017-12-08 | 2020-06-09 | International Business Machines Corporation | Generating slices from a broadcast message and a recipient identity |
US11412041B2 (en) | 2018-06-25 | 2022-08-09 | International Business Machines Corporation | Automatic intervention of global coordinator |
US10936452B2 (en) | 2018-11-14 | 2021-03-02 | International Business Machines Corporation | Dispersed storage network failover units used to improve local reliability |
US10834196B2 (en) * | 2019-03-21 | 2020-11-10 | International Business Machines Corporation | Segmenting and merging data in a dispersed storage network |
US11593026B2 (en) | 2020-03-06 | 2023-02-28 | International Business Machines Corporation | Zone storage optimization using predictive protocol patterns |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5485474A (en) * | 1988-02-25 | 1996-01-16 | The President And Fellows Of Harvard College | Scheme for information dispersal and reconstruction |
US5644695A (en) * | 1994-01-03 | 1997-07-01 | International Business Machines Corporation | Array combinatorial decoding with multiple error and erasure detection and location using cyclic equivalence testing |
US6088330A (en) * | 1997-09-09 | 2000-07-11 | Bruck; Joshua | Reliable array of distributed computing nodes |
US6370670B1 (en) * | 1997-12-30 | 2002-04-09 | Canon Kabushiki Kaisha | Interlacer, coding device, permutation method, coding method, decoding device and method and systems using them |
US6614366B2 (en) * | 1998-09-23 | 2003-09-02 | Digital Fountain, Inc. | Information additive code generator and decoder for communication systems |
US20050088986A1 (en) * | 2003-10-27 | 2005-04-28 | Feng-Wen Sun | Systems and methods for distributing data |
US7133853B2 (en) * | 2001-06-15 | 2006-11-07 | Qualcomm Incorporated | Methods and apparatus for decoding LDPC codes |
-
2004
- 2004-03-23 US US10/806,488 patent/US7240236B2/en not_active Expired - Fee Related
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5485474A (en) * | 1988-02-25 | 1996-01-16 | The President And Fellows Of Harvard College | Scheme for information dispersal and reconstruction |
US5644695A (en) * | 1994-01-03 | 1997-07-01 | International Business Machines Corporation | Array combinatorial decoding with multiple error and erasure detection and location using cyclic equivalence testing |
US6088330A (en) * | 1997-09-09 | 2000-07-11 | Bruck; Joshua | Reliable array of distributed computing nodes |
US6370670B1 (en) * | 1997-12-30 | 2002-04-09 | Canon Kabushiki Kaisha | Interlacer, coding device, permutation method, coding method, decoding device and method and systems using them |
US6614366B2 (en) * | 1998-09-23 | 2003-09-02 | Digital Fountain, Inc. | Information additive code generator and decoder for communication systems |
US7133853B2 (en) * | 2001-06-15 | 2006-11-07 | Qualcomm Incorporated | Methods and apparatus for decoding LDPC codes |
US20050088986A1 (en) * | 2003-10-27 | 2005-04-28 | Feng-Wen Sun | Systems and methods for distributing data |
Cited By (57)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2006117769A3 (en) * | 2005-05-04 | 2007-03-29 | Nat Univ Ireland | Method and apparatus for generating error-correcting and error-detecting codes using zero-divisors and units in group rings |
WO2006117769A2 (en) * | 2005-05-04 | 2006-11-09 | National University Of Ireland, Galway | Method and apparatus for generating error-correcting and error-detecting codes using zero-divisors and units in group rings |
US20090089744A1 (en) * | 2005-05-04 | 2009-04-02 | National University Of Ireland | Method and apparatus for generating error-correcting and error-detecting codes using zero-divisors and units in group rings |
US7814078B1 (en) * | 2005-06-20 | 2010-10-12 | Hewlett-Packard Development Company, L.P. | Identification of files with similar content |
US20070233828A1 (en) * | 2006-03-31 | 2007-10-04 | Jeremy Gilbert | Methods and systems for providing data storage and retrieval |
CN101064840B (en) * | 2006-04-26 | 2010-11-03 | 日电(中国)有限公司 | Snake-shaped matrix coder/decoder and visual sign coding and presenting system |
US20070294310A1 (en) * | 2006-06-06 | 2007-12-20 | Hitachi, Ltd. | Method and apparatus for storing and recovering fixed content |
US20080046764A1 (en) * | 2006-08-04 | 2008-02-21 | Lsi Logic Corporation | Data Shredding RAID Mode |
US8806227B2 (en) * | 2006-08-04 | 2014-08-12 | Lsi Corporation | Data shredding RAID mode |
US20080068899A1 (en) * | 2006-09-15 | 2008-03-20 | Fujitsu Limited | Storage management process, storage management apparatus, and computer-readable medium storing storage management program |
US7483318B2 (en) * | 2006-09-15 | 2009-01-27 | Fujitsu Limited | Storage management process, storage management apparatus, and computer-readable medium storing storage management program |
US8352692B1 (en) * | 2007-03-30 | 2013-01-08 | Symantec Corporation | Utilizing peer-to-peer services with single instance storage techniques |
EP2288991A2 (en) * | 2008-05-05 | 2011-03-02 | Amplidata | Method of storing a data set in a distributed storage system, distributed storage system and computer program product for use with said method |
EP2288991B1 (en) * | 2008-05-05 | 2018-01-10 | Amplidata | Method of storing a data set in a distributed storage system, distributed storage system and computer program product for use with said method |
US20100199123A1 (en) * | 2009-02-03 | 2010-08-05 | Bittorrent, Inc. | Distributed Storage of Recoverable Data |
EP2394220A1 (en) * | 2009-02-03 | 2011-12-14 | Bittorrent, Inc. | Distributed storage of recoverable data |
EP2394220A4 (en) * | 2009-02-03 | 2013-02-20 | Bittorrent Inc | Distributed storage of recoverable data |
US8522073B2 (en) | 2009-02-03 | 2013-08-27 | Bittorrent, Inc. | Distributed storage of recoverable data |
US9454444B1 (en) | 2009-03-19 | 2016-09-27 | Veritas Technologies Llc | Using location tracking of cluster nodes to avoid single points of failure |
US20100266131A1 (en) * | 2009-04-20 | 2010-10-21 | Bart Cilfone | Natural action heuristics for management of network devices |
US8819781B2 (en) * | 2009-04-20 | 2014-08-26 | Cleversafe, Inc. | Management of network devices within a dispersed data storage network |
US8311964B1 (en) | 2009-11-12 | 2012-11-13 | Symantec Corporation | Progressive sampling for deduplication indexing |
US8458515B1 (en) | 2009-11-16 | 2013-06-04 | Symantec Corporation | Raid5 recovery in a high availability object based file system |
US8473463B1 (en) | 2010-03-02 | 2013-06-25 | Symantec Corporation | Method of avoiding duplicate backups in a computing system |
US8370315B1 (en) | 2010-05-28 | 2013-02-05 | Symantec Corporation | System and method for high performance deduplication indexing |
US8983952B1 (en) | 2010-07-29 | 2015-03-17 | Symantec Corporation | System and method for partitioning backup data streams in a deduplication based storage system |
US8756197B1 (en) | 2010-08-13 | 2014-06-17 | Symantec Corporation | Generating data set views for backup restoration |
US8291170B1 (en) | 2010-08-19 | 2012-10-16 | Symantec Corporation | System and method for event driven backup data storage |
US8392376B2 (en) | 2010-09-03 | 2013-03-05 | Symantec Corporation | System and method for scalable reference management in a deduplication based storage system |
US8782011B2 (en) | 2010-09-03 | 2014-07-15 | Symantec Corporation | System and method for scalable reference management in a deduplication based storage system |
US8396841B1 (en) | 2010-11-30 | 2013-03-12 | Symantec Corporation | Method and system of multi-level and multi-mode cloud-based deduplication |
US8495323B1 (en) | 2010-12-07 | 2013-07-23 | Symantec Corporation | Method and system of providing exclusive and secure access to virtual storage objects in a virtual machine cluster |
US8392384B1 (en) | 2010-12-10 | 2013-03-05 | Symantec Corporation | Method and system of deduplication-based fingerprint index caching |
US8726127B2 (en) | 2011-02-01 | 2014-05-13 | Cleversafe, Inc. | Utilizing a dispersed storage network access token module to access a dispersed storage network memory |
US10402268B2 (en) | 2011-02-01 | 2019-09-03 | Pure Storage, Inc. | Utilizing a dispersed storage network access token module to acquire digital content from a digital content provider |
US9081714B2 (en) | 2011-02-01 | 2015-07-14 | Cleversafe, Inc. | Utilizing a dispersed storage network access token module to store data in a dispersed storage network memory |
US9081715B2 (en) | 2011-02-01 | 2015-07-14 | Cleversafe, Inc. | Utilizing a dispersed storage network access token module to retrieve data from a dispersed storage network memory |
WO2012106329A1 (en) * | 2011-02-01 | 2012-08-09 | Cleversafe, Inc. | Utilizing a dispersed storage network access token module to store data in a dispersed storage network memory |
US8677214B2 (en) * | 2011-10-04 | 2014-03-18 | Cleversafe, Inc. | Encoding data utilizing a zero information gain function |
US9069786B2 (en) | 2011-10-14 | 2015-06-30 | Pure Storage, Inc. | Method for maintaining multiple fingerprint tables in a deduplicating storage system |
US11341117B2 (en) | 2011-10-14 | 2022-05-24 | Pure Storage, Inc. | Deduplication table management |
US10540343B2 (en) | 2011-10-14 | 2020-01-21 | Pure Storage, Inc. | Data object attribute based event detection in a storage system |
US10061798B2 (en) | 2011-10-14 | 2018-08-28 | Pure Storage, Inc. | Method for maintaining multiple fingerprint tables in a deduplicating storage system |
CN103023968B (en) * | 2012-11-15 | 2015-12-23 | 中科院成都信息技术有限公司 | A kind of network distributed file stores and read method |
CN103023968A (en) * | 2012-11-15 | 2013-04-03 | 中科院成都信息技术有限公司 | Network distributed storage and reading method for file |
US10275397B2 (en) | 2013-02-22 | 2019-04-30 | Veritas Technologies Llc | Deduplication storage system with efficient reference updating and space reclamation |
US9575680B1 (en) | 2014-08-22 | 2017-02-21 | Veritas Technologies Llc | Deduplication rehydration |
US10423495B1 (en) | 2014-09-08 | 2019-09-24 | Veritas Technologies Llc | Deduplication grouping |
US20170272209A1 (en) * | 2016-03-15 | 2017-09-21 | Cloud Crowding Corp. | Distributed Storage System Data Management And Security |
US10608784B2 (en) | 2016-03-15 | 2020-03-31 | ClineHair Commercial Endeavors | Distributed storage system data management and security |
US10735137B2 (en) * | 2016-03-15 | 2020-08-04 | ClineHair Commercial Endeavors | Distributed storage system data management and security |
US10931402B2 (en) | 2016-03-15 | 2021-02-23 | Cloud Storage, Inc. | Distributed storage system data management and security |
US11777646B2 (en) | 2016-03-15 | 2023-10-03 | Cloud Storage, Inc. | Distributed storage system data management and security |
US20190173948A1 (en) * | 2017-03-06 | 2019-06-06 | At&T Intellectual Property I, L.P. | Reliable data storage for decentralized computer systems |
US11394777B2 (en) * | 2017-03-06 | 2022-07-19 | At&T Intellectual Property I, L.P. | Reliable data storage for decentralized computer systems |
US11182247B2 (en) | 2019-01-29 | 2021-11-23 | Cloud Storage, Inc. | Encoding and storage node repairing method for minimum storage regenerating codes for distributed storage systems |
US11349500B2 (en) * | 2020-01-15 | 2022-05-31 | EMC IP Holding Company LLC | Data recovery in a geographically diverse storage system employing erasure coding technology and data convolution technology |
Also Published As
Publication number | Publication date |
---|---|
US7240236B2 (en) | 2007-07-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7240236B2 (en) | Fixed content distributed data storage using permutation ring encoding | |
US11233643B1 (en) | Distributed data storage system data decoding and decryption | |
US11182247B2 (en) | Encoding and storage node repairing method for minimum storage regenerating codes for distributed storage systems | |
US8464096B2 (en) | Method and apparatus for rebuilding data in a dispersed data storage network | |
US8744071B2 (en) | Dispersed data storage system data encryption and encoding | |
US8171102B2 (en) | Smart access to a dispersed data storage network | |
Xin et al. | Reliability mechanisms for very large storage systems | |
US8392805B2 (en) | Non-MDS erasure codes for storage systems | |
US20100218037A1 (en) | Matrix-based Error Correction and Erasure Code Methods and Apparatus and Applications Thereof | |
US20090094250A1 (en) | Ensuring data integrity on a dispersed storage grid | |
US20060242542A1 (en) | Correcting multiple block data loss in a storage array using a combination of a single diagonal parity group and multiple row parity groups | |
US8928503B2 (en) | Data encoding methods, data decoding methods, data reconstruction methods, data encoding devices, data decoding devices, and data reconstruction devices | |
US20200319973A1 (en) | Layered error correction encoding for large scale distributed object storage system | |
Hou et al. | A new construction and an efficient decoding method for Rabin-like codes | |
Li et al. | Beyond the MDS bound in distributed cloud storage | |
TW202001920A (en) | Method and apparatus for improved data recovery in data storage systems | |
Fu et al. | A scheme of data confidentiality and fault-tolerance in cloud storage | |
US20210055993A1 (en) | Methods and Apparatuses for Robust Data Partition And Recovery | |
Sengupta et al. | An efficient secure distributed cloud storage for append-only data | |
US20220114053A1 (en) | Reconstructing Data Segments in a Storage Network and Methods for Use Therewith | |
Hou et al. | Cauchy MDS array codes with efficient decoding method | |
Oggier et al. | Self-repairing codes: local repairability for cheap and fast maintenance of erasure coded data | |
Wang et al. | Rotary-code: Efficient MDS array codes for RAID-6 disk arrays | |
Vins et al. | A survey on regenerating codes | |
Bhuvaneshwari et al. | LDPC Codes for Distributed Storage systems |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ARCHIVAS, INC., MASSACHUSETTS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CUTTS, SHAUN;RODRIGUEZ, ANDRES;REEL/FRAME:019357/0452;SIGNING DATES FROM 20040318 TO 20040322 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
REMI | Maintenance fee reminder mailed | ||
FPAY | Fee payment |
Year of fee payment: 4 |
|
SULP | Surcharge for late payment | ||
AS | Assignment |
Owner name: HITACHI DATA SYSTEMS CORPORATION, CALIFORNIA Free format text: MERGER;ASSIGNOR:ARCHIVAS, INC.;REEL/FRAME:030345/0287 Effective date: 20090331 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
AS | Assignment |
Owner name: HITACHI VANTARA CORPORATION, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:HITACHI DATA SYSTEMS CORPORATION;REEL/FRAME:045369/0785 Effective date: 20170926 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20190703 |