Abstract: In order to avoid writing duplicates of blocks of data into a storage platform, any virtual disk within the storage platform may have a de-duplication feature enabled. Or, all virtual disks have this feature enabled. For virtual disks with de-duplication enabled, a unique message digest is calculated for every block of data written to that virtual disk. Upon a write, these message digests are consulted in order to determine if a particular block of data has already been written, if so, it is not written again, and if not, it is written. All de-duplication virtual disks are written to a single system virtual disk within the storage platform. De-duplication occurs over the entire storage platform and over all its virtual disks because all message digests are consulted before a write is performed for any virtual disk. A read for a de-duplication virtual desk reads from the system virtual disk.

Abstract: In order to avoid writing duplicates of blocks of data into a storage platform, any virtual disk within the storage platform may have a de-duplication feature enabled. Or, all virtual disks have this feature enabled. For virtual disks with de-duplication enabled, a unique message digest is calculated for every block of data written to that virtual disk. Upon a write, these message digests are consulted in order to determine if a particular block of data has already been written, if so, it is not written again, and if not, it is written. All de-duplication virtual disks are written to a single system virtual disk within the storage platform. De-duplication occurs over the entire storage platform and over all its virtual disks because all message digests are consulted before a write is performed for any virtual disk. A read for a de-duplication virtual desk reads from the system virtual disk.

Abstract: Encryption of data occurs before it is written to the storage platform; decryption occurs after it is read from the storage platform on a computer separate from the storage platform. By encrypting data before it travels over a wide-area network to a storage platform (and by only decrypting that data once it has arrived at an enterprise from the storage platform), we address data security over the network. Application data is encrypted at the virtual disk level before it leaves a controller virtual machine, and is only decrypted at that controller virtual machine after being received from the storage platform. Encryption and decryption of data is compatible with other services of the storage system such as de-duplication. Any number of key management services can be used in a transparent manner.

Abstract: A computer receives a write request including an offset within a virtual disk. The computer writes the data block to a remote platform and calculates a hash value of the data. If the hash value does not exist in a first table of a block cache of the computer, the computer adds a pair to the first table: hash value/block cache data offset. Next, the computer adds a pair in a second table of the block cache: virtual disk offset of the data/hash value. A read request uses these tables to find the data in the cache without accessing the storage platform. The read consults the second table to find the hash value corresponding to the virtual disk offset of block. The hash value is used as a key into the first table to find the block cache data offset of the data; the data is read from the block cache at that offset.

Abstract: A client machine writes to and reads from a virtual disk on a remote storage platform. Metadata is generated and stored in replicas on different metadata nodes of the storage platform. A modified log-structured merge tree is used to store and compact string-sorted tables of metadata. During file storage and compaction, a consistent file identification scheme is used across all metadata nodes. A fingerprint file is calculated for each SST (metadata) file on disk that includes hash values corresponding to regions of the SST file. To synchronize, the fingerprint files of two SST files are compared, and if any hash values are missing from a fingerprint file then the key-value-timestamp triplets corresponding to these missing hash values are sent to the SST file that is missing them. The SST file is compacted with the missing triplets to create a new version of the SST file. The synchronization is bi-directional.

Abstract: A client machine writes to and reads from a virtual disk on a remote storage platform. Metadata is generated and stored in replicas on different metadata nodes of the storage platform. A modified log-structured merge tree is used to store and compact string-sorted tables of metadata. During file storage and compaction, a consistent file identification scheme is used across all metadata nodes. A fingerprint file is calculated for each SST (metadata) file on disk that includes hash values corresponding to regions of the SST file. To synchronize, the fingerprint files of two SST files are compared, and if any hash values are missing from a fingerprint file then the key-value-timestamp triples corresponding to these missing hash values are sent to the SST file that is missing them. The SST file is compacted with the missing triples to create a new version of the SST file. The synchronization is bi-directional.

Abstract: A client machine writes to a virtual disk on a remote storage platform. Metadata is generated and stored in replicas on different nodes of the storage platform. A modified log-structured merge tree is used to store and compact string-sorted tables of metadata. During file storage and compaction, a consistent file identification scheme is used across all metadata nodes. A fingerprint file is calculated for each SST (metadata) file on disk that includes hash values corresponding to regions of the SST file. To synchronize, the fingerprint files of two SST files are compared, and if any hash values are missing from a fingerprint file then the key-value-timestamp triplets corresponding to these missing hash values are sent to the SST file that is missing them. The SST file is compacted with the missing triplets to create a new version of the SST file. The synchronization is bi-directional as between distinct computer nodes.

Abstract: A computer receives a write request including an offset within a virtual disk. The computer writes the data block to a remote platform and calculates a hash value of the data. If the hash value does not exist in a first table of a block cache of the computer, the computer adds a pair to the first table: hash value/block cache data offset. Next, the computer adds a pair in a second table of the block cache: virtual disk offset of the data/hash value. A read request uses these tables to find the data in the cache without accessing the platform. The read consults the second table to find the hash value corresponding to the virtual disk offset of the block. The hash value is a key into the first table to find the block cache data offset of the data; the data is read from the cache.

Abstract: A client machine writes to and reads from a virtual disk on a remote storage platform using a storage protocol. Metadata is generated and is stored in replicas on different metadata nodes of the storage platform. A modified log-structured merge tree is used to store and compact string-sorted tables of metadata. During file storage and compaction, a consistent file identification scheme is used across all metadata nodes. A fingerprint file is calculated for each SST file on disk that includes hash values corresponding to regions of the SST file. To synchronize, the fingerprint files of two SST files are compared, and if any hash values are missing from a fingerprint file then the key-value-timestamp triples corresponding to these missing hash values are sent to the SST file that is missing them in the SST file is compacted with the missing triples to create a new version of the SST file. The other fingerprint file is then analyzed the same way.

Abstract: An application within a virtual machine is an iSCSI Initiator and is allowed to use as an iSCSI Target another virtual machine within the same hypervisor in order to make a persistent reservation for a virtual disk within a remotely-located storage platform. Any number of virtual machines within different hypervisors, and perhaps on different computers, use a local controller virtual machine to make a persistent reservation for the same virtual disk. The registration list and the current reservation holder data for an iSCSI persistent reservation for a particular virtual disk are held on a storage node of the storage platform rather than within a single virtual machine of a remote computer. A metadata module on the storage platform handles the incoming requests. A coordinator module within the storage platform uses a lock mechanism to guarantee that the reserve, release, preempt and clear commands are handled properly.

Abstract: An application within a virtual machine is an iSCSI Initiator and is allowed to use as an iSCSI Target another virtual machine within the same hypervisor in order to make a persistent reservation for a virtual disk within a remotely-located storage platform. Any number of virtual machines within different hypervisors, and perhaps on different computers, use a local controller virtual machine to make a persistent reservation for the same virtual disk. The registration list and the current reservation holder data for an iSCSI persistent reservation for a particular virtual disk are held on a storage node of the storage platform rather than within a single virtual machine of a remote computer. A metadata module on the storage platform handles the incoming requests. A coordinator module within the storage platform uses a lock mechanism to guarantee that the reserve, release, preempt and clear commands are handled properly.

Abstract: An application within a virtual machine is an iSCSI Initiator and is allowed to use as an iSCSI Target another virtual machine within the same hypervisor in order to make a persistent reservation for a virtual disk within a remotely-located storage platform. Any number of virtual machines within different hypervisors, and perhaps on different computers, use a local controller virtual machine to make a persistent reservation for the same virtual disk. The registration list and the current reservation holder data for an iSCSI persistent reservation for a particular virtual disk are held on a storage node of the storage platform rather than within a single virtual machine of a remote computer. A metadata module on the storage platform handles the incoming requests. A coordinator module within the storage platform uses a lock mechanism to guarantee that the reserve, release, preempt and clear commands are handled properly.

Abstract: An application within a virtual machine is an iSCSI Initiator and is allowed to use as an iSCSI Target another virtual machine within the same hypervisor in order to make a persistent reservation for a virtual disk within a remotely-located storage platform. Any number of virtual machines within different hypervisors, and perhaps on different computers, use a local controller virtual machine to make a persistent reservation for the same virtual disk. The registration list and the current reservation holder data for an iSCSI persistent reservation for a particular virtual disk are held on a storage node of the storage platform rather than within a single virtual machine of a remote computer. A metadata module on the storage platform handles the incoming requests. A coordinator module within the storage platform uses a lock mechanism to guarantee that the reserve, release, preempt and clear commands are handled properly.

Abstract: An administrator provisions a virtual disk in a remote storage platform and defines policies for that virtual disk. A virtual machine writes to and reads from the storage platform using any storage protocol. Virtual disk data within a failed storage pool is migrated to different storage pools while still respecting the policies of each virtual disk. Snapshot and revert commands are given for a virtual disk at a particular point in time and overhead is minimal. A virtual disk is cloned utilizing snapshot information and no data need be copied. Any number of Zookeeper clusters are executing in a coordinated fashion within the storage platform, thus increasing overall throughput. A timestamp is generated that guarantees a monotonically increasing counter, even upon a crash of a virtual machine. Any virtual disk has a “hybrid cloud aware” policy in which one replica of the virtual disk is stored in a public cloud.

Abstract: An administrator provisions a virtual disk in a remote storage platform and defines policies for that virtual disk. A virtual machine writes to and reads from the storage platform using any storage protocol. Virtual disk data within a failed storage pool is migrated to different storage pools while still respecting the policies of each virtual disk. Snapshot and revert commands are given for a virtual disk at a particular point in time and overhead is minimal. A virtual disk is cloned utilizing snapshot information and no data need be copied. Any number of Zookeeper clusters are executing in a coordinated fashion within the storage platform, thus increasing overall throughput. A timestamp is generated that guarantees a monotonically increasing counter, even upon a crash of a virtual machine. Any virtual disk has a “hybrid cloud aware” policy in which one replica of the virtual disk is stored in a public cloud.

Abstract: An administrator provisions a virtual disk in a remote storage platform and defines policies for that virtual disk. A virtual machine writes to and reads from the storage platform using any storage protocol. Virtual disk data within a failed storage pool is migrated to different storage pools while still respecting the policies of each virtual disk. Snapshot and revert commands are given for a virtual disk at a particular point in time and overhead is minimal. A virtual disk is cloned utilizing snapshot information and no data need be copied. Any number of Zookeeper clusters are executing in a coordinated fashion within the storage platform, thus increasing overall throughput. A timestamp is generated that guarantees a monotonically increasing counter, even upon a crash of a virtual machine. Any virtual disk has a “hybrid cloud aware” policy in which one replica of the virtual disk is stored in a public cloud.

Abstract: An application within a virtual machine is an iSCSI Initiator and is allowed to use as an iSCSI Target another virtual machine within the same hypervisor in order to make a persistent reservation for a virtual disk within a remotely-located storage platform. Any number of virtual machines within different hypervisors, and perhaps on different computers, use a local controller virtual machine to make a persistent reservation for the same virtual disk. The registration list and the current reservation holder data for an iSCSI persistent reservation for a particular virtual disk are held on a storage node of the storage platform rather than within a single virtual machine of a remote computer. A metadata module on the storage platform handles the incoming requests. A coordinator module within the storage platform uses a lock mechanism to guarantee that the reserve, release, preempt and clear commands are handled properly.

Abstract: A computer receives a write request including an offset within a virtual disk. The computer writes the data block to a remote platform and calculates a hash value of the data. If the hash value does not exist in a first table of a block cache of the computer, the computer adds a pair to the first table: hash value/block cache data offset. Next, the computer adds a pair in a second table of the block cache: virtual disk offset of the data/hash value. A read request uses these tables to find the data in the cache without accessing the storage platform. The read consults the second table to find the hash value corresponding to the virtual disk offset of block. The hash value is used as a key into the first table to find the block cache data offset of the data; the data is read from the block cache at that offset.

Abstract: In order to avoid writing duplicates of blocks of data into a storage platform, any virtual disk within the storage platform may have a de-duplication feature enabled. Or, all virtual disks have this feature enabled. For virtual disks with de-duplication enabled, a unique message digest is calculated for every block of data written to that virtual disk. Upon a write, these message digests are consulted in order to determine if a particular block of data has already been written, if so, it is not written again, and if not, it is written. All de-duplication virtual disks are written to a single system virtual disk within the storage platform. De-duplication occurs over the entire storage platform and over all its virtual disks because all message digests are consulted before a write is performed for any virtual disk. A read for a de-duplication virtual desk reads from the system virtual disk.

Abstract: An administrator provisions a virtual disk in a remote storage platform and defines policies for that virtual disk. A virtual machine writes to and reads from the storage platform using any storage protocol. Virtual disk data within a failed storage pool is migrated to different storage pools while still respecting the policies of each virtual disk. Snapshot and revert commands are given for a virtual disk at a particular point in time and overhead is minimal. A virtual disk is cloned utilizing snapshot information and no data need be copied. Any number of Zookeeper clusters are executing in a coordinated fashion within the storage platform, thus increasing overall throughput. A timestamp is generated that guarantees a monotonically increasing counter, even upon a crash of a virtual machine. Any virtual disk has a “hybrid cloud aware” policy in which one replica of the virtual disk is stored in a public cloud.