Abstract: Disclosed are techniques for compressing data in a data storage system comprising searching a cluster of nearest neighbors, wherein the cluster has been created using a locality sensitive hashing algorithm, to determine if a data block can be compressed. In alternate embodiments, nearest neighbor clusters can be formed using unsupervised learning. Additionally, nearest neighbors can also be formed in alternate embodiments using one or more of the following algorithms: a k-means clustering algorithm, a k-medoids clustering algorithm, a mean shift algorithm, a generalized method of moment (GMM) algorithm, or a density based spatial clustering of applications with noise (DBSCAN) algorithm.

Abstract: One or more aspects of the present disclosure relate to controlling threads across several processors. For example, at least one input/output (IO) workflow message can be stored in a storage array's hardware queue. Additionally, an IO workflow message can be read from the hardware queue. Further, a local thread wake-up or an interrupt-wakeup operation can be performed based on a target of the IO workflow message.

Abstract: A host IO devices directly implements host read operations on both local memory, and on peer memory via a PCIe non-transparent bridge. When a host read operation is received by a host IO device from a host, the host IO device uses an API to obtain the physical address of the requested data on the peer memory, and generates a PCIe Transaction Layer Packet (TLP) addressed to the address in the peer memory. The TLP addressed to an address in the peer memory is passed over the NTB to the peer compute node to retrieve the data stored in the addressed slot of peer memory. The requested data is returned to the host IO device over the NTB, stored in a buffer, and read out to the host to directy respond to the host read operation.

Abstract: Techniques for data deduplication may include: receiving write operations that write first data; partitioning the first data into a plurality of data portions; generating, using a first hash function, a plurality of data deduplication hash values for the plurality of data portions, wherein a first data deduplication hash value of the plurality of data deduplication hash values is produced by said generating for a first of the plurality of data portions; performing first processing using a Bloom filter to determine whether the first data deduplication hash value has a corresponding first entry in a data store of deduplication hash values; and responsive to the first processing determining the first data deduplication hash value does not have the corresponding first entry in the data store of deduplication hash values, performing second processing, said second processing including adding the corresponding first entry in the data store of deduplication hash values.

Abstract: Disclosed are techniques for data deduplication, which include methods, systems, or computer products for reducing data redundancy in a data storage system comprising searching a cluster of nearest neighbors, wherein the cluster has been created using a locality sensitive hashing algorithm, to determine if a data block has been stored in the data storage system prior to writing the data block. In alternate embodiments, the nearest neighbor clusters could be created using one or more of the following algorithms: k-means clustering algorithm, a k-medoids clustering algorithm, a mean shift algorithm, a generalized method of moment (GMM) algorithm, or a density based spatial clustering of applications with noise (DBSCAN) algorithm.

Abstract: First and second virtual storage controllers represent first and second physical storage controllers that provide access to different physical storage devices that back first and second virtual disks. One or more computing nodes host a virtual storage array that includes paired virtual storage directors. Each virtual storage director has access to both the first and second virtual storage controllers. Thus, the virtual disks may be presented to the storage directors as dual-ported disks even if the physical storage devices are single-ported. The virtual storage directors and virtual storage controllers may each be hosted by a separate virtual machine or instantiated in a separate container.

Abstract: A storage array is implemented as a virtual machine on a server. The virtual storage array includes a virtual engine with virtual directors and virtualized processor and memory resources. Embedded storage software converts Input/Output requests into Input/Output requests that can be serviced by the virtual machine. A virtual backend driver enables the virtual storage directors to access a set of virtual drives. A virtualized shared memory driver provides emulated input/output, XOR, direct memory access and copy services for send and receive threads.

Abstract: Disclosed are techniques for compressing data in a data storage system comprising searching a cluster of nearest neighbors, wherein the cluster has been created using a locality sensitive hashing algorithm, to determine if a data block can be compressed. In alternate embodiments, nearest neighbor clusters can be formed using unsupervised learning. Additionally, nearest neighbors can also be formed in alternate embodiments using one or more of the following algorithms: a k-means clustering algorithm, a k-medoids clustering algorithm, a mean shift algorithm, a generalized method of moment (GMM) algorithm, or a density based spatial clustering of applications with noise (DBSCAN) algorithm.

Abstract: Disclosed are techniques for data deduplication, which include methods, systems, or computer products for reducing data redundancy in a data storage system comprising searching a cluster of nearest neighbors, wherein the cluster has been created using a locality sensitive hashing algorithm, to determine if a data block has been stored in the data storage system prior to writing the data block. In alternate embodiments, the nearest neighbor clusters could be created using one or more of the following algorithms: k-means clustering algorithm, a k-medoids clustering algorithm, a mean shift algorithm, a generalized method of moment (GMM) algorithm, or a density based spatial clustering of applications with noise (DBSCAN) algorithm.

Abstract: Techniques for data deduplication may include: receiving write operations that write first data; partitioning the first data into a plurality of data portions; generating, using a first hash function, a plurality of data deduplication hash values for the plurality of data portions, wherein a first data deduplication hash value of the plurality of data deduplication hash values is produced by said generating for a first of the plurality of data portions; performing first processing using a Bloom filter to determine whether the first data deduplication hash value has a corresponding first entry in a data store of deduplication hash values; and responsive to the first processing determining the first data deduplication hash value does not have the corresponding first entry in the data store of deduplication hash values, performing second processing, said second processing including adding the corresponding first entry in the data store of deduplication hash values.

Abstract: Techniques for data deduplication may include: receiving write operations that write first data; partitioning the first data into a plurality of data portions; generating, using a first hash function, a plurality of data deduplication hash values for the plurality of data portions, wherein a first data deduplication hash value of the plurality of data deduplication hash values is produced by said generating for a first of the plurality of data portions; performing first processing using a Bloom filter to determine whether the first data deduplication hash value has a corresponding first entry in a data store of deduplication hash values; and responsive to the first processing determining the first data deduplication hash value does not have the corresponding first entry in the data store of deduplication hash values, performing second processing, said second processing including adding the corresponding first entry in the data store of deduplication hash values.

Abstract: Embodiments are described for a executing a processing job using one or more nodes of a storage area network using computing resources on the SAN that are predicted to be idle. A predictive model is generated by monitoring idle states of resources of nodes of the SAN and using machine learning to build the predictive model. A scheduler executes jobs on one or more nodes of the SAN with sufficient predicted idle resources to process the job, in accordance with resource requirements and job attributes in a manifest of the job. If a job cannot be completed during a window of time that the necessary resources are predicted to be idle, or if one or more resources become unavailable, the job can be paused and resumed, migrated to another node, or restarted at a later time when the required resources to complete the job are predicted to be idle.

Abstract: Software emulations of physical devices allow protocol neutral communications between heterogeneous operating systems. Within a computing node of a storage cluster, a logical communications cut-through device connects a guest operating system to a front end adapter and enables the guest operating system to perform operations on an allocated portion of computing node memory without hypervisor intervention. The cut-through device may implement an industry standard memory-mapped interface, thereby enabling a guest operating system to discover the cut-through device and obtain access to memory that is shared with the hypervisor and/or another guest OS. Moreover, if guest OSs and/or the hypervisor share any communications protocol then the channel may be utilized at memory speeds. This may be advantageous in symmetric multiprocessing environments in which multiple guest OSs can have core affinities.

Abstract: The system and methods disclosed herein relate to an improvement in automated data tiering technology. In these embodiments, users are able to weigh the importance of read, write, and pre-fetch operations in terms of tier placement within the data storage system. Data relocation evaluations are performed by an off-load engine, e.g., a graphics processing unit, which utilizes parallel processing on data arrays/vectors/extents. The data relocation candidates are identified via calculating a mobility score on an extent-by-extent basis. Data are then relocated within the various tiers of the data storage system.

Abstract: A storage node such as a storage array or storage server has storage controllers with heterogeneous parallel processors. The CPUs may be used to perform read and write operations. GPUs are used to produce transformed data from raw data. The GPUs may be used to perform various analytic calculations. The GPUs may be interconnected via dedicated communication links.

Abstract: An interface mitigates the need for inter-application locks in a storage system with a HPP. The interface includes a virtual device driver that can be accessed by both emulation threads and guest OS Container threads and separate memory pages that are pinned for use as buffer spaces. The pinned buffer memory pages are shared by CPU cores and by individual virtual HPP client drivers. The interface enables virtualization of GPUs and HPPs.

Abstract: Data is compressed based on non-identical similarity between a first data set and a second data set. A representation of the differences is used to represent one of the data sets. For example, a probabilistically unique value may be generated as a new block label. Probabilistic comparison of the new block label with a plurality of training labels associated with training blocks produces a plurality of training labels that are potentially similar to the new block label. The Hamming distance between each potentially similar training label and the new block label is determined to select the training label with the smallest calculated Hamming distance from the new block label. A bitmap of differences between the new block and the training block associated with the selected training label is compressed and stored as a compressed representation of the new block.

Abstract: A driver enables a first virtual storage director running in a container maintained by a hypervisor to achieve direct memory access to memory of a second virtual storage director running in a different container. An address space is made available to the first virtual storage director. A first portion of the address space is associated with memory allocated to the first virtual storage director by the container. A second portion of the address space is mapped to memory allocated to the second virtual storage director.