Abstract: One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.

Abstract: A write request is received to write a data block having a logical block address to a nonvolatile storage device. The method includes writing a value of the data block to the nonvolatile storage device. The writing includes locating a position in a tree-based data structure that includes first and second nodes. The first node is configured to store a first set of data blocks having logical block addresses in a first numerical range, and the second node is configured to store a second set of data blocks having logical block addresses in a second numerical range. The position is located in the first node or the second node depending on the value of the logical block address. The writing includes storing the value of the data block in the position in the tree-based data structure.

Abstract: Systems and methods for efficiently using solid-state devices are provided. Some embodiments provide for a data processing system that uses a non-volatile solid state device as a circular log, with the goal of aligning data access patterns to the underlying, hidden device implementation, in order to maximize performance. In addition, metadata can be interspersed with data in order to align data access patterns to the underlying device implementation. Multiple input/output (I/O) buffers can also be used to pipeline insertions of metadata and data into a linear log. The observed queuing behavior of the multiple I/O buffers can be used to determine when the utilization of the storage device is approaching saturation (e.g., in order to predict excessively-long response times). Then, the I/O load on the storage device may be shed when utilization approaches saturation. As a result, the overall response time of the system is improved.

Abstract: Systems and methods for efficiently using solid-state devices are provided. Some embodiments provide for a data processing system that uses a non-volatile solid-state device as a circular log, with the goal of aligning data access patterns to the underlying, hidden device implementation, in order to maximize performance. In addition, metadata can be interspersed with data in order to align data access patterns to the underlying device implementation. Multiple input/output (I/O) buffers can also be used to pipeline insertions of metadata and data into a linear log. The observed queuing behavior of the multiple I/O buffers can be used to determine when the utilization of the storage device is approaching saturation (e.g., in order to predict excessively-long response times). Then, the I/O load on the storage device may be shed when utilization approaches saturation. As a result, the overall response time of the system is improved.

Abstract: A write request is received to write a data block having a logical block address to a nonvolatile storage device. The method includes writing a value of the data block to the nonvolatile storage device. The writing includes locating a position in a tree-based data structure that includes first and second nodes. The first node is configured to store a first set of data blocks having logical block addresses in a first numerical range, and the second node is configured to store a second set of data blocks having logical block addresses in a second numerical range. The position is located in the first node or the second node depending on the value of the logical block address. The writing includes storing the value of the data block in the position in the tree-based data structure.

Abstract: One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.

Abstract: One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.

Abstract: One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.

Abstract: Technology is disclosed for improving the storage efficiency and communication efficiency for a storage client device by maximizing the cache hit rate and minimizing data requests to the storage server. The storage server provides a duplication list to the storage client device. The duplication list contains references (e.g. storage addresses) to data blocks that contain duplicate data content. The storage client uses the duplication list to improve the cache hit rate. The duplication list is pruned to contain references to data blocks relevant to the storage client device. The storage server can prune the duplication list based on a working set of storage objects for a client. Alternatively, the storage server can prune the duplication list based on content characteristics, e.g. duplication degree and access frequency. Duplicate blocks to which the client does not have access can be excluded from the duplication list.

Abstract: A system and method for creating and managing a space-efficient, durable key-value map is disclosed. A key management engine initializes a key-value map by associating a plurality of keys with a first slot of the slots in the key-value map. A first key-value pair is then assigned to the first slot of the key-value map. The key management engine subsequently receives an indication that the first key is to be invalidated; and responsively reuses the first slot of the key-value map by assigning a second key-value pair to the first slot. The first key is then recycled in response to the first key becoming valid. The first and second key-value pairs include respective first and second values and respective first and second keys of the plurality of keys associated with the first slot. The first key becomes valid when it is no longer referenced by a data buffer.

Abstract: One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.

Abstract: One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.

Abstract: Systems and methods for efficiently using solid-state devices are provided. Some embodiments provide for a data processing system that uses a non-volatile solid-state device as a circular log, with the goal of aligning data access patterns to the underlying, hidden device implementation, in order to maximize performance. In addition, metadata can be interspersed with data in order to align data access patterns to the underlying device implementation. Multiple input/output (I/O) buffers can also be used to pipeline insertions of metadata and data into a linear log. The observed queuing behavior of the multiple I/O buffers can be used to determine when the utilization of the storage device is approaching saturation (e.g., in order to predict excessively-long response times). Then, the I/O load on the storage device may be shed when utilization approaches saturation. As a result, the overall response time of the system is improved.

Abstract: Automatic network configuration of cluster paths to access data within a cluster is described. Each node of the cluster presents to clients an image of the cluster as a single system. Each node also stores information regarding what node has particular data, as well as information about network addresses and corresponding network capabilities for the various network addresses for each node. When a node receives a request for data, the node determines a node where the request data can be accessed, determines network addresses and network capabilities for each network address of the node, and selects a path to access the node based on the determined information. The receiving node may select the path based on network type and capability, as well as load and performance of the network.

Abstract: A persistent cache is implemented in a flash memory that includes a journal section that stores metadata and a low frequency section and a high frequency section that store data entries. Writing new metadata to the persistent cache includes sequentially advancing to a next sector containing an invalid metadata entry, saving a working copy of the sector in RAM, writing metadata corresponding to one or more new data entries in the working copy, and overwriting the sector in the flash memory containing the invalid entry with the working copy. Writes to the low frequency and high frequency sections occur sequentially in the current locations of a low frequency section pointer and a high frequency section pointer, respectively. In a persistent cache, the reconstruction of a non-persistent cache utilizes the metadata entry that has the most recent timestamp.

Abstract: Automatic network configuration of cluster paths to access data within a cluster is described. Each node of the cluster presents to clients an image of the cluster as a single system. Each node also stores information regarding what node has particular data, as well as information about network addresses and corresponding network capabilities for the various network addresses for each node. When a node receives a request for data, the node determines a node where the request data can be accessed, determines network addresses and network capabilities for each network address of the node, and selects a path to access the node based on the determined information. The receiving node may select the path based on network type and capability, as well as load and performance of the network.

Abstract: Automatic network configuration of cluster paths to access data within a cluster is described. Each node of the cluster presents to clients an image of the cluster as a single system. Each node also stores information regarding what node has particular data, as well as information about network addresses and corresponding network capabilities for the various network addresses for each node. When a node receives a request for data, the node determines a node where the request data can be accessed, determines network addresses and network capabilities for each network address of the node, and selects a path to access the node based on the determined information. The receiving node may select the path based on network type and capability, as well as load and performance of the network.