Abstract: Presented herein are methodologies for implementing erasure coding in a distributed log structured storage system. A method includes receiving a write request for first data from a file system, selecting a physical sector on a selected storage device in an array of storage devices on which to store the first data, assigning a key to the physical sector, storing the key and an indication of the physical sector in a key-to-physical medium map, erasure coding the data, including generating parity data associated with the first data, writing the first data and the parity data as a data stripe to each storage device in the array of storage devices and, in response to receiving the write request, sending the key to the file system. Read, update, and delete procedures in the context of a log structured framework are also described.

Abstract: Presented herein are methodologies for implementing erasure coding in a distributed log structured storage system. A method includes receiving a write request for first data from a file system, selecting a physical sector on a selected storage device in an array of storage devices on which to store the first data, assigning a key to the physical sector, storing the key and an indication of the physical sector in a key-to-physical medium map, erasure coding the data, including generating parity data associated with the first data, writing the first data and the parity data as a data stripe to each storage device in the array of storage devices and, in response to receiving the write request, sending the key to the file system. Read, update, and delete procedures in the context of a log structured framework are also described.

Abstract: Presented herein are methodologies for implementing erasure coding in a distributed log structured storage system. A method includes receiving a write request for first data from a file system, selecting a physical sector on a selected storage device in an array of storage devices on which to store the first data, assigning a key to the physical sector, storing the key and an indication of the physical sector in a key-to-physical medium map, erasure coding the data, including generating parity data associated with the first data, writing the first data and the parity data as a data stripe to each storage device in the array of storage devices and, in response to receiving the write request, sending the key to the file system. Read, update, and delete procedures in the context of a log structured framework are also described.

Abstract: Presented herein are methodologies for implementing erasure coding in a distributed log structured storage system. A method includes receiving a write request for first data from a file system, selecting a physical sector on a selected storage device in an array of storage devices on which to store the first data, assigning a key to the physical sector, storing the key and an indication of the physical sector in a key-to-physical medium map, erasure coding the data, including generating parity data associated with the first data, writing the first data and the parity data as a data stripe to each storage device in the array of storage devices and, in response to receiving the write request, sending the key to the file system. Read, update, and delete procedures in the context of a log structured framework are also described.

Abstract: Creating and using highly available transaction logs in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server receives a transaction entry at a first storage pool of the distributed storage system. The storage controller server further looks up a transaction log to store the transaction entry, where the transaction log is associated with a second storage pool. The storage controller server routes the transaction entry to the second storage pool, wherein the second storage pool stores the transaction entry.

Abstract: A distributed storage system called StorFS that performs distributed data deduplication is described. In an exemplary embodiment, a storage controller server determines if there is duplicative data in a distributed storage system. In this embodiment, the storage controller server determines if an entity is duplicated in the distributed storage system in line with an incoming input/output operation. The storage controller server determines if the entity is duplicated in the distributed storage system by receiving the entity and looking up the entity in a data deduplication table. If the entity exists in the data deduplication table, the storage controller server updates the metadata for the entity to point to the duplicate entity.

Abstract: A distributed storage system that dispatches an input/output request is described. In an exemplary embodiment, a storage controller client receives the input/output request, wherein the distributed storage system includes the storage controller client, a plurality of storage controller servers, a plurality of virtual nodes distributed among a plurality of physical nodes, and each of the plurality of physical nodes is hosted on one of the plurality of storage controller servers. The storage controller client further computes a target virtual node for the input/output request, where the target virtual node is one of the plurality of virtual nodes. Using the computed target virtual node, the storage controller client determines a target physical node that corresponds to the target virtual node, where the target physical node is one of the plurality of physical nodes.

Abstract: A distributed snapshot in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server receives a request to create the snapshot in the distributed storage system, where the distributed storage system includes a plurality of virtual nodes and a plurality of physical nodes, and the source object includes a plurality of stripes. The storage controller server further determines a set of virtual nodes from the plurality of virtual nodes, where each of the set of virtual nodes owns one of the plurality of stripes of the source object. For each of the set of virtual nodes, the storage controller server sends a clone request to that virtual node, where the request is to create a snapshot for the stripe hosted by that virtual node.

Abstract: A distributed multi-layer cache in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server determines if an object is in a cache that stores a plurality of objects. In addition, the distributed storage system includes the cache and a distributed object layer for persistently storing the plurality of objects. The cache further includes a de-duplicated cache layer. The storage controller server accesses the object from the cache if the object is in the cache and accesses the object from the distributed object layer if the object is not in the cache.

Abstract: A distributed storage system that dispatches an input/output request is described. In an exemplary embodiment, a storage controller client receives the input/output request, wherein the distributed storage system includes the storage controller client, a plurality of storage controller servers, a plurality of virtual nodes distributed among a plurality of physical nodes, and each of the plurality of physical nodes is hosted on one of the plurality of storage controller servers. The storage controller client further computes a target virtual node for the input/output request, where the target virtual node is one of the plurality of virtual nodes. Using the computed target virtual node, the storage controller client determines a target physical node that corresponds to the target virtual node, where the target physical node is one of the plurality of physical nodes.

Abstract: A distributed garbage collection in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server generates a live object map of live objects stored on the distributed storage system in a plurality of block segments distributed across a plurality of storage controller servers. The storage controller server further scans the plurality of block segments to generate segment summary statistics, where the segment summary statistics indicates the number of live objects stored in the plurality of block segments. In addition, the storage controller server compacts each of the plurality of block segments that have a low utilization based on the segment summary statistics. Furthermore, the live object map is a probabilistic data structure storing a list of valid objects.

Abstract: A distributed storage system that performs automated load balancing is described. In an exemplary embodiment, a storage controller server determines if there is duplicative data in a distributed storage system. In this embodiment, the storage controller server detects a load balancing event in the distributed storage system, where the distributed storage system includes a plurality of virtual nodes distributed across a plurality of physical nodes. In response to detecting the load balancing event, the storage controller server determines that a current virtual node is to move from a source physical node to a destination physical node. In addition, the current virtual node is one of the plurality of virtual nodes and the source and destination physical nodes are in the plurality of physical nodes. The storage controller server further moves the current virtual node from the source physical node to the destination physical node.

Abstract: Systems and methods of processing network packets are described. These systems and methods provide improved packet processing results by logically and physically separating packet header processing functions from packet data processing functions. In this way, a network processing system may perform network handling operations and data processing operations substantially in parallel. One or more embodiments feature a network adapter for exchanging with a computer network information in the form of packets each including a packet header and packet data. The network adapter includes a packet parser configured to parse an information packet into a packet header and packet data and to direct the packet header to a first memory address for protocol processing and to direct packet data to a second memory address for data processing.

Abstract: A software architecture for the hot add and swap of adapters. The software architecture allows users to replace failed components, upgrade outdated components, and add new functionality, such as new network interfaces, disk interface adapters and storage, without impacting existing users. The software architecture supports the hot add and swap of off-the-shelf adapters, including those adapters that are programmable.

Abstract: A software architecture for the hot add and swap of adapters. The software architecture allows users to replace failed components, upgrade outdated components, and add new functionality, such as new network interfaces, disk interface adapters and storage, without impacting existing users. The software architecture supports the hot add and swap of off-the-shelf adapters, including those adapters that are programmable.

Abstract: A method and apparatus for interfacing buses includes a system interface processor coupled to a first bus and including a command register accessible via a second bus. A request buffer and a response buffer are provided which are accessible via the second bus and coupled to the interface processor. The request buffer can be used to store information to be transmitted from the second bus to the first via the interface processor while the response buffer can be used to store information to be transmitted from the first bus to the second bus via the interface processor. The interface processor may include a status register to indicate the status of the interface controller. The interface controller may also include a command register to receive commands transmitted over the second bus.

Abstract: A software architecture for the hot add and swap of adapters. The software architecture allows users to replace failed components, upgrade outdated components, and add new functionality, such as new network interfaces, disk interface adapters and storage, without impacting existing users. The software architecture supports the hot add and swap of off-the-shelf adapters, including those adapters that are programmable.

Abstract: A software architecture for the hot add and swap of adapters. The software architecture allows users to replace failed components, upgrade outdated components, and add new functionality, such as new network interfaces, disk interface adapters and storage, without impacting existing users. The software architecture supports the hot add and swap of off-the-shelf adapters, including those adapters that are programmable.

Abstract: A method for displaying the slot number and Hardware Instance Number (HIN) of devices attached to a Peripheral Component Interface (PCI) bus is disclosed. The method determines the slot numbers and HINs for every device attached to the PCI bus. This information is displayed to the user. The method thus provides the user with a mechanism for determining the physical slot location of PCI devices in a computer system once the HIN for the PCI device is known.

Abstract: A software architecture for the hot add and swap of adapters. The software architecture allows users to replace failed components, upgrade outdated components, and add new functionality, such as new network interfaces, disk interface adapters and storage, without impacting existing users. The software architecture supports the hot add and swap of off-the-shelf adapters, including those adapters that are programmable.