Abstract: A method includes receiving an object from a client as part of a data ingestion process, directing the object to an object access microservice, providing a copy of the object to a masker worker microservice, masking the copy of the object to create a masked object, and the masking of the copy of the object is performed inline prior to storage of the object, deduplicating the object, and storing the masked object and the object in storage.

Abstract: A method includes retrieving an object from storage and copying the object, generating a list that identifies one or more byte ranges, of the copy of the object, to be masked, providing the list to a masker controller microservice that examines a recipe corresponding to the copy of the object, where the recipe references a slice of the copy of the object, and the slice includes one or more data segments, masking, by the masker controller microservice, a segment of the slice that is in one of the byte ranges, to create a masked segment, and replacing, in the slice, the segment with the masked segment, to create a masked slice and creating a masked object recipe that contains a reference to the masked slice, creating a masked object that includes the masked slice, and that references any unmasked segments of the slice, and deduplicating the masked object.

Abstract: A method includes receiving an object from a client as part of a data ingestion process, by a masker worker, generating a list that identifies one or more byte ranges, of the object, to be masked, providing the list to an object access microservice, by the object access microservice, creating an object recipe of the object, and the object recipe includes one or more slices, and each of the slices includes one or more segments of data, masking only those segments of a slice that are in a byte range identified in the list, creating a masked slice that comprises the masked segments, and creating a masked object recipe that contains a reference to the masked slice, and the masking is performed before the object is stored in storage, creating a masked object that includes the masked slice, and deduplicating the masked object.

Abstract: One example method includes identifying objects that each include one or more segments to be transferred from a source storage tier to a target storage tier, determining a total amount of data to be transferred, using a tiering controller to create worker nodes operable to transfer the segments to the target storage tier, where a number of worker nodes created is based on the amount of data, transferring, from the source storage tier to the target storage tier, only those segments of the objects not already present in the target storage tier, and the transferring of the segments is performed by the worker nodes, and for each of the objects, placing metadata associated with that object in a bucket.

Abstract: Systems and methods for managing content in a flash memory. A data structure such as a Bloom filter is implemented in flash memory such that updates to the data can be performed by overwriting pages in the memory.

Abstract: Efficient data transfer is disclosed. A server provides an endpoint for a client. The client can communicate with a cloud storage system through the server rather than connect directly to the cloud storage system. The server is configured to perform requests in a manner that reduces the amount of data transferred to and from the cloud storage system.

Abstract: Systems and methods for a low-overhead index for a cache. The index is used to access content or segments in the cache by storing at least an identifier and a location. The index is accessed using the identifier. The identifier may be shortened or be a short identifier. Because a collision may occur, the index may also include one or more meta-data values associated with the data segment. Collisions can be resolved by also comparing the metadata of the segment with the metadata stored in the index. If both the short identifier and metadata match those of the segment, the segment is likely in the cache and can be accessed. Segments can also be inserted into the cache.

Abstract: Systems and methods for managing content in a flash memory. Content or data in a flash memory is overwritten when the write operation only requires bits to be set. This improves performance of the flash and extends the life of the flash memory.

Abstract: Systems and methods for rebuilding an index for a flash cache are provided. The index is rebuilt by reading headers of containers stored in the cache and inserting information from the headers into the index. The index is enabled while being rebuild such that lookup operations can be performed using the index even when the index is incomplete. New containers can be inserted into used or unused regions of the cache while the index is being rebuilt.

Abstract: Systems and methods for managing data structures in a flash memory. A library is provided that supports read requests and write requests. The library allows reads and writes to be implemented without requiring the client to understand how the data structure is implemented in the flash memory.

Abstract: Efficient data transfer is disclosed. A server provides an endpoint for a client. The client can communicate with a cloud storage system through the server rather than connect directly to the cloud storage system. The server is configured to perform requests in a manner that reduces the amount of data transferred to and from the cloud storage system.

Abstract: Systems and methods for managing content in a flash memory. Data structures such as trees and graphs are implemented in a flash memory. Nodes or field nodes such as pointers or invalidation bits are updated using an overwrite operation where possible or by invalidating certain fields and temporarily storing changes in an in-memory table.

Abstract: Systems and methods for managing content in a flash memory. A locking data structure is used to control access to data structures and the locking data structure is implemented in flash memory. The locking data structure is updated by overwriting the data such that the associated data structure is identified as locked or unlocked.

Abstract: Systems and methods for determining an access pattern in a computing system. Accesses to a file may contain random accesses and sequential accesses. The file may be divided into multiple regions and the accesses to each region are tracked. The access pattern for each region can then be determined independently of the access patterns of other regions of the file.

Abstract: Systems and methods for managing content in a flash memory. A data structure such as a linked is implemented in flash memory. Nodes can be added to the linked list by overwriting portions of the node when only sets are required to update the node. The nodes may include multiple pointer portions such that existing pointers can be invalided and open pointer portions used for the update to the node.

Abstract: Systems and methods for managing content in a flash memory and to managing a lifespan of the flash memory. The lifespan is managed by throttling writes when usage exceeds certain quotas. The throttling may be performed in a cooperative manner such that the clients writing to the flash memory can implement their own throttling. The quotas are not rigid and allow flexibility to certain situations. When excessive quota is used, future quotas can be managed to bring the overall usage into an expected range.

Abstract: Systems and methods for accessing data stored in multiple locations. A cache and a storage system are associated with an index. Entries in the index identify locations of data in both the cache and the storage system. When an index lookup occurs and an entry in the index identifies at least two locations for the data, the locations are ordered based on at least one factor and the data stored in the optimal location as determined from the at least one factor is returned.

Abstract: Systems and methods for managing content in a flash memory. Systems and methods for implementing hash tables in a flash memory are disclosed. A hash table may include a flat array or an array of buckets that are each associated with a linked list. Adding or removing entries from the hash table or from the linked list are achieved by performing an overwrite operation where possible to pointers affected by the table operation.

Abstract: Systems and methods for managing content in a flash memory. Systems and methods for implementing hash tables in a flash memory are disclosed. A hash table may include a flat array or an array of buckets that are each associated with a linked list. Adding or removing entries from the hash table or from the linked list are achieved by performing an overwrite operation where possible to pointers affected by the table operation.

Abstract: Systems and methods for writing data are provided. A lock-free container and methods of writing to the lock-free container are disclosed. The container is associated with a tail pointer that identifies free space in the container. Threads writing to the container access the tail pointer and update an offset in the tail pointer to account for a size of a write to the container. Multiple threads can write to the same container without having to contend for a container lock.