Abstract: Methods and systems for adaptive compression include compressing input data according to a first compression ratio; pausing compression after a predetermined amount of input data is compressed; estimating which of a set of ranges a compressed output size will fall within using current settings; and performing compression on a remainder of the input data according to a second compression ratio based on the estimated range.

Abstract: Systems and methods for generating and storing a data structure for maintaining cache supporting compression and cache-wide deduplication, including generating data structures with fixed size memory regions configured to hold multiple signatures as keys, wherein the number of the fixed size memory regions is bounded. A first mapping is generated from short-length signatures to a storage location and a quantized length measure on a cache storage device; and unused contiguous regions on the cache device are allocated. Metadata and cache page content is retrieved using a single input/output operation; a correctness of a full value of hash functions of uncompressed cache page content is validated; a second mapping is generated from short-length signatures to entries in the first mapping; and verification of whether the cached page content corresponds to a full-length original logical block address using the metadata is performed.

Abstract: Systems and methods are disclosed for parallel machine learning with a cluster of N parallel machine network nodes by determining k network nodes as a subset of the N network nodes to update learning parameters, wherein k is selected to disseminate the updates across all nodes directly or indirectly and to optimize predetermined goals including freshness, balanced communication and computation ratio in the cluster; sending learning unit updates to fewer nodes to reduce communication costs with learning convergence; and sending reduced learning updates and ensuring that the nodes send/receive learning updates in a uniform fashion.

Abstract: A method supports data communication in a mobile application by specifying in the mobile application a program intent and one or more course or fine-grained properties of data objects in terms of tolerance to delay and loss; selecting a transfer policy for a set of data objects based on the application intent; receiving and coalescing intents of one or more applications for object data for the one or more applications; crafting an aggregate transfer policy, and communicating data from one or more applications as an aggregate based on the aggregate transfer policy to programmatically incorporate and benefit from tolerance to delay in the transfer of data.

Abstract: Methods and systems for adaptive compression include compressing input data according to a first compression ratio; pausing compression after a predetermined amount of input data is compressed; estimating which of a set of ranges a compressed output size will fall within using current settings; and performing compression on a remainder of the input data according to a second compression ratio based on the estimated range.

Abstract: Disclosed herein are methods and structures for networks of mobile computers which efficiently synchronizes table data across the mobile computers while exhibiting great tolerance for temporary disconnects.

Abstract: Systems and methods for generating and storing a data structure for maintaining cache supporting compression and cache-wide deduplication, including generating data structures with fixed size memory regions configured to hold multiple signatures as keys, wherein the number of the fixed size memory regions is bounded. A first mapping is generated from short-length signatures to a storage location and a quantized length measure on a cache storage device; and unused contiguous regions on the cache device are allocated. Metadata and cache page content is retrieved using a single input/output operation; a correctness of a full value of hash functions of uncompressed cache page content is validated; a second mapping is generated from short-length signatures to entries in the first mapping; and verification of whether the cached page content corresponds to a full-length original logical block address using the metadata is performed.

Abstract: Disclosed herein are methods and structures for a computer cache that includes its own garbage collection component that reclaims space occupied by free objects in the cache such that the cache avoids retaining deleted objects thereby increasing cache hit ratios and further permits short-lived dirty objects to be deleted without requiring them to be written back to an underlying store.

Abstract: Disclosed are methods and structures that facilitate the synchronization of mobile devices and apps with cloud storage systems. Our disclosure, Simba, provides a unified synchronization mechanism for object and table data in the context of mobile clients. Advantageously, Simba provides application developers a single, API where object data is logically embedded with the table data. On the mobile device, Simba uses a specialized data layout to efficiently store both table data and object data. SQL-like queries are used to store and retrieve all data via a table abstraction. Simba also provides efficient synchronization by splitting object data into chunks which can be synchronized independently. Therefore, if only a small part of an object changes, the full object need not be synced. Advantageously only the changed chunks need be synched.

Abstract: A method supports data communication in a mobile application by specifying in the mobile application a program intent and one or more course or fine-grained properties of data objects in terms of tolerance to delay and loss; selecting a transfer policy for a set of data objects based on the application intent; receiving and coalescing intents of one or more applications for object data for the one or more applications; crafting an aggregate transfer policy, and communicating data from one or more applications as an aggregate based on the aggregate transfer policy to programmatically incorporate and benefit from tolerance to delay in the transfer of data.

Abstract: A system and method for storing data in a content-addressable system is provided. The system includes a content-addressable storage system and a persistent cache. The persistent cache includes a temporary address generator that is configured to generate a temporary address which is associated with data to be stored in the persistent cache, and a non-content-addressable storage system configured to store and retrieve data in the persistent cache using the temporary address. The persistent cache further comprises an address translator configured to map a temporary address associated with the data in the non-content addressable storage system with a content address associated with the data in the content-addressable storage system.

Abstract: In accordance with exemplary embodiments of the present invention, a content addressable data structure system may include directed acyclic graphs (DAGs) of data content that are addressed using both a user-defined search key and content of data blocks. Internal keys of retention roots of the DAGs may be derived from the user-defined search key while the remaining blocks may be content addressed. As opposed to using a content address, the user may provide the search key when retrieving and deleting DAGs retaining the data content. In addition, the internal keys may be implemented using internal content addressable storage operations, such as applying a hash function and employing a distributed hash table.

Abstract: System(s) and method(s) are provided for data management and data processing. For example, various embodiments may include systems and methods relating to relatively larger groups of data being selected with comparable or better performing selection results (e.g., high data redundancy elimination and/or average chunk size). In various embodiments, the system(s) and method(s) may include, for example a data group, block, or chunk combining technique or/and a data group, block, or chunk splitting technique. Various embodiments may include a first standard or typical data grouping, blocking, or chunking technique and/or data group, block, or chunk combining technique or/and a data group, block, or chunk splitting technique. Exemplary system(s) and method(s) may relate to data hashing and/or data elimination. Embodiments may include a look-ahead buffer and determine whether to emit small chunks or large chunks based on characteristics of underlying data and/or particular application of the invention (e.g.

Abstract: A system and method for storing data in a content-addressable system is provided. The system includes a content-addressable storage system and a persistent cache. The persistent cache includes a temporary address generator that is configured to generate a temporary address which is associated with data to be stored in the persistent cache, and a non-content-addressable storage system configured to store and retrieve data in the persistent cache using the temporary address. The persistent cache further comprises an address translator configured to map a temporary address associated with the data in the non-content addressable storage system with a content address associated with the data in the content-addressable storage system.

Abstract: A fixed prefix peer to peer network has a number of physical nodes. The nodes are logically divided into a number of storage slots. Blocks of data are erasure coded into original and redundant data fragments and the resultant fragments of data are stored in slots on separate physical nodes such that no physical node has more than one original and/or redundant fragment. The storage locations of all of the fragments are organized into a logical virtual node (e.g., a supernode). Thus, the supernode and the original block of data can be recovered even if some of the physical nodes are lost.

Abstract: A system and method are disclosed for improving the efficiency of a storage system. At least one application-oriented property is associated with data to be stored on a storage system. Based on the at least one application-oriented property, a manner of implementing at least one caching function in the storage system is determined. Data placement and data movement are controlled in the storage system to implement the at least one caching function.

Abstract: The present invention generally provides a method for grid storage including balancing read and write requests from applications across a first group of nodes in a grid storage system for avoiding hot spots and optimizing performance through smart caching; balancing storage capacity across a second group of nodes in the grid storage system, nodes in the first and second groups being at least one of hardware interchangeable online, capable of being added to change performance or capacity of the grid storage system and capable of being removed to change performance or capacity of the grid storage system; and self managing of the first and second groups of nodes for providing at least one of scalability, self healing after failure of components in the grid storage, non-disruptive upgrades to the grid storage system, and eliminating duplicate data on an object or sub-object level in the grid storage system.

Abstract: A decentralized cloud storage system is provided which comprises at least one client having local resources. The local resources include a first portion which is accessible to the at least one client and a second portion designated to be accessible to and managed by a cloud storage service having a computer readable storage medium. A server logic module is configured to allocate responsibility for performing a set of functions between back-end resources of the cloud storage service and the local resources designated by the at least one client. The system also includes a controller configured to manage the resources designated by the at least one client to implement the functions which have been allocated to the at least one client.

Abstract: Exemplary systems and methods in accordance with embodiments of the present invention may provide a plurality of data services by employing splittable, mergable and transferable redundant chains of data containers. The chains and containers may be automatically split and/or merged in response to changes in storage node network configurations and may be stored in erasure coded fragments distributed across different storage nodes. Data services provided in a distributed secondary storage system utilizing redundant chains of containers may include global deduplication, dynamic scalability, support for multiple redundancy classes, data location, fast reading and writing of data and rebuilding of data due to node or disk failures.

Abstract: The present invention is directed to improving utilization in a peer-to-peer network. Each node in a peer-to-peer network is assigned a number of preferably fixed-size storage slots proportional to the node storage, a first portion of which is allocated to storage zones and a second portion of which is allocated as a free slot reserve. In accordance with an embodiment of an aspect of the invention, this split of a storage zone is delayed until the storage zone becomes full, referred to herein as “lazy splitting”. In accordance with an embodiment of another aspect of the invention, it is advantageous to allocate more storage slots at a node than its actual physical capacity referred to herein as “oversubscription”. In accordance with an embodiment of another aspect of the invention, each physical node is allocated at least one storage zone, even as a new node is added to the system.