Abstract: A client device may transmit a data stream including an object (such as a data file or record) to a storage system for storing the object in a storage device. In response to receiving the data stream, the storage system may store or write data of the object into a plurality of logical blocks of the storage device with an end-to-end data protection based at least in part on a comparison of a combination of check codes of a header of the object, the object and padding data with a combination of a plurality of check codes that are generated separately for metadata associated with the object and corresponding parts of the object stored in the plurality of logical blocks. The storage system may further provide an end-to-end data protection for reading data of an object stored in multiple logical blocks of a storage device.

Abstract: A storage engine may obtain one or more object access properties of an object to be received, and determine a type of storage device that is suitable or desirable for storing the object from among different types of storage devices based at least in part on the one or more object access properties of the object to be received. In response to determining the type of storage device, the storage engine may allocate a storage device of such type for the object. The storage engine may then receive the object, and store the object into the allocated storage device.

Abstract: A storage engine may be configured to receive data including a plurality of records from a client device, and generate a plurality of record headers for the plurality of records. The storage engine may then transfer a group of record headers of the plurality of record headers to a storage device to cause the storage device to store the group of record headers consecutively in a data sector of the storage device, and further transfer a subset of payload data of one or more records associated with the group of record headers to the storage device to cause the storage device to store the one or more records after the group of record headers in the data sector of the storage device.

Abstract: A storage engine may be configured to employ different formats for index fragments and index entries of respective records in the index fragments based at least in part on record properties of the respective records in the index fragments, to reduce an amount of memory space that is consumed or used for storing the index fragments in a memory associated with the storage engine, without compromising the efficiency of searching the records stored in a storage device. Using different formats for index fragments covering records of different record properties, the storage engine may further be configured to create, maintain, and update index mappings for records stored or included in the storage device, to provide functionalities of point-lookup, range query, deletion, and additions of the records in the storage device.

Abstract: When performing a recycling operation on a storage device, a storage system may use or create a data buffer in the storage device, and designate the data buffer to temporarily store data of data blocks to be recycled in the storage device using direct memory access (DMA) operations that are performed internally in the storage device, without the need of reading the data of the data blocks from the storage device and writing the data into a host memory of the storage system, thereby saving or reducing the consumptions of the communication bandwidth of a communication channel between the storage system and the storage device, and the memory bandwidth of the host memory.

Abstract: Methods and systems for in-memory metadata reduction in cloud storage system are provided. According to an aspect, a method comprises receiving a first command to write a data stream to a storage device; writing the data stream into a plurality of fragments having logical addresses corresponding to physical addresses on the storage device; and generating an index for individual fragment of the plurality of fragments, the index indicating information to locate the physical addresses of the individual fragment. Individual records in the individual fragment have a same pre-set logical size and all individual records in the individual fragment are continuous, and the index indicates the information including at least: an offset value of the individual record in the individual fragment; the pre-set logical size of the individual record; and a pre-set physical size of the individual record.

Abstract: A storage record engine implemented on a storage system is provided. The storage record engine further organizes hosted storage of the storage system into superblocks and chunks organized by respective metadata, the chunks being further organized into chunk segments amongst superblocks. Persistent storage operations may cause modifications to the metadata, which may be recorded in a transaction log, records of which may be replayed to commit the modifications to hosted storage. The replay functionality may establish recovery of data following a system failure, wherein replay of records of transaction logs in a fashion interleaved with checkpoint metadata avoids preemption of normal storage device activity during a recovery process, and improves responsiveness of the storage system from the perspective of end devices.

Abstract: An electrochemical biosensor based on magnetically coupled LC sensors for the rapid detection of microbial growth and sensitivity to microbials. The engineered LC sensors can be placed in 96 well plates and communicate the reading remotely with a receiver coil for signal analysis. The sensors were validated by testing the growth of Escherichia. coli, Staphylococcus. aureus, and Pseudomonas aeruginosa in the presence and absence of different antibiotics. Drug-resistant strains were used as controls. Bacterial growth was detected within 30 mins of culture inoculation, allowing rapid determination of antibiotic susceptibility at the phenotypic level. The pattern shown in the LC sensor AST is consistent with results collected with traditional optical density (OD) 600 nm measurement, additional validation was also performed with lysogeny broth (LB) dosed with fetal bovine serum (FBS). With the compatibility with 96-well plates, this rapid AST may be used for low-cost, point-of-care applications.

Abstract: A key-value store is provided, implementing multiple-tiered sorted data structures in memory and storage, including concurrent write buffers in memory, and page-level consolidation of updates on storage, where pages are trivially translated in physical-to-virtual address mapping. The key-value store is built on an indexed sorted data structure on storage, occupying much less storage space and incurring much less disk activity in consolidating updates than a conventional log-structured merge tree organized into files. Concurrent write buffers operate concurrently and independently so that data is committed from memory to storage in an efficient manner, while maintaining chronological sequence of delta pages. Trivial mapping allows mappings of a number of physical pages to be omitted, enabling page mapping tables to occupy less storage space, and simplifying processing workload of read operation retrievals from storage.

Abstract: In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata.

Abstract: Methods and systems are provided for improving a journaling file system by reducing journaling overhead utilizing a remapping interface to log-structured cloud block devices such that blocks in a journaling transaction are atomically mapped to their final disk locations with minimum overhead. With log-structured cloud block devices, all blocks may be stored in logfiles and indexed for access. By modifying the index of the destination block to point to the corresponding journal block, the additional write may be avoided, and a journal block may effectively be remapped from its original LBA to its destination LBA without any actual data movement, thereby reducing the file system journaling overhead.

Abstract: In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata.

Abstract: In various embodiments, methods and systems for implementing garbage collection in distributed storage systems are provided. The distributed storage system operates based on independent management of metadata of extent and stream data storage resources. A hybrid garbage collection system based on reference counting garbage collection operations and mark-and-sweep garbage collection operations is implemented. An extent lifetime table that tracks reference weights and mark sequences for extents is initialized and updated based on indications from extent managers and stream managers, respectively. Upon determining that an extent is to be handed-off from weighted reference counting garbage collection operations to mark-and-sweep garbage collection operations, a reference weight field for the extent is voided and a mark sequence field of the extent is updated. The mark sequence field is updated with a latest global sequence number.

Abstract: Methods and systems are provided for improving a journaling file system by reducing journaling overhead utilizing a remapping interface to log-structured cloud block devices such that blocks in a journaling transaction are atomically mapped to their final disk locations with minimum overhead. With log-structured cloud block devices, all blocks may be stored in logfiles and indexed for access. By modifying the index of the destination block to point to the corresponding journal block, the additional write may be avoided, and a journal block may effectively be remapped from its original LBA to its destination LBA without any actual data movement, thereby reducing the file system journaling overhead.

Abstract: In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata.

Abstract: In various embodiments, methods and systems for implementing garbage collection in distributed storage systems are provided. The distributed storage system operates based on independent management of metadata of extent and stream data storage resources. A hybrid garbage collection system based on reference counting garbage collection operations and mark-and-sweep garbage collection operations is implemented. An extent lifetime table that tracks reference weights and mark sequences for extents is initialized and updated based on indications from extent managers and stream managers, respectively. Upon determining that an extent is to be handed-off from weighted reference counting garbage collection operations to mark-and-sweep garbage collection operations, a reference weight field for the extent is voided and a mark sequence field of the extent is updated. The mark sequence field is updated with a latest global sequence number.

Abstract: In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata.

Abstract: In various embodiments, methods and systems for implementing multiple transaction logs in a distributed storage system are provided. A log stream component detects performance metrics of a plurality of log streams. The performance metrics are associated with requests from partitions in the distributed storage system. A transaction component receives a request to execute a transaction using a log stream. The request is received from a partition of the distributed storage system. The performance metrics of the plurality of log streams can be referenced, where the performance metrics indicate a performance capacity of a selected log stream to process the request. A log stream for executing the transaction is determined based on the performance capacity. The log stream selected can also factor request attributes of the request. The transaction component communicates the request to be executed, using the log stream to perform the transaction.

Abstract: In various embodiments, methods and systems for implementing multiple transaction logs in a distributed storage system are provided. A log stream component detects performance metrics of a plurality of log streams. The performance metrics are associated with requests from partitions in the distributed storage system. A transaction component receives a request to execute a transaction using a log stream. The request is received from a partition of the distributed storage system. The performance metrics of the plurality of log streams can be referenced, where the performance metrics indicate a performance capacity of a selected log stream to process the request. A log stream for executing the transaction is determined based on the performance capacity. The log stream selected can also factor request attributes of the request. The transaction component communicates the request to be executed, using the log stream to perform the transaction.

Abstract: Techniques are described that enable enterprise management of public instant message (IM) communications. When a user requests a connection with a public IM service, the connection request is redirected to a gateway server that is associated with a domain specified in the user's user ID. The gateway server acts as a pass through server between the user's IM client application and the public IM service. The gateway server may be configured to log IM communications in which the user participates. The gateway server may also be configured to control whether or not the user is able to participate in point-to-point communications through the user's IM client application. An enterprise routing server may also be implemented in a geo-distributed enterprise to route a redirected public IM connection request to an enterprise IM gateway server based on a geographic location associated with the user.