Abstract: Example apparatus and methods combine erasure coding with data deduplication to simultaneously reduce the overall redundancy in data while increasing the redundancy of unique data. In one embodiment, an efficient representation of a data set is produced by deduplication. The efficient representation reduces duplicate data in the data set. Redundancy is then added back into the data set using erasure coding. The redundancy that is added back in adds protection to the unique data associated with the efficient representation. How much redundancy is added back in and what type of redundancy is added back in may be controlled based on an attribute (e.g., value, reference count, symbol size, number of symbols) of the unique data. Decisions concerning how much and what type of redundancy to add back in may be adapted over time based, for example, on observations of the efficiency of the overall system.

Abstract: Example apparatus and methods provide improved reclamation, garbage collection (GC) and defragmentation (defrag) for data storage devices including solid state drives (SSD) or shingled magnetic recording (SMR) drives. An erasure code (EC) layer that facilitates logically or physically erasing data from the SSD or SMR as a comprehensive GC or defrag is added to the SSD or SMR. Erased data may be selectively recreated from the EC layer as needed. Pre-planned EC write zones may be established to further optimize GC and defrag. Recreated data may be written to selected locations to further optimize SSD and SMR performance. Erasure code data may be distributed to co-operating devices to further improve GC or defrag. Example apparatus and methods may also facilitate writing data to an SMR drive using tape or VTL applications or processes and providing a pseudo virtual tape library on the SMR drive.

Abstract: Example apparatus and methods control a data storage system to store data in a self-describing logical data storage capsule using a logical cylindrical recording format. Example apparatus and methods assign a searchable, globally unique identifier to the capsule and associate the globally unique identifier with a user. The logical data storage capsule is migrated from a first data storage medium to a second data storage medium without translating or reformatting the data storage capsule. The data storage capsule contains information describing to a data storage device how to migrate the capsule without translating or reformatting the data storage capsule. Example apparatus and methods dynamically select an error correction approach for storing data in the data storage capsule, de-duplicate, and encrypt the data storage capsule. The data storage capsule may be local, or may be part of a cloud-based storage system.

Abstract: Example apparatus and methods monitor conditions in an object storage system. The conditions monitored may include a load balance measure in the system, a capacity balance measure in the system, a fault tolerance measure in the system, or a usage pattern measure in the system. A distribution plan or redistribution plan for storing or moving erasure codes in the object storage system may be determined based on the conditions. The distribution plan or the redistribution plan for the erasure codes may be updated dynamically in response to changing conditions in the object storage system. The distribution or redistribution may depend on a weighted combination of the load balance measure, the capacity balance measure, the fault tolerance measure, or the usage pattern measure so that responding to one sub-optimal condition (e.g., load imbalance) does not create a different sub-optimal condition (e.g., unacceptable fault tolerance).

Abstract: Example apparatus and methods control an error correcting code (ECC) approach for data stored on a solid state device (SSD). The control may be based on a property (e.g., reliability, error state, speed) of an SSD, or on an attribute of the data to be stored. Approaches including a hybrid rateless Reed-Solomon ECC approach or a fountain code ECC approach may be selected. Example apparatus and methods may store padded portions of an ECC at different locations in an SSD. Example apparatus and methods may dynamically generate performance test data about the SSD, and dynamically control the ECC approach based on the performance test data. Different types or numbers of ECC may be produced, stored, and provided for different data sets stored at different SSDs or at different physical locations within an SSD. The SSD may be local, or may be part of a cloud-based storage system.

Abstract: Example apparatus and methods reserve space in a journal using an observation based approach instead of a fixed sized approach or a worst case scenario approach. One example method receives a request to allocate space in a journal to support a file system transaction. The example method reserves an amount of space in the journal based on a pre-existing reservation size estimate. Unlike conventional systems, the estimate is not based on a worst-case scenario. The example method observes the actual amount of storage used in the journal by the file system transaction and then selectively automatically adjusts the pre-existing reservation size estimate. The estimate may slowly shrink if no overflows are encountered but may quickly grow if an overflow is detected.

Abstract: Example apparatus and methods provide a file system that stores files in multiple different logical units. The logical units are associated with different classes of devices. The file system interacts with the logical units through block-based interfaces. The file system is a single file system that spans the multiple logical units. The multiple different logical units include data storage devices for which the file system manages block storage allocation and an object storage system that manages its own block storage allocation rather than having the file system manage its block storage allocation. The file system may maintain a single namespace, hierarchy, or directory structure for the multiple logical units. The object storage system may protect files using redundant copies, erasure codes, or other approaches. An archive process may selectively move files from other devices to the object storage system and leave a symbolic link behind.

Abstract: Example apparatus and methods concern metadata peering that allows a file system to address a file in a local namespace while actually accessing the file using an address used by a shared secondary storage. This allows the file system to operate unmodified and more efficiently in a native mode that addresses files in the shared secondary storage using addresses local to a computer running the file system. An example method accesses an inode associated with a file that is stored in a shared secondary storage. The method accesses a local namespace identifier used by the file system to access the file and accesses a reference used by the shared secondary storage to access the file. The reference is not associated with the local namespace. The method stores an opaque value in the inode. The opaque value facilitates accessing the file in the shared secondary storage using the local namespace.

Abstract: Methods, apparatus, and other embodiments facilitate transitioning a database between two modes. The two modes include a mode where table information for separate tables in the database is stored in a single monolithic file and a mode where the table information for separate tables is stored in separate files. Conventional attempts to make this type of transition lock the database for an unacceptable amount of time. Example methods and apparatus make a copy of a database that is to be transitioned, lock the copy, make a second instance of the copy, and then perform the work necessary for the transition in the second instance. This allows the original database to remain running. When the second instance is complete, the second instance is synchronized to the original database and then the transition is completed by copying the second instance to the original or failing the original over to the second instance.

Abstract: A magnetic tape for use in a tape drive includes a plurality of tracks and a plurality of media defect characteristics. The plurality of tracks are laid out along a length of the magnetic tape, each of the plurality of tracks being configured to receive data that is organized into a plurality of fixed-size subdata sets each including a plurality of codeword interleaves. The data is laid out on the plurality of tracks in accordance with a tape layout allocation strategy as determined by a tape layout module. The tape layout module includes a tape layout optimization system that receives information on the plurality of media defect characteristics, the tape layout optimization system determining the tape layout allocation strategy based at least in part on at least one of the plurality of media defect characteristics; and the tape layout optimization system determining the tape layout allocation strategy further based on distance spectra between pairs of the plurality of codeword interleaves.

Abstract: Methods and apparatus deduplicate and erasure code a message in a data storage system. One example apparatus includes a first chunking circuit that generates a set of data chunks from a message, an outer precoding circuit that generates a set of precoded data chunks and a set of parity symbols from the set of data chunks, a second chunking circuit that generates a set of chunked parity symbols from the set of parity symbols, a deduplication circuit that generates a set of deduplicated data chunks by deduplicating the set of precoded chunks or the set of chunked parity symbols, an unequal error protection (UEP) circuit that generates an encoded message from the set of deduplicated data chunks, and a storage circuit that controls the data storage system to store the set of deduplicated data chunks, the set of parity symbols, or the encoded message.

Abstract: Embodiments disclosed herein provide systems and method for storing metadata to unused portions of a virtual disk file. In a particular embodiment, a method provides selecting a virtual disk file stored on a data storage volume and identifying unused portions of the virtual disk file. The method further provides writing metadata for the virtual disk file in the unused portions of the virtual disk file.

Abstract: Methods and apparatus associated with data cold storage are described. Example apparatus include an array of data storage devices arranged in rows and columns. Columns of the array are orthogonal to rows. A row has an associated row-centric power supply, and a column has an associated column-centric local electronics module (LEM) that controls a data storage device in the column independently of other data storage devices in the array. Example apparatus include logics that control a power mode of a data storage device independently of other data storage devices in the array, that control a power mode of an LEM, that adaptively regulate the level of data stored in a buffer, and that determine whether a data object will be stored in the buffer or stored on a data storage device in the array, based on the probability the data object will be accessed within a threshold period of time.

Abstract: Example apparatus and methods provide a log structured block device for a hard disk drive (HDD). Data that is to be stored on an HDD is serialized and written as a series of data blocks using a sequential write. Information about where individual data blocks were supposed to be written (e.g., actual address, neighboring data blocks), where data blocks were actually written, and how often data blocks are accessed is maintained. During garbage collection, data blocks that are being accessed with similar frequencies may be relocated together, with the most frequently accessed (e.g., hottest) data blocks migrating to the outer cylinders of the disk and the least frequently accessed (e.g., coldest) data blocks migrating to the inner cylinders. Blocks stored in the same temperature regions that were intended to be located together when written may be repositioned to facilitate sequential reads.

Abstract: Example apparatus, methods, and computers control configurable, variable data reduction. One example method includes identifying data reduction controlling attributes in an object to be data reduced by a configurable variable data reducer. The attributes provide information upon which decisions concerning whether and/or how to data reduce the object can be based. The example method also includes controlling a configurable variable data reducer to selectively data reduce the object based, at least in part, on the data reduction controlling attributes. The control exercised can determine whether, where, when, and/or how data reduction will proceed.

Abstract: Example apparatus and methods access a Linear Tape File System (LTFS) repository that stores data associated with an external entity (e.g., file system, application). Data is copied from the LTFS repository, selectively converted to a form associated with a cloud repository, and provided to the cloud repository. A mapping between corresponding addresses in the LTFS repository and cloud repository is established. Requests by the external entity for data may be satisfied from either the LTFS repository or the cloud repository until a changeover event is detected and are then satisfied from the cloud repository afterwards. The external entity runs on a different computer than the method. The method is transparent to the external entity, is performed independent of the external entity, and continues under its own control until a selected amount of data is moved from the LTFS repository to the cloud repository.

Abstract: Storage conditioning for a data storage system having D data storage devices (DSDs) is provided. E erasure codes (ECs) for an object are stored in the system, D>E. A map of d E-sized vectors of the D DSDs is produced. A DSD appears in e vectors. The ratio d/e is the reduced form of D/E. A hash value is produced for the object. A destination vector for storing the ECs is selected using the hash value according to a pre-determined, substantially uniform distribution. A compromised vector affected by a first DSD becoming unable to store ECs is identified. An intact vector that is not affected by the first DSD is identified. A complete set of ECs is produced from an incomplete set of ECs in the compromised vector and is distributed to the intact vector then copied back when the compromised vector is once again intact.

Abstract: Example apparatus and methods create and maintain a functional mirror of file system metadata for an active file system. The functional mirror may be implemented as a set of feature vectors in a feature space stored in a spatial database that allow a journal functionality for information received from the active file system. Mirror creation and maintenance can operate in parallel. The functional mirror may be created primarily using a systematic, system-wide, multi-threaded crawler approach whose results are updated using an event-driven approach that accounts for changes that may obsolete information acquired during the crawler approach. Transaction identifiers may be stored and potentially obsolete information preserved to facilitate providing views and rewind features.

Abstract: Methods and apparatus associated with storing data in high or low energy zones are described. Example apparatus include a data storage system (DSS) that protects a message using an erasure code (EC). A location in the DSS may have an energy efficiency rating or a latency. Example apparatus include logics that produce an EC that has a likelihood of use, that select a location to store the EC in the DSS based on the energy efficiency rating, the latency, or the likelihood of use, that store the EC in the location, and that compute an order of retrieval for an EC stored in the location. The order of retrieval may be based on the energy efficiency rating or the latency. The EC may also have a priority based on the number of erasures for which the EC corrects.