Abstract: Methods and systems implement local file systems of computing nodes which translate random-access read and random-access write operation calls to append-based operation calls in accordance with a distributed file system (“DFS”) implemented across storage nodes of a cloud network. Computer-executable applications running on the computing nodes may generate kernel-level read and write system calls by application programming interfaces (“APIs”) such as the Portable Operating System Interface (“POSIX”) standard, to a local file system. The local file system may translate these read and write system calls to file operations at a DFS implementing an append-only file system, as well as perform storage reclamation upon the DFS periodically and/or upon storage thresholds being exceeded.

Abstract: Methods and systems implement local file systems of computing nodes which translate random-access read and random-access write operation calls to append-based operation calls in accordance with a distributed file system (“DFS”) implemented across storage nodes of a cloud network. Computer-executable applications running on the computing nodes may generate kernel-level read and write system calls by application programming interfaces (“APIs”) such as the Portable Operating System Interface (“POSIX”) standard, to a local file system. The local file system may translate these read and write system calls to file operations at a DFS implementing an append-only file system, as well as perform storage reclamation upon the DFS periodically and/or upon storage thresholds being exceeded.

Abstract: Techniques for performing data encryption on data to be stored within a storage system are provided. A client application executing on a host machine may generate a data storage write request to write data to a storage system. A host-side module, executing on the host machine receives the write request. The host-side module is configured to generate one or more fingerprints for the data corresponding to the write request, where the one or more fingerprints are unique identifiers used to identify data blocks that make up the data. The host-side module generates encrypted data by encrypting the data blocks using an encryption technique. The encrypted data is then sent to a storage node within the storage system. Deduplication may be performed on the encrypted data using the one or more generated fingerprints.

Abstract: Techniques for performing data encryption on data to be stored within a storage system are provided. A client application executing on a host machine may generate a data storage write request to write data to a storage system. A host-side module, executing on the host machine receives the write request. The host-side module is configured to generate one or more fingerprints for the data corresponding to the write request, where the one or more fingerprints are unique identifiers used to identify data blocks that make up the data. The host-side module generates encrypted data by encrypting the data blocks using an encryption technique. The encrypted data is then sent to a storage node within the storage system. Deduplication may be performed on the encrypted data using the one or more generated fingerprints.

Abstract: Techniques for performing data encryption on data to be stored within a storage system are provided. A client application executing on a host machine may generate a data storage write request to write data to a storage system. A host-side module, executing on the host machine receives the write request. The host-side module is configured to generate one or more fingerprints for the data corresponding to the write request, where the one or more fingerprints are unique identifiers used to identify data blocks that make up the data. The host-side module generates encrypted data by encrypting the data blocks using an encryption technique. The encrypted data is then sent to a storage node within the storage system. Deduplication may be performed on the encrypted data using the one or more generated fingerprints.

Abstract: Techniques for copying forward live data within storage regions used by a file system to new storage regions are provided. Storage regions refer to ranges of persistent storage within a storage medium. One or more live-clump-identifying workers identify a set of live-clump-references. The live-clump references are references to live clumps that belong to live objects. Live clumps represent a unit of data. After identifying the set of live-clump-references, one or more container-identifying workers identify a set of storage regions that contain clumps that correspond to the set of live-clump references. After identifying the set of storage regions corresponding to the set of live-clump-references, a plurality of copy-forward task workers perform copy-forward operations on the set of storage regions to copy live data to new storage regions.

Abstract: Techniques for copying forward live data within storage regions used by a file system to new storage regions are provided. Storage regions refer to ranges of persistent storage within a storage medium. One or more live-clump-identifying workers identify a set of live-clump-references. The live-clump references are references to live clumps that belong to live objects. Live clumps represent a unit of data. After identifying the set of live-clump-references, one or more container-identifying workers identify a set of storage regions that contain clumps that correspond to the set of live-clump references. After identifying the set of storage regions corresponding to the set of live-clump-references, a plurality of copy-forward task workers perform copy-forward operations on the set of storage regions to copy live data to new storage regions.

Abstract: Techniques for performing data encryption on data to be stored within a storage system are provided. A client application executing on a host machine may generate a data storage write request to write data to a storage system. A host-side module, executing on the host machine receives the write request. The host-side module is configured to generate one or more fingerprints for the data corresponding to the write request, where the one or more fingerprints are unique identifiers used to identify data blocks that make up the data. The host-side module generates encrypted data by encrypting the data blocks using an encryption technique. The encrypted data is then sent to a storage node within the storage system. Deduplication may be performed on the encrypted data using the one or more generated fingerprints.

Abstract: Programs, systems and methods are described for efficiently storing data as used under a workflow-driven model. A workflow process is defined to control the processing of data objects through different states, e.g., such as an insurance claim document passing through different stages of processing. The workflow process is modeled and employed to manage the storage system based upon predicted state changes derived from state statistics that can be applied to enhance efficiency. For example, copies of the data object may be automatically made when the data object is expected to change state. Some anticipated states implicating high access may direct a storage location with low access time. Hints or requirements for the data object may be applied upon occurrence of an expected state change. Storage management of expected state changes may be further enhanced through dynamic adjustment of the state statistics using collected historical state information to further enhance efficiency.

Abstract: Programs, systems and methods are described for efficiently storing data as used under a workflow-driven model. A workflow process is defined to control the processing of data objects through different states, e.g., such as an insurance claim document passing through different stages of processing. The workflow process is modeled and employed to manage the storage system based upon predicted state changes derived from state statistics that can be applied to enhance efficiency. For example, copies of the data object may be automatically made when the data object is expected to change state. Some anticipated states implicating high access may direct a storage location with low access time. Hints or requirements for the data object may be applied upon occurrence of an expected state change. Storage management of expected state changes may be further enhanced through dynamic adjustment of the state statistics using collected historical state information to further enhance efficiency.

Abstract: Programs, systems and methods for providing differentiated service levels for a search index are disclosed. Data object documents are processed by extracting terms and scoring each of the terms associated with each document according to criteria to indicate relative importance of the associated document. A plurality of posting lists are generated for each term each comprising entries identifying documents that include the term. The entries are allocated to the different posting lists for the given term depending upon the score for the term associated with particular document. The different posting lists, e.g. a high score and low score posting list, may then be stored as data objects managed according to their indicated importance. For example, the high score posting list data object may be stored in higher performance storage than the low score posting list data object. Scores may be regularly updated.

Abstract: A method, system, and computer program product is provided for efficient policy rule update in a data management system. A policy rule is stored along with the attributes of a data object when the application of the policy rule results in action taken on the data object. A stored policy rule, called an effective policy rule, is subsequently used to restrict the number of data objects examined when a policy rule is added, deleted, modified, or otherwise updated.

Abstract: A virtual binding system ensures that the WORM logic for protecting data immutability cannot be circumvented, effectively guaranteeing WORM property of a WORM storage system composed of rewritable magnetic hard disks. To close the security hole between the rewritable media and the WORM logic, virtual binding securely authenticates the legitimacy of a WORM logic controller before granting data access on a WORM storage media. Furthermore, the system verifies the legitimacy of the WORM logic controller during data access. This approach virtually binds together the WORM logic controller and the WORM storage media even though the WORM logic controller and the WORM storage media may be physically separate.

Abstract: A data migration system performs a tamper-resistant data migration for regulatory compliance systems. The system generates a secure hash for the data object, adds a timestamp to the hash, produces a signature for the data object using a private key, and includes the signature in a signature summary of data objects. Immediately prior to data migration, the system signs the signature summary of the set of data objects to be migrated. The signature of the data object maintains integrity of the data object by preventing undetectable modification to a data object during migration. The signed signature summary maintains completeness by preventing undetectable removal of a data object from or insertion of a data object into the set of data objects during migration.

Abstract: An object storage system provides a storage object to support index structures, logs, and audit trails in WORM storage. The storage object includes a mutable object, an immutable object, or an appendable object. For each storage object, the system maintains an indication of type, such as mutable, immutable, or appendable object type. A mutable object can be arbitrarily modified, an immutable object cannot be modified, and the contents of the appendable object cannot be modified but can be appended with new data. Appended data becomes immutable. The system enables a protection level represented by the object type to be upgraded but not downgraded. Consequently, data committed to be immutable cannot be changed by changing the type of the storage object.

Abstract: According to the present invention, there is provided a system for securing data with a storage system. The system includes at least one storage device. In addition, the system includes a security mechanism for recognizing an attempt to insert or remove the storage device. Moreover, the system includes a management unit to control the insertion and removal of the storage device.

Abstract: A record disposition system classifies records into disposition groups based on an expiration date or disposition date of the records. For each disposition group, the system generates an encoding and decoding function. An encoding module uses the encoding function associated with the disposition group of a record to transform a record locator in an index entry associated with the record. To obtain the record from the index entry, a decoding module decodes the transformed record locator using the corresponding decoding function. The decoding function is stored to allow it to be individually disposed. When a disposition group of records are disposed, disposition of the associated decoding function is also carried out. A record locator in an index entry identifies a block in the expiration control block that identifies a corresponding record. In another embodiment, the encoding function includes an encryption algorithm.

Abstract: According to the present invention, there is provided a method of providing a WORM storage system, the method including a sector-append capability. The method includes receiving data to be written to a WORM storage system. In addition, the method includes identifying a target sector at which the data is to be written. Also, the method includes determining if the received data can be added to the target sector. Moreover, the method includes adding the received data to the target sector if it is determined that the received data can be added to the target sector.

Abstract: According to the present invention, there is provided a system for organizing data objects for fast retrieval. The system includes at least one data storage medium defining data sectors. In addition, the system includes at least one data object on the data storage medium. Also, the system includes at least one key associated with the at least one data object. Moreover, the system includes at least one write-once index on the data storage medium to manage the at least one data object.

Abstract: Files are encrypted and stored on a WORM media device along with their encryption keys, the binary values of which are stored as a combination of written and unwritten sectors in a key storage portion of the media. To shred a file, the associated key is destroyed simply by writing into the unwritten sectors that are associated with the key.