Abstract: Example object storage systems, bookkeeping engines, and methods provide quota usage monitoring for control entities, such as accounts, users, and buckets. An object data store is configured to enable control entities to access data objects associated with each control entity. Data objects are mapped to the control entities and the data objects are processed to identify object usage values corresponding to each combination of data object and control entity. Total usage values are calculated for each control entity and used to determine a data object access response for a target data object and associated control entities.

Abstract: Example object storage systems, meta object generators, and methods provide versioned meta objects for internal operational data that may be replicated between data object stores. A meta object may be generated that includes meta object data, such as internal operational data. A meta object identifier may be assigned to the meta object. A first version identifier may be associated with the meta object data and a second version identifier may be associated with a marker for the meta object, where the marker prevents exposure as a user data object.

Abstract: Example object storage systems, replication managers, and methods provide meta objects for priority replication of configuration data between data object stores. A meta object may be generated that includes configuration data. Data objects are selected for replication from a first replication queue. Meta objects associated with the data objects in the first replication queue are replicated, independent of the first replication queue, between object data stores.

Abstract: Example object storage systems, meta object generators, and methods provide meta objects for replication of configuration data between data object stores. A meta object may be generated that includes configuration data. A meta object identifier may be assigned to the meta object. The meta object may be stored in a first object store with related data objects and may be replicated to a second object data store with the related data objects.

Abstract: The described methods, systems, and other aspects can advantageously provide balanced multi-stage processing of non-uniform object data. An example method may receive a list of buckets. Each of the buckets in the list of buckets can store one or more restorable objects. The method further comprises distributing the list of buckets to the two or more second nodes; determining a number of the one or more restorable objects in each bucket; determining a size of the one or more restorable objects in each bucket; generating batches of to-be-restored data objects based on the determined number of the one or more restorable objects in each bucket and the determined size of the one or more restorable objects in each bucket; and distributing the batches among the two or more second nodes for storage-related task processing.

Abstract: These methods, systems, and other aspects can provide for adaptive distribution/re-distribution of data processing tasks. An example method may create, by a processor in a first node, a status object for a data processing task and assign, by the processor in the first node, the data processing task to a worker of a second node. Further, after a first period of time, the method may query, by the processor in the first node, the status object for a task status of the assigned data processing task reported by the second node. The task status reflects a status of execution of the assigned data processing task by the second node. The method determines, by the processor in the first node, to reassign the assigned data processing task based on a received task status reported by the second node, and reassign, by the processor in the first node, the assigned task.

Abstract: Example object storage systems and methods provide priority metadata processing. Metadata operations are received in response to change events for at least one data object. The metadata operations may include system operations configured to manage changes to data objects and user-method operations configured to execute user-defined methods using the data objects. System operations are executed with a first priority in response to system operations with the first priority being available for processing. User-method operations are executed with a second priority in response to no metadata operations with the first priority being available for processing.

Abstract: Example object storage systems, replication managers, and methods provide meta objects for priority replication of configuration data between data object stores. A meta object may be generated that includes configuration data. Data objects are selected for replication from a first replication queue. Meta objects associated with the data objects in the first replication queue are replicated, independent of the first replication queue, between object data stores.

Abstract: Example object storage systems, meta object generators, and methods provide meta objects for replication of configuration data between data object stores. A meta object may be generated that includes configuration data. A meta object identifier may be assigned to the meta object. The meta object may be stored in a first object store with related data objects and may be replicated to a second object data store with the related data objects.

Abstract: The described methods, systems, and other aspects can advantageously provide balanced multi-stage processing of non-uniform object data. An example method may receive a list of buckets. Each of the buckets in the list of buckets can store one or more restorable objects. The method further comprises distributing the list of buckets to the two or more second nodes; determining a number of the one or more restorable objects in each bucket; determining a size of the one or more restorable objects in each bucket; generating batches of to-be-restored data objects based on the determined number of the one or more restorable objects in each bucket and the determined size of the one or more restorable objects in each bucket; and distributing the batches among the two or more second nodes for storage-related task processing.

Abstract: Example object storage systems, meta object generators, and methods provide versioned meta objects for internal operational data that may be replicated between data object stores. A meta object may be generated that includes meta object data, such as internal operational data. A meta object identifier may be assigned to the meta object. A first version identifier may be associated with the meta object data and a second version identifier may be associated with a marker for the meta object, where the marker prevents exposure as a user data object.

Abstract: These methods, systems, and other aspects can provide for adaptive distribution/re-distribution of data processing tasks. An example method may create, by a processor in a first node, a status object for a data processing task and assign, by the processor in the first node, the data processing task to a worker of a second node. Further, after a first period of time, the method may query, by the processor in the first node, the status object for a task status of the assigned data processing task reported by the second node. The task status reflects a status of execution of the assigned data processing task by the second node. The method determines, by the processor in the first node, to reassign the assigned data processing task based on a received task status reported by the second node, and reassign, by the processor in the first node, the assigned task.

Abstract: Example object storage systems, bookkeeping engines, and methods provide quota usage monitoring for control entities, such as accounts, users, and buckets. An object data store is configured to enable control entities to access data objects associated with each control entity. Data objects are mapped to the control entities and the data objects are processed to identify object usage values corresponding to each combination of data object and control entity. Total usage values are calculated for each control entity and used to determine a data object access response for a target data object and associated control entities.

Abstract: Example object storage systems and methods provide priority metadata processing. Metadata operations are received in response to change events for at least one data object. The metadata operations may include system operations configured to manage changes to data objects and user-method operations configured to execute user-defined methods using the data objects. System operations are executed with a first priority in response to system operations with the first priority being available for processing. User-method operations are executed with a second priority in response to no metadata operations with the first priority being available for processing.

Abstract: A hardware and/or software facility to enable emulated storage devices to share data stored on physical storage resources of a storage system. The facility may be implemented on a virtual tape library (VTL) system configured to back up data sets that have a high level of redundancy on multiple virtual tapes. The facility organizes all or a portion of the physical storage resources according to a common store data layout. By enabling emulated storage devices to share data stored on physical storage resources, the facility enables deduplication across the emulated storage devices irrespective of the emulated storage device to which the data is or was originally written, thereby eliminating duplicate data on the physical storage resources and improving the storage consumption of the emulated storage devices on the physical storage resources.

Abstract: Method and system for managing storage units are provided. A free space module scans a storage unit data structure and a reference data structure to generate an intermediate data structure that identifies storage units that are not referenced by any storage unit client. A lookup module is initiated and the storage unit clients are notified that all new references to any storage unit should be verified with the lookup module. The free space module then verifies if any of the storage units in the intermediate data structure have been referenced since the intermediate data structure was created. Any referenced storage units are removed from the intermediate data structure and a data structure identifying unreferenced storage units is generated. The data structure is then used to allocate the identified storage units.

Abstract: A hardware and/or software facility to enable emulated storage devices to share data stored on physical storage resources of a storage system. The facility may be implemented on a virtual tape library (VTL) system configured to back up data sets that have a high level of redundancy on multiple virtual tapes. The facility organizes all or a portion of the physical storage resources according to a common store data layout. By enabling emulated storage devices to share data stored on physical storage resources, the facility enables deduplication across the emulated storage devices irrespective of the emulated storage device to which the data is or was originally written, thereby eliminating duplicate data on the physical storage resources and improving the storage consumption of the emulated storage devices on the physical storage resources.

Abstract: A hardware and/or software facility to enable emulated storage devices to share data stored on physical storage resources of a storage system. The facility may be implemented on a virtual tape library (VTL) system configured to back up data sets that have a high level of redundancy on multiple virtual tapes. The facility organizes all or a portion of the physical storage resources according to a common store data layout. By enabling emulated storage devices to share data stored on physical storage resources, the facility enables deduplication across the emulated storage devices irrespective of the emulated storage device to which the data is or was originally written, thereby eliminating duplicate data on the physical storage resources and improving the storage consumption of the emulated storage devices on the physical storage resources.

Abstract: Method and system for managing storage units are provided. A free space module scans a storage unit data structure and a reference data structure to generate an intermediate data structure that identifies storage units that are not referenced by any storage unit client. A lookup module is initiated and the storage unit clients are notified that all new references to any storage unit should be verified with the lookup module. The free space module then verifies if any of the storage units in the intermediate data structure have been referenced since the intermediate data structure was created. Any referenced storage units are removed from the intermediate data structure and a data structure identifying unreferenced storage units is generated. The data structure is then used to allocate the identified storage units.

Abstract: Method and system for managing storage units are provided. A free space module scans a storage unit data structure and a reference data structure to generate an intermediate data structure that identifies storage units that are not referenced by any storage unit client. A lookup module is initiated and the storage unit clients are notified that all new references to any storage unit should be verified with the lookup module. The free space module then verifies if any of the storage units in the intermediate data structure have been referenced since the intermediate data structure was created. Any referenced storage units are removed from the intermediate data structure and a data structure identifying unreferenced storage units is generated. The data structure is then used to allocate the identified storage units.